Image generation method, electronic device and storage medium

Abstract

The embodiment of the present invention discloses an image generation method, an electronic device and a storage medium. The method includes: obtaining a historical sample image; performing image recognition based on the historical sample image to determine the target main image, and performing mask segmentation based on the target main image to obtain a first mask image; determining the target redrawing area in the first mask image based on a preset area selection method, and performing image generation in the target redrawing area based on the target item image in the item resource pool to obtain a second mask image; performing image redrawing based on the historical sample image and the second mask image to obtain an augmented sample image containing the target item image. Through the technical solution of the embodiment of the present invention, effective augmentation processing of historical sample images can be achieved to obtain richer augmented sample images, that is, effectively increase negative sample images, improve the generation efficiency and accuracy of augmented sample images, and ensure the balance in quantity between positive and negative sample images.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to an image generation method, an electronic device, and a storage medium. Background Art

[0002] As the global digital transformation accelerates, the fifth generation mobile communication technology (5G) in the communications industry has been widely deployed through communication base stations, which has greatly promoted the development of the communications industry. After the construction of a communication base station is completed, it is necessary to conduct environmental testing inside the communication base station to achieve acceptance and maintenance of the communication base station.

[0003] At present, manual inspection is usually used to collect base station images and detect base station environment at the communication base station site. It is also possible to train an environment detection model through the collected historical base station images, and use the trained environment detection model to perform base station environment detection.

[0004] However, most of the historical base station images collected are qualified images, that is, positive sample images. Using such a sample image set where the number of positive sample images is much larger than the number of negative sample images, the environment detection model trained has the problem of inaccurate environment detection results output by the model. Summary of the invention

[0005] The embodiments of the present invention provide an image generation method, an electronic device and a storage medium to realize effective augmentation processing of historical sample images to obtain richer augmented sample images, that is, to effectively increase negative sample images, improve the generation efficiency and accuracy of augmented sample images, and ensure the balance in quantity between positive and negative sample images.

[0006] In a first aspect, an embodiment of the present invention provides an image generation method, comprising:

[0007] Acquire a historical sample image; the historical sample image includes a target subject;

[0008] Performing image recognition based on the historical sample image to determine a target subject image, and performing mask segmentation based on the target subject image to obtain a first mask image;

[0009] Determine a target redrawing area in the first mask image based on a preset area selection method, and generate an image in the target redrawing area based on a target object image in an object resource pool to obtain a second mask image;

[0010] Image redrawing is performed based on the historical sample image and the second mask image to obtain an augmented sample image containing the target object image; wherein the augmented sample image is a negative sample image.

[0011] In a second aspect, an embodiment of the present invention further provides an image generating device, the device comprising:

[0012] A historical sample image acquisition module, used to acquire historical sample images; the historical sample images include target subject images;

[0013] A first mask image determination module, configured to perform image recognition based on the historical sample image, determine a target subject image, and perform mask segmentation based on the target subject image to obtain a first mask image;

[0014] a second mask image determination module, configured to determine a target redrawing area in the first mask image based on a preset area selection method, and generate an image in the target redrawing area based on a target item image in an item resource pool to obtain a second mask image;

[0015] An augmented sample image determination module is used to redraw the image based on the historical sample image and the second mask image to obtain an augmented sample image containing the target object image; wherein the augmented sample image is a negative sample image.

[0016] In a third aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0017] one or more processors;

[0018] A memory for storing one or more programs;

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the image generating method provided by any embodiment of the present invention.

[0020] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an image generating method as provided in any embodiment of the present invention.

[0021] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the image generation method provided by any embodiment of the present invention.

[0022] The technical solution of the embodiment of the present invention is to obtain a historical sample image; perform image recognition based on the historical sample image to determine the target subject image, and perform mask segmentation based on the target subject image, so as to obtain a first mask image that effectively distinguishes the target subject in the historical sample image from other subjects different from the target subject; determine the target redrawing area in the first mask image based on a preset area selection method, and perform image generation in the target redrawing area based on the target item image in the item resource pool, so as to obtain a second mask image that can be used to add the target item image to the historical sample image; perform image redrawing based on the historical sample image and the second mask image to obtain an augmented sample image containing the target item image, thereby realizing effective augmentation processing of the historical sample image to obtain a richer augmented sample image, that is, effectively increase the negative sample image, improve the generation efficiency and accuracy of the augmented sample image, and ensure the balance in quantity of positive and negative sample images.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 is a flow chart of an image generating method provided by Embodiment 1 of the present invention;

[0026] Figure 2 is an example diagram of a historical sample image involved in the first embodiment of the present invention;

[0027] Figure 3 is an example diagram of a first mask image involved in Embodiment 1 of the present invention;

[0028] Figure 4 This is an example diagram of an online deployment method involved in Embodiment 1 of the present invention;

[0029] Figure 5 is a flow chart of an image generating method provided by Embodiment 2 of the present invention;

[0030] Figure 6 is an example diagram of a target redrawing area involved in the second embodiment of the present invention;

[0031] Figure 7 is an example diagram of a comparison image involved in the second embodiment of the present invention;

[0032] Figure 8 is a structural schematic diagram of an image generating device provided by Embodiment 3 of the present invention;

[0033] Fig. 9 It is a schematic diagram of the structure of an electronic device for implementing the image generating method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] Embodiment 1

[0037] Figure 1 A flowchart of an image generation method is provided for the first embodiment of the present invention. This embodiment is applicable to the case where a historical sample image is augmented to generate a negative sample image. The method can be executed by an image generation device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0038] S110: Acquire historical sample images.

[0039] Among them, the historical sample image includes the target subject. It can be understood that there is an inclusion relationship between the target subject and the historical sample image. For example, taking a communication base station as an example, the historical sample image of the communication base station contains the ground that needs to be tested for environment, that is, the ground is the target subject. The historical sample image may refer to the collected scene image. The historical sample image has corresponding label information for model training. Any image collected from a certain type of area that needs to be tested can be used as a scene image. Using the above example, the image collected from a specific area in a communication base station that needs to be tested for environment is used as a scene image. The historical sample image can be used to characterize the real scene of the scene. For example, the historical sample image may include but is not limited to indoor sample images and outdoor sample images. The target subject may refer to the main part to be generated for the image. The internal adjustment operation of the target subject can be performed to achieve the image augmentation of the overall historical sample image to generate an augmented sample image. For example, the target subject may be but is not limited to the sky, ground or wall in the historical sample image.

[0040] Specifically, the historically collected on-site images are stored in a resource library as historical sample images, and the historical sample images in the resource library are acquired when an image generation instruction is received.

[0041] Exemplarily, taking a communication base station as an example, the historical sample image is a field image collected when verifying the communication base station. Figure 2 An example of a historical sample image in a communication base station is given. Figure 2 , showing three historical sample images. The target subject in the historical sample images is the ground in the communication base station. In addition to the target subject, the historical sample images also include non-target subject parts, such as tables, cabinets, and ladders.

[0042] S120 , performing image recognition based on the historical sample image to determine a target main image, and performing mask segmentation based on the target main image to obtain a first mask image.

[0043] Among them, the target subject image may refer to an image containing the area where the target subject is located. The target subject image has the same image size as the historical sample image. The first mask image may refer to an image that can significantly distinguish the area where the target subject is located from the area in the image that is different from the target subject. The mask image can be used to protect the area that needs to be retained, or to protect the area that does not need to be modified. The mask image can also be used to modify the image content of the area that needs to be modified in the original image (such as the historical sample image) while protecting the area that needs to be retained.

[0044] Exemplarily, a visual language model can be used to implement an operation of determining a first mask image based on a historical sample image. The visual language model can be used to detect and recognize an input image through a prompt text, determine and segment the image content related to the prompt text in the input image, obtain a segmented image, and output it.

[0045] Continuing with the above example, in this embodiment, the prompt text used by the visual language model can be set to the ground by default. The historical sample image is input into the visual language model, and image recognition is performed on the historical sample image in the visual language model to determine the area where the ground is located and perform contour segmentation on the area to obtain the target main image. In the visual language model, the target main image is binarized to distinguish the area where the ground is located from the non-ground area to obtain the first mask image.

[0046] It should be noted that in other scenarios, such as in the process of augmenting related images of wall cleanliness, the target subject may be a wall. Then, when the historical sample image containing the wall is input into the visual language model, the wall can be input into the visual language model as a prompt text, so that the visual language model can perform target subject recognition and mask segmentation for the target subject that the user wants to retain, and obtain a first mask image.

[0047] On the basis of the above technical solution, "performing image recognition based on historical sample images to determine the target subject image" may include: inputting the historical sample images into a pre-trained target subject recognition model; performing target subject recognition on the historical sample images in the target subject recognition model to determine the target subject in the historical sample images; performing image segmentation on the target subject to obtain a target subject image with the same image size as the historical sample image.

[0048] The target subject recognition model may refer to a pre-trained image recognition model. The target subject recognition model may be used to recognize and mark the target subject.

[0049] Specifically, the historical sample images are input into a pre-trained target subject recognition model; the target subject recognition is performed on the historical sample images in the target subject recognition model, the target subject in the historical sample images is determined and the contour information of the target subject is marked; the target subject is segmented based on the determined contour information, and the part where the target subject is located is retained to obtain a target subject image with the same image size as the historical sample image, so that the target subject recognition model with targeted recognition function can be used to more accurately and conveniently determine the target subject and the contour information of the target subject, and image segmentation is performed on the basis of the accurate contour of the target subject to obtain the target subject image, thereby further improving the accuracy of obtaining the target subject image and ensuring the effect of subsequent image generation.

[0050] On the basis of the above technical solution, "performing mask segmentation based on the target subject image to obtain a first mask image" may include: adjusting the pixel values ​​of the pixel points corresponding to the target subject in the target subject image to a first value, and adjusting the pixel values ​​of other pixel points in the target subject image that are different from the target subject to a second value, to obtain the first mask image.

[0051] The first value and the second value may both refer to preset pixel values. The first value and the second value may be used to significantly distinguish image regions corresponding to the two values.

[0052] Exemplarily, following the above example, the target subject image is binarized to accurately distinguish the target subject from the non-target subject that is different from the target subject. The pixel values ​​of the pixels corresponding to the target subject in the target subject image are set to 1, and the pixel values ​​of other pixels that are different from the target subject in the target subject image are set to 0, thereby obtaining a first mask image that can accurately distinguish the target subject and can be effectively edited in the area where the target subject is located. Figure 3 illustrate, Figure 3 Zhongweijiang Figure 2 An example of converting each historical sample image in a communication base station into a first mask image, wherein the ground is white, and other parts different from the ground are black.

[0053] S130 , determining a target redrawing area in the first mask image based on a preset area selection method, and generating an image in the target redrawing area based on a target object image in an object resource pool to obtain a second mask image.

[0054] Among them, the preset area selection method may refer to a preset method for selecting a desired area in an image. For example, the preset area selection method may be, but is not limited to, a grayscale similarity selection method or a geometric shape selection method. The target redrawing area may refer to a part of the area where the target body is located. The target redrawing area may be used to indicate an editable or modifiable area in an image. The item resource pool may refer to a database storing multiple types of item images. A large number of item images may be stored in the item resource pool. The item resource pool may be used to provide target item images that users expect to add to an original image (such as a historical sample image). The target item image may refer to an image or map containing only the target item. Following the above example, in a communication base station scenario, the target item may refer to an item that cannot appear on the ground or that may affect the environmental detection result when the communication base station is subjected to environmental detection. For example, the target item may include, but is not limited to, waste paper, waste cables, stones, and water bottles. The second mask image may refer to an image containing the target item to be added to the original image (such as a historical sample image). The second mask image may be used to add the target item to the original image (such as a historical sample image). For example, the second mask image may be an image obtained by adding the target object image to the target redrawing region in the first mask image.

[0055] Specifically, based on the predetermined target object contour, the target object main body area in the first mask image is selected, and the target object contour is added at the selected target position, and the partial area in the first mask image surrounded by the target object contour is used as the target redrawing area. The target object image of the target object corresponding to the target object contour is added to the target redrawing area to obtain the second mask image.

[0056] S140: redrawing the image based on the historical sample image and the second mask image to obtain an augmented sample image including the target object image.

[0057] In this embodiment, the augmented sample image is a negative sample image. The augmented sample image may refer to a sample image with added target object features. The augmented sample image may be used to pair with the positive sample image for training the model. Image augmentation may be understood as changing an image (such as adding a regional image to an image to generate a new image) to increase the breadth of the image. For example, image augmentation may be performed by adding multiple elements (such as target objects) to the image to increase the number or content richness of the image.

[0058] Specifically, image redrawing, such as mask processing, is performed based on the historical sample image and the second mask image, and a target object image is added at the same position as the target object in the second mask image in the historical sample image to obtain an augmented sample image containing the target object image.

[0059] It should be noted that, in this embodiment, a positive sample image can be selected as a historical sample image, and a corresponding negative sample generation operation can be performed on the positive sample image, so as to quickly narrow the gap in the number of positive and negative sample images and effectively achieve a balance in the number of positive and negative sample images.

[0060] The technical solution of the embodiment of the present invention is as follows: acquiring a historical sample image; performing image recognition based on the historical sample image to determine a target subject image, and performing mask segmentation based on the target subject image, thereby obtaining a first mask image that effectively distinguishes the target subject in the historical sample image from other subjects different from the target subject; determining a target redrawing area in the first mask image based on a preset area selection method, and performing image generation in the target redrawing area based on a target item image in an item resource pool, thereby obtaining a second mask image that can be used to add a target item image to the historical sample image; performing image redrawing based on the historical sample image and the second mask image to obtain an augmented sample image containing the target item image, thereby achieving effective augmentation processing of the historical sample image to obtain a richer augmented sample image, that is, effectively increasing negative sample images, improving the generation efficiency and accuracy of the augmented sample images, and ensuring the balance in quantity between positive and negative sample images.

[0061] On the basis of the above technical solution, the method also includes: obtaining multiple training sample data, wherein the training sample data includes: historical sample images, label information corresponding to the historical sample images, augmented sample images, and label information corresponding to the augmented sample images; inputting the historical sample images and the augmented sample images into a pre-built preset classification model for classification prediction to obtain the classification prediction results of the preset classification model; determining the loss value based on the classification prediction results, the label information corresponding to the model input image and the preset loss function; and correcting the model parameters in the preset classification model based on the loss value to obtain a usable target classification model.

[0062] Among them, the preset classification model may refer to a classification model that has not been trained. The preset classification model can be used to perform binary classification processing on the input of the model and output the classification prediction result. The label information may refer to the accurate classification result of the image corresponding to the label information. The preset loss function may refer to the cross entropy loss function of the preset classification model. The loss value can be used to characterize the degree of difference between the classification prediction result output by the model and the label information. The target classification model may refer to a binary classification model that meets the preset convergence conditions to complete the training.

[0063] Exemplarily, the preset classification model may be composed of a convolution layer, a pooling layer, and a fully connected layer. The input data received by the preset classification model is usually an image. Before the image is input into the convolution layer, the input image needs to be normalized in the channel or time / frequency dimension, that is, the original pixel value in the image is normalized to the range of [0,1]. The standardized processing of features in the input data helps to improve the learning efficiency and performance of the preset classification model. The convolution layer can use a set of learnable filters (or convolution kernels) to extract features from the image. In order to perform feature selection and information filtering, the pooling layer is generally used for sampling to reduce the computational complexity by reducing the dimension of each feature map through the pooling layer, and more accurately extract more abstract feature representations. After multiple convolution and pooling operations, the model will obtain a high-dimensional feature. Finally, the model judges and classifies the features through the classification head to obtain the corresponding classification results.

[0064] It should be noted that in the process of training the preset classification model using historical sample images and augmented sample images, the size of the input image can be 512×512. For the training of the preset classification model, the Stochastic Gradient Descent (SGD) algorithm can also be used, and the batch size is set to 32. After feature normalization, the cross entropy loss function is minimized, the initial learning rate is set to 0.01, the momentum coefficient is set to 0.9, and the weight decay coefficient is set to 1e-5. In addition, the cosine annealing algorithm can be used to dynamically adjust the learning rate during the training of the preset classification model.

[0065] On the basis of the above technical solution, the method also includes: when receiving an environmental detection instruction, obtaining a field image of the communication base station to be detected; inputting the field image into a pre-trained target classification model for environmental detection; based on the output of the target classification model, determining the environmental detection result of the communication base station to be detected; based on the environmental detection result and the field image, generating and feeding back an acceptance report of the communication base station to be detected.

[0066] Among them, the environmental detection instruction may refer to the environmental detection instruction for the communication base station to be detected. The on-site image may refer to the real image of the area to be detected in the communication base station collected. The on-site image includes the target body to be detected, such as the ground. The target classification model may refer to a trained binary classification model. The environmental detection result may refer to the binary classification result for the on-site image. The environmental detection result may include a normal environmental result and an abnormal environmental result. The normal environmental result may mean that the target body does not contain target items that are not allowed to be placed. The abnormal environmental result may mean that the target body contains target items. The acceptance report may refer to the overall detection result of the communication base station. The acceptance report can be used to list the on-site images corresponding to the environmental detection results, and to comprehensively evaluate the overall acceptance qualification of the communication base station.

[0067] Specifically, when an environmental detection instruction is received, the camera device in the communication base station to be detected is called through the environmental detection instruction to obtain the on-site image of the communication base station to be detected; the on-site image is transmitted to the main server that issues the environmental detection instruction, and the on-site image is input into the pre-trained target classification model for environmental detection; based on the output of the target classification model, the environmental detection result of the communication base station to be detected is determined; based on the environmental detection result and the on-site image, the acceptance report of the communication base station to be detected is generated and fed back, thereby reducing the frequency of acceptance personnel going to the on-site communication base station to be detected, saving manpower and material resources, and reducing the risks caused by human factors while improving the efficiency of detection and acceptance and reducing costs. Among them, human factors can refer to different acceptance personnel may have different understandings and judgments on the same acceptance criteria. This subjectivity leads to inconsistency in the acceptance results, and some minor quality problems may be ignored due to negligence in manual inspection, affecting the final project quality.

[0068] Exemplarily, in this embodiment, an online deployment method is provided for implementing remote environment detection operations of communication base stations, namely, designing an application programming interface (API) for remote service call environment detection to further improve the accuracy and security of remote environment detection.

[0069] Among them, in the process of algorithm model development, it is necessary to consider the data transmission between the front-end and back-end of the intelligent quality inspection platform. The lightweight target classification model is integrated into the server in the form of a remote call service (such as an image recognition API) so that the on-site staff can call the remote service interface through the client to perform environmental detection. In order to implement the on-site call model, a remote call service interface is built using message queues (such as ZeroMQ) and protocols (such as JSONRPC). On-site staff can take pictures with their mobile phones or on-site cameras and send the on-site images to the server through the built interface, so that the service calls the target classification model for classification prediction and returns the classification results; if it is "PASS", it means that the item (such as the on-site image) has passed the acceptance, otherwise, it has failed the acceptance.

[0070] Specifically, combined Figure 4 Describes the online deployment method. Figure 4In the process of remote service call, the client and the engine (equivalent to the server) communicate through the ZeroMQ message queue and use the JSONRPC protocol to format the message. The request format is {"jsonrpc":"2.0","method":"foor_AD_detection","params":{"image1":exp.jpg},"id":1}. The response format is {"jsonrpc":"2.0","result":"PASS","id":1}. Among them, the "method" in the request format is the function API interface, and "params" is the input parameter of the function, representing the image to be detected. The detailed workflow of the remote service call is as follows.

[0071] 1) The client first builds a JSONRPC request. JSONRPC is a remote procedure call protocol based on JSON. It defines a simple message format to enable communication between the client and the engine. In this example, the client builds a request message that contains the JSONRPC version number "2.0", a method name "method", which specifies the remote method to be called, that is, the API interface, here refers to "floor_AD_detection". In addition, the request also contains a "params" field, which serves as the input parameter of "floor_AD_detection". Finally, the request also contains a unique request ID "id:1" to match the response with the request.

[0072] 2) The client sends the request message to the engine request queue (Engine-Request) through ZeroMQ's PUSH socket. ZeroMQ is a message queue library that provides a series of socket types to support different messaging modes. PUSH sockets are used to send messages to one or more recipients without having to worry about how the messages are distributed.

[0073] 3) The engine pulls the request message from the request queue through the ZeroMQ PULL socket. The PULL socket is the counterpart of the PUSH socket, which receives messages from the sender. The engine acts as a server and listens to requests from the client.

[0074] 4) The engine processes the request, performs the required operations, and runs the model.

[0075] 5) After the engine completes the processing, it creates a JSONRPC response message. The response message contains the same version number and request ID as the request, as well as a "result" field, which contains the processing result. In this example, the result is "PASS", indicating that the quality inspection item is completed.

[0076] 6) The engine sends the response message to the engine response queue (Engine-Response) through the ZeroMQ PUSH socket. This process is similar to sending a request message, but in the opposite direction.

[0077] 7) The client pulls the response message from the response queue through the ZeroMQ PULL socket. The client monitors the response queue and waits for the engine's response.

[0078] 8) After receiving the response message, the client parses the result in JSONRPC format. The client needs to check whether the request ID in the response message matches the request ID sent to ensure that the response is for a specific request. The client can then perform subsequent processing based on the result, such as updating the webUI interface of the quality inspection platform, storing the result data, etc.

[0079] Embodiment 2

[0080] Figure 5 This is a flowchart of an image generation method provided by Embodiment 2 of the present invention. Based on the above embodiments, this embodiment describes in detail the process of determining the target redrawing area. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. Figure 5 As shown, the method includes:

[0081] S510: Acquire historical sample images.

[0082] Among them, the historical sample image includes the target subject.

[0083] S520 , performing image recognition based on the historical sample image to determine the target main image, and performing mask segmentation based on the target main image to obtain a first mask image.

[0084] S530: Determine, from the first mask image, a plurality of target pixel points whose pixel values ​​are first values.

[0085] The target pixel point may refer to a pixel point selected to meet a preset requirement. The target pixel point may be used to form a target redrawing area. For example, the target pixel point may be used as a center point or an end point of the redrawing area.

[0086] Specifically, a target pixel point having a pixel value of the first value can be determined from the first mask image. A subsequent operation of generating a target redrawing area can be performed based on the number of target pixel points. For example, if one or two target pixel points are determined, each target pixel point is used as a center point to determine the target redrawing area.

[0087] S540: Connect each target pixel point to obtain a target redrawing area in the first mask image.

[0088] Specifically, if the number of target pixel points is at least three, adjacent target pixel points are connected to form a target redrawing area in the first mask image. Figure 6 An example of a target redrawing area is given. The first mask image in this embodiment is different from the first mask image example in the above embodiment. Figure 6 ,The black area contained in the dotted circle in the first mask image is the target redrawing area.

[0089] On the basis of the above technical solution, "connecting each target pixel point to obtain a target redrawing area in the first mask image" may include: connecting each target pixel point to form a candidate redrawing area; if a pixel point with a pixel value of the second value is detected on the boundary of the candidate redrawing area, adjusting the curvature of the boundary where the pixel point is located to obtain the target redrawing area in the first mask image.

[0090] Among them, the candidate redrawing area may refer to an area that needs to be judged whether it meets the preset generation condition. For example, the preset generation condition may be, but is not limited to, that there is no pixel point with a pixel value of the second value in the generation area. The curvature may refer to the degree of curvature of the line segment connecting the target pixel points. For example, the curvature of a straight line is zero, the curvature of the curvature bending inside the candidate redrawing area is a positive value, and the curvature of the curvature bending outside the candidate redrawing area is a negative value. The boundary may refer to the line segment connecting the target pixel points.

[0091] Specifically, each target pixel point can be connected by a straight line or a curve to obtain a closed candidate redrawing area. The connection method of the pixel points may include but is not limited to a straight line and a Bezier curve. In order to ensure that the target redrawing area does not cover the area of ​​the non-target subject, the candidate redrawing area needs to be detected. If it is detected that there are no pixel points with a pixel value of the second value inside and on the boundary of the candidate redrawing area, the candidate redrawing area is determined as the target redrawing area. If it is detected that there are pixel points with a pixel value of the second value inside and / or on the boundary of the candidate redrawing area, the curvature of the boundary is adjusted to the inside of the candidate redrawing and the area, and the detection is performed again to obtain the target redrawing area in the first mask image, thereby ensuring the integrity of the target object image added in the target redrawing area, and further improving the rationality of the augmented sample image with the target object added.

[0092] On the basis of the above technical solution, "adjusting the curvature of the boundary where the pixel points are located to obtain the target redrawing area in the first mask image" may include: determining a boundary segment composed of pixel points whose pixel values ​​in the boundary are the second value and boundary pixel points adjacent to both ends of the boundary segment; using the boundary pixel points as the adjustment base point, adjusting the curvature of the boundary segment toward the inner direction of the candidate redrawing area to obtain the target redrawing area in the first mask image.

[0093] The boundary segment may refer to a boundary segment of the candidate redrawing region in the non-target subject region. The boundary pixel point may refer to a pixel point on the boundary of the candidate redrawing region in the target subject region that is adjacent to the boundary segment. The adjustment base point is fixed.

[0094] Specifically, a boundary segment consisting of pixel points whose pixel values ​​in the boundary are the second value and boundary pixel points adjacent to both ends of the boundary segment are determined; the boundary pixel points are used as adjustment base points, and according to the regional contour of the target subject, the curvature of the boundary segment is adjusted toward the inner direction of the candidate redrawing area so that the boundary segment fits the partial contour of the region of the target subject, and the target redrawing area in the first mask image is obtained, so that the boundary adjustment of the redrawing area can be performed based on the contour of the region of the target subject, thereby further improving the efficiency of determining the target redrawing area.

[0095] S550 , adding an image to the target redrawing area based on the target object image in the object resource pool to obtain a second mask image.

[0096] On the basis of the above technical solution, "generating an image in a target redrawing area based on a target object image in an object resource pool to obtain a second mask image" may include: determining a target object image corresponding to the target object from the object resource pool; generating the target object image in the target redrawing area, and adjusting image attributes of the generated target object image to obtain a second mask image containing the target object.

[0097] The image attributes include: at least one of image size, image rotation angle, image brightness and image saturation.

[0098] Specifically, the target object image corresponding to the target object is determined from the object resource pool; based on the pre-trained target image generation model and generation information, the target object image is added to the target redrawing area, and image attribute adjustment or edge fusion is performed to ensure the integrity of the target object in the target redrawing area, and further improve the rationality of the subsequently generated augmented sample image. For example, the text description can be but is not limited to "this is a photo of the floor, which is {not} clean and tidy, with {clutter} on the floor". The target image generation model can be used to perform corresponding image processing based on the text description.

[0099] S560: Redraw the image based on the historical sample image and the second mask image to obtain an augmented sample image containing the target object image.

[0100] Among them, the augmented sample image is a negative sample image. Figure 7 An example of a contrast image is given. Figure 7 , the four images in the upper part (a1, b1, c1 and d1) are historical sample images, and the four images in the lower part (a2, b2, c2 and d2) are augmented sample images after image augmentation. The items in the dotted circle in each augmented sample image are the added target items.

[0101] The technical solution of the embodiment of the present invention increases the randomness of selecting the target pixel points by determining multiple target pixel points whose pixel values ​​are the first value from the first mask image, and connects the target pixel points to obtain the target redrawing area in the first mask image, thereby improving the randomness of generating the target redrawing area, further improving the diversity of the area where the target object image can be placed, and improving the richness of the augmented sample image.

[0102] The embodiment of the present invention also provides an optional image generation process, which, based on the above-mentioned achievable effects, can also augment the positive sample image to obtain a richer positive sample image and improve the versatility of image generation. The optional image generation process is as follows:

[0103] 1) Based on the sample image generation instruction, the historical sample image to be augmented is obtained. If the sample image generation instruction is a positive sample image generation instruction, the historical positive sample image to be augmented is obtained; if the sample image generation instruction is a negative sample image generation instruction, the historical negative sample image to be augmented is obtained.

[0104] 2) Input the information related to the target subject in the sample image generation instruction and the historical positive sample image into the visual language model to obtain the first mask image after the target subject is identified and binarized. The information related to the target subject can be a prompt text, such as "floor".

[0105] 3) Inputting the generation information in the sample image generation instruction, the first mask image and the historical sample image into the image generation model to generate a new sample image. The image generation model can generate a specified image in a specified area in the first mask image according to the input generation information, and perform mask processing with the input original image (such as the historical sample image) to obtain a new sample image (such as an image with a background transformation, an image with a target subject feature transformation, or an image of a newly generated object in the target subject).

[0106] Specifically, if the sample image generation instruction is a positive sample image generation instruction, the generation information in the positive sample image generation instruction includes: mask image binarization inversion information and background generation information. The generation information in the positive sample image generation instruction, the first mask image and the historical positive sample image are input into the image generation model. In the image generation model, based on the mask image binarization inversion information, the target subject area in the first mask image is set to zero, becoming a non-editable area, and the non-target subject area is set to one, becoming an editable area. Based on the background generation information, a background image related to the background generation information is generated in the editable area of ​​the inverted mask image (such as generating a shopping mall background). The mask image of the generated background image is used to perform mask processing with the historical positive sample image to obtain a new positive sample image.

[0107] Specifically, if the sample image generation instruction is a negative sample image generation instruction, the generation information in the negative sample image generation instruction includes item generation information (such as "this is a photo of the floor, which is {not} clean and tidy, with {detritus} on the floor"). The generation information in the negative sample image generation instruction, the first mask image, and the historical negative sample image are input into the image generation model, and the target redrawing area in the first mask image is determined in the image generation model based on a preset area selection method, and the target item image is determined from the item resource pool based on the item generation information, or the target item image is generated based on the item generation information. The target item image is generated in the target redrawing area to obtain a second mask image, and mask processing is performed based on the historical negative sample image and the second mask image to obtain a new negative sample image containing the target item image.

[0108] Specifically, if the sample image generation instruction is a negative sample image generation instruction, the generation information in the negative sample image generation instruction includes subject transformation information (such as "a clean floor with wood texture"). The generation information in the negative sample image generation instruction, the first mask image, and the historical negative sample image are input into the image generation model, and an image corresponding to the subject transformation information (such as a textured ground) is generated in the area of ​​the target subject in the first mask image based on the subject transformation information in the image generation model, and a new negative sample image with the target subject characteristic transformed is obtained by performing mask processing on the image and the historical negative sample image.

[0109] The following is an embodiment of an image generating device provided by an embodiment of the present invention. The device and the image generating methods of the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the image generating device, reference can be made to the embodiment of the above-mentioned image generating method.

[0110] Embodiment 3

[0111] Figure 8 This is a schematic diagram of the structure of an image generating device provided by Embodiment 3 of the present invention. Figure 8 As shown, the device includes: a historical sample image acquisition module 810, a first mask image determination module 820, a second mask image determination module 830 and an augmented sample image determination module 840.

[0112] Among them, the historical sample image acquisition module 810 is used to acquire the historical sample image; the historical sample image includes the target subject; the first mask image determination module 820 is used to perform image recognition based on the historical sample image, determine the target subject image, and perform mask segmentation based on the target subject image to obtain the first mask image; the second mask image determination module 830 is used to determine the target redrawing area in the first mask image based on a preset area selection method, and generate an image in the target redrawing area based on the target item image in the item resource pool to obtain the second mask image; the augmented sample image determination module 840 is used to perform image redrawing based on the historical sample image and the second mask image to obtain an augmented sample image containing the target item image; wherein the augmented sample image is a negative sample image.

[0113] The technical solution of the embodiment of the present invention is as follows: acquiring a historical sample image; performing image recognition based on the historical sample image to determine a target subject image, and performing mask segmentation based on the target subject image, thereby obtaining a first mask image that effectively distinguishes the target subject in the historical sample image from other subjects different from the target subject; determining a target redrawing area in the first mask image based on a preset area selection method, and performing image generation in the target redrawing area based on a target item image in an item resource pool, thereby obtaining a second mask image that can be used to add a target item image to the historical sample image; performing image redrawing based on the historical sample image and the second mask image to obtain an augmented sample image containing the target item image, thereby achieving effective augmentation processing of the historical sample image to obtain a richer augmented sample image, that is, effectively increasing negative sample images, improving the generation efficiency and accuracy of the augmented sample images, and ensuring the balance in quantity between positive and negative sample images.

[0114] Based on the above technical solution, the “determining the target redrawing area in the first mask image based on the preset area selection method” in the second mask image determination module 830 may include:

[0115] A target pixel point determination submodule, used to determine a plurality of target pixel points whose pixel values ​​are first values ​​from the first mask image;

[0116] The target redrawing area determination submodule is used to connect each target pixel point to obtain the target redrawing area in the first mask image.

[0117] Based on the above technical solution, the target redrawing area determination submodule may include:

[0118] A candidate redrawing area determination unit is used to connect various target pixels to form a candidate redrawing area;

[0119] The target redrawing area determination unit is used to adjust the curvature of the boundary where the pixel point is located if it is detected that there is a pixel point with a second value on the boundary of the candidate redrawing area, so as to obtain the target redrawing area in the first mask image.

[0120] Based on the above technical solution, the target redrawing area determination unit is specifically used to: determine the boundary segment composed of pixel points whose pixel values ​​in the boundary are the second value and the boundary pixel points adjacent to the two ends of the boundary segment; using the boundary pixel points as the adjustment base point, adjust the curvature of the boundary segment toward the internal direction of the candidate redrawing area to obtain the target redrawing area in the first mask image.

[0121] On the basis of the above technical solution, the "generating an image in the target redrawing area based on the target object image in the object resource pool to obtain a second mask image" in the second mask image determination module 830 is specifically used to: determine the target object image corresponding to the target object from the object resource pool; generate the target object image in the target redrawing area, and adjust the image attributes of the generated target object image to obtain a second mask image containing the target object; the image attributes include: at least one of image size, image rotation angle, image brightness and image saturation.

[0122] On the basis of the above technical solution, the "performing image recognition based on historical sample images to determine the target subject image" in the first mask image determination module 820 is specifically used to: input the historical sample image into a pre-trained target subject recognition model; perform target subject recognition on the historical sample image in the target subject recognition model to determine the target subject in the historical sample image; perform image segmentation on the target subject to obtain a target subject image with the same image size as the historical sample image.

[0123] On the basis of the above technical solution, the "mask segmentation based on the target subject image to obtain the first mask image" in the first mask image determination module 820 is specifically used to: adjust the pixel values ​​of the pixel points corresponding to the target subject in the target subject image to the first value, and adjust the pixel values ​​of other pixel points in the target subject image that are different from the target subject to the second value, so as to obtain the first mask image.

[0124] On the basis of the above technical solution, the device also includes:

[0125] A scene image acquisition module, used to acquire a scene image of the communication base station to be detected when receiving an environment detection instruction;

[0126] The environment detection module is used to input the scene image into the pre-trained target classification model for environment detection;

[0127] An environment detection result determination module is used to determine the environment detection result of the communication base station to be detected based on the output of the target classification model;

[0128] The acceptance report generation module is used to generate and feedback the acceptance report of the communication base station to be tested based on the environmental detection results and on-site images.

[0129] The image generating device provided in the embodiment of the present invention can execute the image generating method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the image generating method.

[0130] It is worth noting that in the above-mentioned image generation embodiment, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0131] Embodiment 4

[0132] Fig. 9 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0133] like Fig. 9 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, ROM 12 and RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0134] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0135] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as an image generation method.

[0136] In some embodiments, the image generation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the image generation method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the image generation method in any other appropriate manner (e.g., by means of firmware).

[0137] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0139] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0140] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0141] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0142] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0143] The present application also discloses a computer program product, which includes a computer program, and when the computer program is executed by a processor, the image generation method provided in any embodiment of the present application is implemented. The program product and the image generation method disclosed in each embodiment of the present application belong to the same inventive concept, so it is not described here.

[0144] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0145] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image generation method, characterized in that: include: Acquire a historical sample image; the historical sample image includes a target subject; Performing image recognition based on the historical sample image to determine a target subject image, and performing mask segmentation based on the target subject image to obtain a first mask image; Determine a plurality of target pixel points whose pixel values ​​are the first value from the first mask image; connect the target pixel points to form a candidate redrawing area; if it is detected that there are pixel points whose pixel values ​​are the second value on the boundary of the candidate redrawing area, determine a boundary segment formed by the pixel points whose pixel values ​​are the second value in the boundary and boundary pixel points adjacent to both ends of the boundary segment; use the boundary pixel points as adjustment base points, adjust the curvature of the boundary segment toward the inner direction of the candidate redrawing area, obtain the target redrawing area in the first mask image, and generate an image in the target redrawing area based on the target item image in the item resource pool to obtain a second mask image; Redrawing the image based on the historical sample image and the second mask image to obtain an augmented sample image containing the target object image; Wherein, the augmented sample image is a negative sample image.

2. The method according to claim 1, characterized in that: The step of generating an image based on the target object image in the object resource pool in the target redrawing area to obtain a second mask image includes: Determine a target object image corresponding to the target object from the object resource pool; The target object image is generated in the target redrawing area, and image attributes of the generated target object image are adjusted to obtain a second mask image containing the target object; the image attributes include: at least one of image size, image rotation angle, image brightness and image saturation.

3. The method according to claim 1, characterized in that The performing image recognition based on the historical sample image to determine the target subject image includes: Inputting the historical sample image into a pre-trained target subject recognition model; Performing target subject recognition on the historical sample image in the target subject recognition model to determine the target subject in the historical sample image; Perform image segmentation on the target subject to obtain a target subject image having the same image size as the historical sample image.

4. The method according to claim 1, characterized in that: The performing mask segmentation based on the target subject image to obtain a first mask image includes: The pixel values ​​of the pixels corresponding to the target subject in the target subject image are adjusted to a first value, and the pixel values ​​of other pixels different from the target subject in the target subject image are adjusted to a second value, so as to obtain the first mask image.

5. The method according to claim 1, characterized in that The method further comprises: Upon receiving an environment detection instruction, acquiring a scene image of the communication base station to be detected; Inputting the scene image into a pre-trained target classification model for environment detection; Determining the environmental detection result of the communication base station to be detected based on the output of the target classification model; Based on the environmental detection result and the on-site image, an acceptance report of the communication base station to be detected is generated and fed back.

6. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image generating method as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image generating method according to any one of claims 1 to 5 is implemented.

Images