Image generation method, device and storage medium

Abstract

This application discloses an image generation method, apparatus, and storage medium, belonging to the field of image processing. The method includes: determining multiple mask images based on first annotation information corresponding to multiple sample images respectively; and determining multiple generated images based on the multiple mask images, multiple noise signals, a background image, and a target type through a diffusion model, so that the target product in the multiple generated images has defects and the defect type is the target type.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to an image generation method, apparatus and storage medium. Background Technology

[0002] In the process of building a model, it is necessary to train the model based on a large number of sample images to obtain a model with high accuracy. However, it is often difficult to obtain a large number of sample images that meet the training requirements. For example, in the industrial field, defect detection models are usually used to identify product images to determine whether the product has defects. However, defective products are relatively rare in industrial scenarios, resulting in a small number of images of defective products. Consequently, the number of sample images that can be used to train the defect detection model is too small, leading to poor accuracy in identifying defects. Summary of the Invention

[0003] This application provides an image generation method, apparatus, and storage medium, which can solve the problem of poor image diversity in related technologies. The technical solution is as follows:

[0004] On the one hand, an image generation method is provided, the method comprising:

[0005] Based on the first annotation information corresponding to multiple sample images, multiple mask images are determined. The multiple sample images are all images of the target product. The target product in the multiple sample images has defects and the defect type includes the target type. The first annotation information includes the size and position of the defect area of ​​the target product in the corresponding sample image. The multiple mask images all include regions of interest, and the size and / or position of the regions of interest in the multiple mask images are different.

[0006] Based on the multiple mask images, multiple noise signals, a background image, and the target type, multiple generated images are determined by a diffusion model, such that the target product in the multiple generated images has a defect and the defect type is the target type. The background image is an image of the target product without defects. The regions of interest included in the multiple mask images are used to guide the regions where the defects of the target product are located in the multiple generated images. The multiple noise signals are used to guide the generation of defects in the target product in the multiple generated images.

[0007] In one possible implementation, determining multiple mask images based on the first annotation information corresponding to multiple sample images includes:

[0008] Determine target indication information, which indicates whether a defect of the target type occurs at a random location in the target product;

[0009] Based on the target indication information and the first annotation information corresponding to the multiple sample images, the multiple mask images are determined.

[0010] In one possible implementation, determining the plurality of mask images based on the target indication information and the first annotation information corresponding to the plurality of sample images respectively includes:

[0011] Based on the target indication information, multiple candidate locations are determined;

[0012] Based on the size of the defective region of the target product in the multiple sample images, multiple candidate sizes are determined;

[0013] Based on the multiple candidate positions and the multiple candidate sizes, multiple mask images are generated such that the position and size of the region of interest in any one of the multiple mask images are any one of the multiple candidate positions and the multiple candidate sizes.

[0014] In one possible implementation, the plurality of sample images are of the same size;

[0015] The determination of multiple candidate locations based on the target indication information includes:

[0016] If the target indication information indicates that a defect of the target type appears at a random location on the target product, then any multiple locations in the sample image are determined as the multiple candidate locations; or,

[0017] If the target indication information indicates that the defect of the target type does not appear at a random location on the target product, then the location of the defect area of ​​the target product in the plurality of sample images is determined as the plurality of candidate locations.

[0018] In one possible implementation, determining multiple candidate sizes based on the size of the region where the defect of the target product is located in the multiple sample images includes:

[0019] The dimensions of the defective region of the target product in the plurality of sample images are determined as the plurality of candidate dimensions; or...

[0020] The dimensions of the defective regions of the target product in the plurality of sample images are multiplied by a scaling factor to obtain the plurality of candidate dimensions; or...

[0021] Based on the dimensions of the defective region of the target product in the multiple sample images, a range of size values ​​is determined, and based on the range of size values, multiple candidate sizes are determined.

[0022] In one possible implementation, determining multiple candidate sizes based on the size of the region where the defect of the target product is located in the multiple sample images includes:

[0023] If the target indication information indicates that the defect of the target type does not appear at a random location on the target product, then the size of the area where the defect of the target product is located in the plurality of sample images is determined as the plurality of candidate sizes.

[0024] In one possible implementation, the plurality of candidate dimensions includes a plurality of first dimensions and a plurality of second dimensions;

[0025] The determination of multiple candidate sizes based on the size of the defective region of the target product in the multiple sample images includes:

[0026] If the target indication information indicates that a defect of the target type appears at a random location on the target product, then the dimensions of the defect area of ​​the target product in the plurality of sample images are multiplied by a scaling factor to obtain the plurality of first dimensions; based on the dimensions of the defect area of ​​the target product in the plurality of sample images, a range of dimension values ​​is determined, and based on the range of dimension values, the plurality of second dimensions are determined.

[0027] In one possible implementation, the dimensions of the defective area of ​​the target product include both horizontal and vertical dimensions, and the range of these dimensions includes both horizontal dimension ranges and aspect ratio ranges.

[0028] Determining the size range based on the size of the defective region of the target product in the multiple sample images includes:

[0029] The maximum value of the horizontal dimension of the defect area of ​​the target product in the plurality of sample images is determined as the upper limit of the horizontal dimension range, and the minimum value of the horizontal dimension of the defect area of ​​the target product in the plurality of sample images is determined as the lower limit of the horizontal dimension range.

[0030] The maximum value among the aspect ratios corresponding to the defective area of ​​the target product in the plurality of sample images is determined as the upper limit of the aspect ratio range, and the minimum value among the aspect ratios corresponding to the defective area of ​​the target product in the plurality of sample images is determined as the lower limit of the aspect ratio range. The aspect ratio is the ratio of the horizontal dimension to the vertical dimension.

[0031] In one possible implementation, the method further includes:

[0032] Based on the multiple mask images, the multiple noise signals, the multiple background images, and the target type, the second annotation information corresponding to the multiple generated images is determined by the diffusion model.

[0033] The second annotation information includes the size, location, and defect type of the defect area of ​​the target product in the corresponding generated image.

[0034] In one possible implementation, the diffusion model includes a downsampling network, an intermediate network, and an upsampling network, each of which includes at least one network layer.

[0035] In this network, the size of the input feature map of any layer in the downsampling network is smaller than the size of the output feature map, the size of the input feature map of any layer in the upsampling network is larger than the size of the output feature map, the input feature map of the first layer in the intermediate network is the output feature map of the last layer in the downsampling network, and the output feature map of the last layer in the intermediate network is the input feature map of the first layer in the upsampling network. The size of the feature map includes the size of the feature map in the horizontal direction and the size of the feature map in the vertical direction.

[0036] In one possible implementation, determining the second annotation information corresponding to each of the multiple generated images based on the multiple mask images, the multiple noise signals, the multiple background images, and the target type using the diffusion model includes:

[0037] Based on the feature maps output by multiple network layers, a target affinity feature map is determined. The target affinity feature map contains a second region, the boundary of which is the outline of the defect in the target affinity feature map. The target affinity feature map has the same size as the target generated image. The target generated image is any one of the multiple generated images. The multiple network layers include at least two of the downsampling network, the intermediate network, and the upsampling network.

[0038] The size of the second region is determined as the size of the defect region of the target product in the target generated image, the position of the second region in the target affinity feature map is determined as the position of the defect region of the target product in the target generated image, and the target type is determined as the defect type of the target product in the target generated image.

[0039] In one possible implementation, determining the target affinity feature map based on the feature maps output by multiple network layers includes:

[0040] Image semantic segmentation is performed on the feature maps output by the multiple network layers to obtain multiple pixel affinity feature maps;

[0041] Based on the plurality of pixel affinity feature maps, the similarity between each pixel affinity feature map in the plurality of pixel affinity feature maps and other pixel affinity feature maps in the plurality of pixel affinity feature maps is determined, thereby obtaining multiple similarity values ​​corresponding to each pixel affinity feature map;

[0042] The pixel affinity feature map with the highest target similarity among the multiple pixel affinity feature maps is determined as the target affinity feature map, and the target similarity is the average of the multiple similarities corresponding to the pixel affinity feature map.

[0043] In one possible implementation, the method further includes:

[0044] The generated images are filtered to obtain multiple target images, which are used to train the target defect detection model.

[0045] In one possible implementation, filtering the plurality of generated images to obtain a plurality of target images includes:

[0046] The multiple generated images are sequentially input into the first defect detection model to obtain the first output results corresponding to the multiple generated images output by the first defect detection model. The first output results indicate whether the target product in the corresponding generated image has a defect.

[0047] The generated images in which the first output result indicates that the target product has a defect are identified as multiple candidate generated images.

[0048] Based on the multiple candidate generated images, the multiple target images are determined;

[0049] Wherein, the number of parameters in the first defect detection model is greater than the number of parameters in the target defect detection model, and / or, the accuracy of the parameters in the first defect detection model is higher than the accuracy of the parameters in the target defect detection model, and / or, the number of network layers in the first defect detection model is greater than the number of network layers in the target defect detection model.

[0050] In one possible implementation, determining multiple target images based on the multiple candidate generated images includes:

[0051] The plurality of candidate generated images are sequentially input into the second defect detection model to obtain the second output results corresponding to the plurality of candidate generated images output by the second defect detection model. The second output results indicate whether the target product in the corresponding candidate generated image has a defect. If the second output results indicate that the target product in the corresponding candidate generated image has a defect, the second output results also include the defect type of the defect in the corresponding candidate generated image.

[0052] The generated image in which the second output result indicates that the target product has no defect or the defect type is not the target defect type is determined as the multiple target images;

[0053] Wherein, the number of parameters in the second defect detection model is less than the number of parameters in the target defect detection model, and / or, the accuracy of the parameters in the second defect detection model is lower than the accuracy of the parameters in the target defect detection model, and / or, the number of network layers in the second defect detection model is less than the number of network layers in the target defect detection model.

[0054] On the other hand, an image generation apparatus is provided, the apparatus comprising:

[0055] The first determining module is used to determine multiple mask images based on the first annotation information corresponding to multiple sample images respectively. The multiple sample images are all images of a target product. The target product in the multiple sample images has a defect and the defect type includes the target type. The first annotation information includes the size and position of the area where the defect of the target product is located in the corresponding sample image. The multiple mask images all include regions of interest, and the size and / or position of the regions of interest in the multiple mask images are different.

[0056] The second determining module is used to determine multiple generated images based on the multiple mask images, multiple noise signals, a background image, and the target type using a diffusion model, such that the target product in the multiple generated images has a defect and the defect type is the target type, the background image is an image of the target product without defects, the regions of interest included in the multiple mask images are used to guide the region where the defect of the target product is located in the multiple generated images, and the multiple noise signals are used to guide the target product to generate defects in the multiple generated images.

[0057] On the other hand, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store a computer program and the processor is used to execute the computer program stored in the memory to implement the steps of the image generation method described above.

[0058] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the steps of the image generation method described above.

[0059] On the other hand, a computer program product containing instructions is provided, which, when run on a computer, cause the computer to perform the steps of the image generation method described above.

[0060] The technical solution provided in this application can bring at least the following beneficial effects:

[0061] Since multiple mask images are generated based on the size and location of the defective region of the target product in multiple real sample images, the generated mask images contain regions of interest of different sizes and / or locations. This allows subsequent production images based on the mask images to simulate the target type of defect appearing in different locations and regions. Thus, while increasing the diversity of generated images, their practicality and realism are enhanced, making them more representative of various situations that may be encountered in actual production or inspection. If the diverse generated images are subsequently used as training data to train the defect detection model, it can help the defect detection model learn a wider range of defect features, improving the model's generalization ability. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a flowchart of an image generation method provided in an embodiment of this application;

[0064] Figure 2 This is a schematic diagram of the feature maps output by each network layer in a Unet sub-model provided in an embodiment of this application;

[0065] Figure 3 This is a flowchart of another image generation method provided in the embodiments of this application;

[0066] Figure 4 This is a schematic diagram of the structure of an image generation device provided in an embodiment of this application;

[0067] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0069] Before providing a detailed explanation of the image generation method provided in the embodiments of this application, the terms and application scenarios involved in the embodiments of this application will be introduced first.

[0070] First, the terms used in the embodiments of this application will be introduced.

[0071] Diffusion Model: A diffusion model is a generative model primarily used for tasks such as image generation. Its basic principle stems from the physical concept of diffusion, simulating a gradual denoising process from high-dimensional to low-dimensional data to generate new samples. The diffusion model modifies the data by progressively adding noise and then progressively removing the noise to generate new data. The diffusion model consists of two processes: a forward process and a reverse process, where the forward process is also called the diffusion process. In the forward process, noise is gradually introduced into the data. This process is similar to gradually blurring an image until it becomes white noise. The introduction of noise is controllable and is usually achieved through a pre-defined noise scheduling strategy (such as linear scheduling). The choice of noise scheduling has a significant impact on the model's performance. The reverse process refers to the diffusion model learning how to recover data from noise. This process requires learning a denoising network that can predict what the data would look like before noise was added, based on the current noise data and its level. By iteratively processing this denoising, new samples that resemble the real data distribution can eventually be generated.

[0072] Secondly, the application scenarios involved in the embodiments of this application will be introduced.

[0073] For some products, defects can lead to safety hazards and even threaten consumer safety. For example, automotive parts and circuit boards may have defects that pose safety risks; if not detected and addressed promptly, these defects could cause serious accidents. Taking circuit boards as an example, various types of defects may exist during industrial production, such as broken circuits, missing components, and oil stains on the panel. Broken circuits refer to breaks or discontinuities in the circuitry on the circuit board. These breaks prevent the normal flow of current, affecting the board's function and performance. Missing components refer to the failure or absence of required electronic components during production or assembly. Since missing components may perform specific circuit functions or signal transmission tasks, their absence can lead to incomplete or malfunctioning circuit boards. Oil stains on the panel refer to oily marks on the circuit board surface. The presence of oil not only affects the appearance of the circuit board but can also negatively impact its performance and reliability, such as causing short circuits or reducing insulation performance. Therefore, product defect detection is crucial for ensuring consumer safety.

[0074] Currently, defect detection models are typically used to identify product images and determine whether a product has defects. However, before a defect detection model can detect product defects, it needs to be trained on a large number of sample images to obtain a model with high accuracy. But it is often difficult to obtain a large number of sample images that meet the training requirements. For example, in industrial scenarios, products rarely have defects, resulting in a small number of images of defective products. Consequently, the number of sample images that can be used to train the defect detection model is too small, leading to poor accuracy in identifying defects.

[0075] Related techniques train deep learning models (such as diffusion models) using a small number of sample images, enabling the trained deep learning model to generate a large number of sample images to meet the model's training needs. However, the images generated by the deep learning model have a high similarity to the images used to train the deep learning model, resulting in poor diversity of the generated images.

[0076] Based on this, embodiments of this application provide an image generation method that can determine multiple mask images based on first annotation information corresponding to multiple sample images, and determine multiple generated images based on multiple mask images, multiple noise signals, multiple background images, and target types through a diffusion model. Since the multiple mask images are generated based on the size and location of the defective region of the target product in multiple real sample images, the generated mask images contain regions of interest of different sizes and / or locations. This allows subsequent production images obtained based on the mask images to simulate target-type defects appearing in different locations and regions. Thus, while increasing the diversity of generated images, it also increases their practicality and realism, making them more representative of various situations that may be encountered in actual production or inspection. If the diverse generated images are subsequently used as training data to train a defect detection model, it can help the defect detection model learn a wider range of defect features, improving the model's generalization ability.

[0077] The method provided in this application is executed by an electronic device, which may be a PC (Personal Computer), mobile phone, PDA (Personal Digital Assistant), PPC (Pocket PC), tablet computer, server, etc.

[0078] It should be noted that the application scenarios and execution entities described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the emergence of new application scenarios and the evolution of technology, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0079] The image generation method provided in the embodiments of this application will now be explained in detail.

[0080] Figure 1 This is a flowchart illustrating an image generation method provided in an embodiment of this application, which is applied to an electronic device. Please refer to... Figure 1 The method includes the following steps.

[0081] Step 101: Based on the first annotation information corresponding to the multiple sample images, determine multiple mask images. The multiple sample images are all images of the target product. The target product in the multiple sample images has defects and the defect type includes the target type. The first annotation information includes the size and location of the defect area of ​​the target product in the corresponding sample image. The multiple mask images all include regions of interest. The size and / or location of the regions of interest in the multiple mask images are different.

[0082] It should be noted that the above-mentioned defect types are product-specific; that is, different products may have different types of defects. Taking circuit boards as an example, in the industrial production process, the types of defects in circuit board products include circuit breaks, missing components, and panel oil stains. Circuit breaks refer to the phenomenon of broken or discontinuous lines on the circuit board; missing components refer to the failure or absence of required electronic components during the production or assembly process; and panel oil stains refer to oil stains formed on the surface of the circuit board. Optionally, the circuit board mentioned above can be the circuit board corresponding to a display panel; this application embodiment does not limit this.

[0083] In some embodiments, the electronic device stores multiple sample images in advance, in which case the electronic device can directly acquire the multiple sample images. In other embodiments, before determining multiple mask images based on the first annotation information corresponding to the multiple sample images, the electronic device can receive multiple sample images sent by other devices. This application does not limit the source of the sample images.

[0084] In some embodiments, before determining multiple mask images based on the first annotation information corresponding to multiple sample images respectively, defects in the multiple sample images can be annotated separately to obtain the first annotation information corresponding to each sample image.

[0085] In one possible implementation, for any one of the multiple sample images, defects can be annotated manually to obtain the first annotation information corresponding to that sample image.

[0086] The process of determining multiple mask images based on the first annotation information corresponding to multiple sample images includes: determining target indication information, which indicates whether a defect of the target type appears at a random location on the target product; and determining multiple mask images based on the target indication information and the first annotation information corresponding to multiple sample images.

[0087] It should be noted that when there are multiple defect types corresponding to the target product, each defect type has its own indication information. The indication information is used to describe whether the defect of the corresponding defect type appears at a random location on the target product.

[0088] In some embodiments, the electronic device stores target indication information, in which case the electronic device can directly determine the target indication information. Of course, the electronic device can also receive target indication information sent by other devices, and this application does not limit this.

[0089] In one possible implementation, the target product has multiple defect types, and the electronic device stores indication information corresponding to each of the multiple defect types. In this case, the indication information corresponding to the target type is determined from the indication information corresponding to each of the multiple defect types to obtain the target indication information.

[0090] It is understandable that the above-mentioned indications are obtained by technicians through analysis of sample images corresponding to the respective defect types. For example, the defect types of the circuit board products mentioned above include circuit breaks, missing components, and panel oil stains. Among them, the indication for circuit breaks is that the defect does not appear at random locations on the target product (i.e., it appears at fixed locations on the target product); the indication for missing components is that the defect does not appear at random locations on the target product; and the indication for panel oil stains is that the defect does not appear at random locations on the target product.

[0091] In some embodiments, the process of determining multiple mask images based on target indication information and first annotation information corresponding to multiple sample images includes: determining multiple candidate positions based on target indication information; determining multiple candidate sizes based on the size of the defect area of ​​the target product in the multiple sample images; and generating multiple mask images based on the multiple candidate positions and multiple candidate sizes, such that the position and size of the region of interest of any one of the multiple mask images are any one of the multiple candidate positions and multiple candidate sizes.

[0092] In one possible implementation, the pixel value of the region of interest in the mask image is 0, and the pixel value of the region outside the region of interest in the mask image is 1.

[0093] In one possible implementation, multiple sample images are of the same size. If the target indication information indicates that a defect of the target type appears at a random location on the target product, then any multiple locations in the sample images are determined as multiple candidate locations.

[0094] Since the defects of the target type are located at random positions in the target product, multiple arbitrary positions in the sample image can be directly determined as candidate positions, thereby enhancing the flexibility of mask image generation while ensuring that the generated mask image conforms to reality.

[0095] In another possible implementation, if the target indication information indicates that the defect of the target type does not appear at a random location on the target product, then the location of the defect area of ​​the target product in multiple sample images is determined as multiple candidate locations.

[0096] Since the defects of the target type do not appear at random locations on the target product, it is necessary to refer to the location of the defects in the real sample image to determine the location of the region of interest in the mask image, so as to ensure that the area where the defects are located in the generated image obtained based on the region of interest matches the actual situation.

[0097] There are several ways to determine multiple candidate sizes based on the size of the defect area of ​​the target product in multiple sample images. The following sections will introduce three of these methods.

[0098] The first method involves determining the dimensions of the defective area of ​​the target product in multiple sample images as multiple candidate dimensions.

[0099] In some embodiments, if the target indication information indicates that the defect of the target type does not appear at a random location on the target product, the size of the area where the defect of the target product is located in multiple sample images is determined as multiple candidate sizes.

[0100] If the defect of the target type is not located at a random position on the target product, it indicates that the location of the defect has a certain regularity or fixed pattern. In this case, directly determining the candidate size based on the size of the area where the defect of the target product is located in multiple sample images can ensure that the generated mask image is more accurately targeted at these fixed or regular defect locations, thereby effectively improving the targeting and practicality of the mask image.

[0101] The second method involves multiplying the size of the defective area of ​​the target product in multiple sample images by a scaling factor to obtain multiple candidate sizes.

[0102] By multiplying the size of the target product defect region in multiple sample images by different scaling factors, mask images of regions of interest with different sizes can be obtained. This ensures that the generated images obtained from the mask images can more flexibly simulate defects of different sizes, increasing the diversity and realism of the generated images. When training the diffusion model with generated images, using generated images with different defect sizes can help the model learn richer feature representations, thereby improving the model's generalization ability to different defect sizes.

[0103] The third approach involves determining the size range based on the dimensions of the defective area of ​​the target product in multiple sample images, and then determining multiple candidate sizes based on the size range.

[0104] In some embodiments, the dimensions of the defective region of the target product include both horizontal and vertical dimensions, and the range of dimensions includes both horizontal dimension range and aspect ratio range. In this case, the maximum horizontal dimension among the multiple sample images of the defective region of the target product is determined as the upper limit of the horizontal dimension range, and the minimum horizontal dimension among the multiple sample images of the defective region of the target product is determined as the lower limit of the horizontal dimension range. The maximum aspect ratio among the multiple sample images of the defective region of the target product is determined as the upper limit of the aspect ratio range, and the minimum aspect ratio among the multiple sample images of the defective region of the target product is determined as the lower limit of the aspect ratio range. The aspect ratio is the ratio of the horizontal dimension to the vertical dimension.

[0105] For each of the multiple candidate dimensions, the candidate dimension includes a horizontal dimension and a vertical dimension. In this case, the process of determining the candidate dimension includes: determining any value in the range of horizontal dimension values ​​as the horizontal dimension included in the candidate dimension; determining any aspect ratio in the range of aspect ratio values ​​as the target aspect ratio; and determining the vertical dimension included in the candidate dimension based on the target aspect ratio and the horizontal dimension included in the candidate dimension.

[0106] In some embodiments, the aspect ratio is the horizontal dimension divided by the vertical dimension. In this case, the value of dividing the horizontal dimension included in the candidate dimension by the target aspect ratio is determined as the vertical dimension included in the candidate dimension.

[0107] In one possible implementation, the multiple candidate sizes include multiple first sizes and multiple second sizes. In this case, the process of determining multiple candidate sizes based on the size of the defect area of ​​the target product in multiple sample images includes: if the target indication information indicates that a defect of the target type appears at a random location on the target product, then multiply the size of the defect area of ​​the target product in multiple sample images by a scaling factor to obtain multiple first sizes; determine the size value range based on the size of the defect area of ​​the target product in multiple sample images; and determine multiple second sizes based on the size value range.

[0108] In other words, when multiple candidate dimensions include multiple first dimensions and multiple second dimensions, and the target indication information indicates that a defect of the target type occurs at a random location on the target product, multiple first dimensions are determined by the second method described above, and multiple second dimensions are determined by the third method described above.

[0109] It should be noted that the number of the multiple first dimensions and the number of the multiple second dimensions may be equal or unequal, and this application embodiment does not limit this. For example, the number of multiple candidate dimensions may be 1000, in which case the number of multiple first dimensions and the number of multiple second dimensions may both be 500.

[0110] Step 102: Based on multiple mask images, multiple noise signals, a background image, and a target type, multiple generated images are determined using a diffusion model, such that the target product in the multiple generated images has a defect and the defect type is the target type, the background image is an image of the target product without defects, the regions of interest included in the multiple mask images are used to guide the regions where the defects of the target product are located in the multiple generated images, and the multiple noise signals are used to guide the generation of defects in the target product in the multiple generated images.

[0111] It should be noted that the multiple mask images, background images, and sample images mentioned above all have the same size. This size includes both the horizontal and vertical dimensions. For example, the horizontal dimension can be the width of the image, and the vertical dimension can be the height of the image. For instance, the size of each of the multiple mask images, background images, and sample images could be 512*512.

[0112] In some embodiments, there may be one or more background images, and this application does not limit this.

[0113] In one possible implementation, when there are multiple background images, the shooting angles of the multiple background images are different. In another possible implementation, the target product is a circuit board. When there are multiple background images, if the target indication information indicates that a defect of the target type appears at a random location on the target product, the circuit boards in the multiple background images can be the same or different. This application embodiment does not limit this. It should be noted that different circuit boards refer to different circuit board traces and / or different specifications of components on the circuit board and / or different positions of components on the circuit board.

[0114] When there are multiple background images, if the target indication information indicates that the defect of the target type does not appear at the location of the target product, and the target type is a circuit break, then the circuit board traces in the multiple background images are the same, while the specifications and positions of the components on the circuit board in the multiple background images can be the same or different. If the target indication information indicates that the defect of the target type does not appear at the location of the target product, and the target type is a missing component, then the positions of the components on the circuit board in the multiple background images are the same, while the specifications of the components on the circuit board and the circuit board traces in the multiple background images can be the same or different.

[0115] In some embodiments, a single generated image can be obtained by inputting any one of multiple mask images, any one of multiple noise signals, any one of multiple background images, and a target type into the diffusion model. Alternatively, multiple mask images, multiple noise signals, background images, and a target type can be input into the diffusion model to obtain multiple generated images. This application does not limit this approach.

[0116] In some embodiments, the diffusion model includes a downsampling network, an intermediate network, and an upsampling network, each of which includes at least one network layer.

[0117] In some embodiments, the electronic device can also determine second annotation information corresponding to multiple generated images based on multiple mask images, multiple noise signals, multiple background images, and target types, using a diffusion model. The second annotation information includes the size, location, and defect type of the defective region of the target product in the corresponding generated image.

[0118] In some embodiments, the second annotation information corresponding to a generated image output by the diffusion model can be obtained by inputting any one of multiple mask images, any one of multiple noise signals, any one of multiple background images, and the target type into the diffusion model. Alternatively, multiple mask images, multiple noise signals, background images, and the target type can be input into the diffusion model to obtain second annotation information corresponding to multiple generated images output by the diffusion model. This application does not limit this approach.

[0119] It should be noted that the second annotation information corresponding to the multiple generated images and the multiple generated images themselves can be output simultaneously by the diffusion model or output separately by the diffusion model; this application embodiment does not limit this. The second annotation information corresponding to the multiple generated images and the multiple generated images being output simultaneously by the diffusion model means that, based on multiple mask images, multiple noise signals, multiple background images, and target types, the diffusion model determines the multiple generated images and the second annotation information corresponding to each of the multiple generated images.

[0120] In one possible implementation, the diffusion model includes a downsampling network, an intermediate network, and an upsampling network, each of which includes at least one network layer. Specifically, the size of the input feature map of any layer in the downsampling network is smaller than the size of the output feature map, the size of the input feature map of any layer in the upsampling network is larger than the size of the output feature map, the input feature map of the first layer in the intermediate network is the output feature map of the last layer in the downsampling network, and the output feature map of the last layer in the intermediate network is the input feature map of the first layer in the upsampling network. The size of the feature map includes both its horizontal and vertical dimensions.

[0121] In this scenario, a target affinity feature map can be determined based on the feature maps output by multiple network layers. A second region exists within the target affinity feature map, and the boundary of this second region is the outline of the defect in the target affinity feature map. The target affinity feature map has the same size as the target generated image, which is any one of multiple generated images. The multiple network layers include at least two of the following: downsampling networks, intermediate networks, and upsampling networks. The size of the second region is determined as the size of the defect region in the target generated image, the position of the second region in the target affinity feature map is determined as the position of the defect region in the target generated image, and the target type is determined as the defect type of the target product in the target generated image.

[0122] In some embodiments, the plurality of network layers include network layers in an intermediate network, and the plurality of network layers also include network layers in at least one of a downsampled network and an upsampled network.

[0123] In one possible implementation, the diffusion model includes a Unet sub-model, which comprises a downsampling network, an intermediate network, and an upsampling network. For an example, please refer to... Figure 2 , Figure 2 This is a schematic diagram of the feature maps output by each network layer in a Unet sub-model provided in an embodiment of this application, wherein the feature maps output by the network layers in the downsampling network, intermediate network, and upsampling network are all in... Figure 2 The following is an illustration.

[0124] The process of determining the target affinity feature map based on the feature maps output by multiple network layers includes: performing image semantic segmentation on the feature maps output by multiple network layers to obtain multiple pixel affinity feature maps; determining the similarity between each pixel affinity feature map and other pixel affinity feature maps in the multiple pixel affinity feature maps, obtaining multiple similarities corresponding to each pixel affinity feature map; and determining the pixel affinity feature map with the highest target similarity in the multiple pixel affinity feature maps as the target affinity feature map, where the target similarity is the average of the multiple similarities corresponding to the pixel affinity feature map.

[0125] In one possible implementation, the process of performing image semantic segmentation on the feature maps output by multiple network layers to obtain multiple pixel affinity feature maps includes: determining an average feature map based on the feature maps output by multiple network layers; and performing image semantic segmentation on the average feature map based on multiple binarization thresholds to obtain multiple pixel affinity feature maps.

[0126] The process of determining the average feature map based on the feature maps output by multiple network layers includes: determining multiple target feature maps based on the feature maps output by multiple network layers, wherein the multiple target feature maps have the same spatial dimension; and determining the average feature map based on the multiple target feature maps.

[0127] In some embodiments, the feature maps output by the network layers in the downsampling network, intermediate network, and upsampling network have different dimensions. The dimensions of the feature maps include spatial dimensions and channel dimensions. The spatial dimensions include dimensions in the horizontal direction and dimensions in the vertical direction.

[0128] In some embodiments, if the feature maps output by multiple network layers have different spatial dimensions, before determining multiple target feature maps, the feature maps output by multiple network layers can be corrected to obtain corrected feature maps. The horizontal dimension of the corrected feature maps is the same as the horizontal dimension of the sample image, and the vertical dimension of the corrected feature maps is the same as the vertical dimension of the sample image.

[0129] For example, the resize function can be used to correct the feature maps output by multiple network layers separately.

[0130] In some embodiments, for each feature map output by multiple network layers, the feature map corresponding to any one of the multiple channels included in the feature map is determined as the target feature map.

[0131] In other embodiments, the feature map includes H channels with a spatial dimension of W*W. Each element in the feature map has corresponding coordinates (x, y, h), where x is the horizontal position of the element, y is the vertical position, and h is the channel containing the element. x and y each have W possible values, and h has H possible values. In this case, the average value of the elements in the feature map that have the same x and y coordinates is determined as the element value at the corresponding x and y coordinates in the target feature map, thus obtaining the target feature map.

[0132] The process of determining the average feature map based on multiple target feature maps includes: for the target position in the average feature map, the element values ​​of the elements located at the target position in the multiple target feature maps are added together, and the sum is divided by the number of multiple target feature maps to obtain the element value of the element located at the target position in the average feature map.

[0133] The process of performing image semantic segmentation on the average feature map based on multiple binarization thresholds to obtain multiple pixel affinity feature maps includes: for any target binarization threshold among the multiple binarization thresholds, the average feature map is binarized using the DenseCRF algorithm and the numerical and spatial positional relationships between pixels on the feature map are calculated to obtain the pixel affinity feature map corresponding to the target binarization threshold, which can also be called the defect segmentation annotation mask.

[0134] For the specific computation process of the DenseCRF algorithm, please refer to relevant technologies; this application embodiment will not elaborate on it. It should also be noted that other algorithms can also be used to achieve image semantic segmentation in this application embodiment, and this application embodiment is not limited to these.

[0135] The binarization thresholds are pre-set and can be adjusted as needed under different circumstances. For example, these binarization thresholds can be 0.25, 0.3, 0.35, 0.4, and 0.45.

[0136] Based on multiple pixel affinity feature maps, the process of determining the similarity between each pixel affinity feature map and other pixel affinity feature maps in the multiple pixel affinity feature maps, and obtaining multiple similarities corresponding to each pixel affinity feature map, includes:

[0137] For each pixel affinity feature map in the multiple pixel affinity feature maps, based on the correlation filter tracking algorithm, the similarity between the pixel affinity feature map and each pixel affinity feature map in the multiple pixel affinity feature maps other than the pixel affinity feature map is determined, and multiple similarity values ​​corresponding to the pixel affinity feature map are obtained.

[0138] In some embodiments, the electronic device can also filter multiple generated images to obtain multiple target images; wherein the multiple target images are used to train a target defect detection model.

[0139] Multiple generated images are sequentially input into a first defect detection model to obtain first output results corresponding to the multiple generated images output by the first defect detection model. The first output results indicate whether the target product in the corresponding generated image has a defect. The generated images in which the first output results indicate that the target product in the corresponding generated image has a defect are identified as multiple candidate generated images. Based on the multiple candidate generated images, multiple target images are identified. The performance of the first defect detection model is higher than that of the target defect detection model.

[0140] In some embodiments, the performance of the first defect detection model is higher than that of the target defect detection model, including: the number of parameters in the first defect detection model is greater than the number of parameters in the target defect detection model, and / or the accuracy of the parameters in the first defect detection model is higher than the accuracy of the parameters in the target defect detection model, and / or the number of network layers in the first defect detection model is greater than the number of network layers in the target defect detection model.

[0141] In some embodiments, the first defect detection model described above can be a model based on the Transformer architecture or a model based on the YOLOv8-L architecture; this application does not limit this.

[0142] In one possible implementation, multiple candidate generated images can be directly identified as multiple target images. In another possible implementation, multiple candidate generated images are sequentially input into a second defect detection model to obtain second output results corresponding to each candidate generated image. These second output results indicate whether a defect exists in the target product within the corresponding candidate generated image. If the second output result indicates a defect, it also includes the defect type of the defect in the candidate generated image. The generated images from the multiple candidate generated images whose second output results indicate that the target product in the corresponding generated image does not have a defect, or that the defect type is not the target defect type, are identified as multiple target images. In this implementation, the performance of the second defect detection model is lower than that of the target defect detection model.

[0143] In some embodiments, the performance of the second defect detection model is lower than that of the target defect detection model, including: the number of parameters in the second defect detection model is less than the number of parameters in the target defect detection model, and / or the accuracy of the parameters in the second defect detection model is lower than the accuracy of the parameters in the target defect detection model, and / or the number of network layers in the second defect detection model is less than the number of network layers in the target defect detection model.

[0144] In some embodiments, the second defect detection model described above may be a model based on the YOLOv8-S architecture.

[0145] In this embodiment, the second output result indicates that the generated image in which the target product has no defects is an image that the low-performance model cannot recognize. The second output result indicates that the generated image in which the defect type is not the target defect type is an image that the low-performance model misidentifies. Images that a high-performance model can recognize, but the low-performance model cannot recognize or misidentifies, typically contain complex or subtle defect features that are difficult for the low-performance model to identify. Using these images as target images to train the target defect model forces the model to learn these hard-to-capture features, thereby specifically improving the model's ability to identify these difficult features. Furthermore, using images that a high-performance model can recognize but a low-performance model cannot recognize as training data for the target defect model helps to overcome the model's performance bottleneck, achieve more accurate defect detection, and thus rapidly improve the performance of the target defect detection model.

[0146] In some embodiments, the process of training a target defect detection model using multiple target images includes: using multiple target images and multiple sample images as training data to train the target defect detection model.

[0147] It should be noted that when the target product corresponds to multiple defect types, the generated images and sample images corresponding to each defect type can be used as training data to train the target defect detection model.

[0148] The image generation method provided in the embodiments of this application will be described again below.

[0149] Please refer to Figure 3 , Figure 3The flowchart illustrates another image generation method provided in this application embodiment. Based on first annotation information corresponding to multiple sample images, multiple mask images are determined. Based on the multiple mask images, multiple noise signals, a background image, and a target type, multiple generated images are determined using a diffusion model. An average feature map is determined from the feature maps output by multiple network layers. Image semantic segmentation is performed on the average feature map based on multiple binarization thresholds to obtain multiple pixel affinity feature maps. Based on the multiple pixel affinity feature maps, the similarity between each pixel affinity feature map and other pixel affinity feature maps in the multiple pixel affinity feature maps is determined, resulting in multiple similarities corresponding to each pixel affinity feature map. The pixel affinity feature map with the highest target similarity in the multiple pixel affinity feature maps is determined as the target affinity feature map. The size of the second region is determined as the size of the defect region of the target product in the target generated image. The position of the second region in the target affinity feature map is determined as the position of the defect region of the target product in the target generated image. The target type is determined as the defect type of the target product in the target generated image. Multiple generated images are sequentially input into a first defect detection model to obtain first output results corresponding to the multiple generated images output by the first defect detection model. The generated images whose first output results indicate that the target product in the corresponding generated image has a defect are identified as multiple candidate generated images. These multiple candidate generated images are sequentially input into a second defect detection model to obtain second output results corresponding to the multiple candidate generated images output by the second defect detection model. The generated images whose second output results indicate that the target product in the corresponding generated image does not have a defect, or that the defect type is not the target defect type, are identified as multiple target images. These multiple target images and multiple sample images are used as training data to train the target defect detection model.

[0150] In this embodiment, since multiple mask images are generated based on the size and location of the defective region of the target product in multiple real sample images, the generated mask images contain regions of interest of different sizes and / or locations. This allows the subsequent production images obtained based on the mask images to simulate the target type of defect appearing in different locations and regions. Thus, while increasing the diversity of the generated images, their practicality and realism are also enhanced, making them more representative of various situations that may be encountered in actual production or inspection. If the diverse generated images are subsequently used as training data to train the defect detection model, it can help the defect detection model learn a wider range of defect features, improving the model's generalization ability.

[0151] Figure 4This is a schematic diagram of an image generation device provided in an embodiment of this application. This image generation device can be implemented as part or all of the aforementioned electronic device by software, hardware, or a combination of both. Please refer to... Figure 4 The device includes: a first determining module 401 and a second determining module 402.

[0152] The first determining module 401 is used to determine multiple mask images based on the first annotation information corresponding to multiple sample images respectively. The multiple sample images are all images of the target product. The target product in the multiple sample images has defects and the defect type includes the target type. The first annotation information includes the size and position of the defect area of ​​the target product in the corresponding sample image. The multiple mask images all include regions of interest. The size and / or position of the regions of interest in the multiple mask images are different.

[0153] The second determining module 402 is used to determine multiple generated images based on multiple mask images, multiple noise signals, a background image, and a target type through a diffusion model, so that the target product in the multiple generated images has a defect and the defect type is the target type, the background image is an image of the target product without defects, the regions of interest included in the multiple mask images are used to guide the region where the defect of the target product is located in the multiple generated images, and the multiple noise signals are used to guide the target product in the multiple generated images to generate defects.

[0154] In one possible implementation, the first determining module 401 is specifically used for:

[0155] Determine the target indication information, which indicates whether a defect of the target type occurs at a random location on the target product;

[0156] Multiple mask images are determined based on the target indication information and the first annotation information corresponding to multiple sample images.

[0157] In one possible implementation, the first determining module 401 is specifically used for:

[0158] Based on the target indication information, multiple candidate locations were identified;

[0159] Based on the size of the defective region of the target product in multiple sample images, multiple candidate sizes are determined.

[0160] Multiple mask images are generated based on multiple candidate locations and multiple candidate sizes, such that the location and size of the region of interest in any one of the multiple mask images are any one of the multiple candidate locations and multiple candidate sizes.

[0161] In one possible implementation, multiple sample images are of the same size;

[0162] The first determining module is specifically used for

[0163] If the target indication information indicates that a defect of the target type appears at a random location on the target product, then multiple arbitrary locations in the sample image are identified as multiple candidate locations; or,

[0164] If the target indication information indicates that the defect of the target type does not appear at a random location on the target product, then the location of the defect area of ​​the target product in multiple sample images is determined as multiple candidate locations.

[0165] In one possible implementation, the first determining module 401 is specifically used for:

[0166] The dimensions of the defective region of the target product in multiple sample images are determined as multiple candidate dimensions; or...

[0167] Multiple candidate sizes are obtained by multiplying the dimensions of the defective regions of the target product in multiple sample images by a scaling factor; or...

[0168] Based on the dimensions of the defective areas of the target product in multiple sample images, the range of dimensions is determined, and based on the range of dimensions, multiple candidate dimensions are determined.

[0169] In one possible implementation, the first determining module 401 is specifically used for:

[0170] If the target indication information indicates that the defect of the target type does not appear at a random location on the target product, then the size of the area where the defect of the target product is located in multiple sample images is determined as multiple candidate sizes.

[0171] In one possible implementation, the multiple candidate dimensions include multiple first dimensions and multiple second dimensions;

[0172] The first determining module 401 is specifically used for:

[0173] If the target indication information indicates that a defect of the target type appears at a random location on the target product, then the dimensions of the defect area of ​​the target product in multiple sample images are multiplied by a scaling factor to obtain multiple first dimensions; based on the dimensions of the defect area of ​​the target product in multiple sample images, the range of dimension values ​​is determined, and based on the range of dimension values, multiple second dimensions are determined.

[0174] In one possible implementation, the dimensions of the area where the defect is located in the target product include both horizontal and vertical dimensions, and the range of dimensions includes both horizontal dimension range and aspect ratio range.

[0175] The first determining module 401 is specifically used for:

[0176] The maximum value of the horizontal dimension of the defect area of ​​the target product in multiple sample images is determined as the upper limit of the horizontal dimension range, and the minimum value of the horizontal dimension of the defect area of ​​the target product in multiple sample images is determined as the lower limit of the horizontal dimension range.

[0177] The maximum aspect ratio of the defective area of ​​the target product in multiple sample images is determined as the upper limit of the aspect ratio range, and the minimum aspect ratio of the defective area of ​​the target product in multiple sample images is determined as the lower limit of the aspect ratio range. The aspect ratio is the ratio of the horizontal dimension to the vertical dimension.

[0178] In one possible implementation, the device further includes:

[0179] The third determination module is used to determine the second annotation information corresponding to the multiple generated images based on multiple mask images, multiple noise signals, multiple background images and target types, using a diffusion model.

[0180] The second annotation information includes the size, location, and defect type of the defective area of ​​the target product in the corresponding generated image.

[0181] In one possible implementation, the diffusion model includes a downsampling network, an intermediate network, and an upsampling network, each of which includes at least one network layer.

[0182] In this network, the size of the input feature map of any layer in the downsampling network is smaller than the size of the output feature map, the size of the input feature map of any layer in the upsampling network is larger than the size of the output feature map, the input feature map of the first layer in the intermediate network is the output feature map of the last layer in the downsampling network, and the output feature map of the last layer in the intermediate network is the input feature map of the first layer in the upsampling network. The size of the feature map includes the size of the feature map in the horizontal direction and the size of the feature map in the vertical direction.

[0183] In one possible implementation, the third determining module is specifically used for:

[0184] Based on the feature maps output by multiple network layers, a target affinity feature map is determined. A second region exists in the target affinity feature map, and the boundary of the second region is the outline of the defect in the target affinity feature map. The target affinity feature map and the target generated image have the same size. The target generated image is any one of the multiple generated images. The multiple network layers include at least two of the network layers in the downsampling network, intermediate network, and upsampling network.

[0185] The size of the second region is determined as the size of the defect region of the target product in the target generated image. The position of the second region in the target affinity feature map is determined as the position of the defect region of the target product in the target generated image. The target type is determined as the defect type of the target product in the target generated image.

[0186] In one possible implementation, the third determining module is specifically used for:

[0187] Image semantic segmentation is performed on the feature maps output by multiple network layers to obtain multiple pixel affinity feature maps;

[0188] Based on multiple pixel affinity feature maps, the similarity between each pixel affinity feature map in the multiple pixel affinity feature maps and other pixel affinity feature maps in the multiple pixel affinity feature maps is determined, and multiple similarity values ​​corresponding to each pixel affinity feature map are obtained;

[0189] The pixel affinity feature map with the highest target similarity among multiple pixel affinity feature maps is determined as the target affinity feature map, and the target similarity is the average of the multiple similarities corresponding to the pixel affinity feature map.

[0190] In one possible implementation, the device further includes:

[0191] The filtering module is used to filter multiple generated images to obtain multiple target images, which are then used to train the target defect detection model.

[0192] In one possible implementation, the filtering module is specifically used for:

[0193] Multiple generated images are sequentially input into the first defect detection model to obtain the first output results corresponding to the multiple generated images output by the first defect detection model. The first output results indicate whether there is a defect in the target product in the corresponding generated image.

[0194] Among the multiple generated images, the generated image whose first output result indicates that the target product in the corresponding generated image has a defect is identified as multiple candidate generated images;

[0195] Multiple target images are determined based on multiple candidate generated images;

[0196] Wherein, the number of parameters in the first defect detection model is greater than the number of parameters in the target defect detection model, and / or, the accuracy of the parameters in the first defect detection model is higher than the accuracy of the parameters in the target defect detection model, and / or, the number of network layers in the first defect detection model is greater than the number of network layers in the target defect detection model.

[0197] In one possible implementation, the filtering module is specifically used for:

[0198] Multiple candidate generated images are sequentially input into the second defect detection model to obtain the second output results corresponding to the multiple candidate generated images output by the second defect detection model. The second output results indicate whether the target product in the corresponding candidate generated image has a defect. If the second output results indicate that the target product in the corresponding candidate generated image has a defect, the second output results also include the defect type of the defect in the corresponding candidate generated image.

[0199] Among multiple candidate generated images, the generated images whose second output result indicates that the target product in the corresponding generated image does not have a defect, or whose defect type is not the target defect type, are determined as multiple target images;

[0200] Specifically, the number of parameters in the second defect detection model is less than the number of parameters in the target defect detection model, and / or the accuracy of the parameters in the second defect detection model is lower than the accuracy of the parameters in the target defect detection model, and / or the number of network layers in the second defect detection model is less than the number of network layers in the target defect detection model.

[0201] In this embodiment, since multiple mask images are generated based on the size and location of the defective region of the target product in multiple real sample images, the generated mask images contain regions of interest of different sizes and / or locations. This allows the subsequent production images obtained based on the mask images to simulate the target type of defect appearing in different locations and regions. Thus, while increasing the diversity of the generated images, their practicality and realism are also enhanced, making them more representative of various situations that may be encountered in actual production or inspection. If the diverse generated images are subsequently used as training data to train the defect detection model, it can help the defect detection model learn a wider range of defect features, improving the model's generalization ability.

[0202] It should be noted that the image generation apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when generating images. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the image generation apparatus and the image generation method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0203] Figure 5This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 500 includes a central processing unit (CPU) 501, a system memory 504 including random access memory (RAM) 502 and read-only memory (ROM) 503, and a system bus 505 connecting the system memory 504 and the CPU 501. The electronic device 500 also includes a basic input / output system (I / O system) 506 that facilitates information transfer between various devices within the computer, and a mass storage device 507 for storing the operating system 513, application programs 514, and other program modules 515.

[0204] The basic input / output system 506 includes a display 508 for displaying information and an input device 509 for user input, such as a mouse or keyboard. Both the display 508 and the input device 509 are connected to the central processing unit 501 via an input / output controller 510 connected to the system bus 505. The basic input / output system 506 may also include the input / output controller 510 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 510 also provides output to a display screen, printer, or other types of output devices.

[0205] Mass storage device 507 is connected to central processing unit 501 via a mass storage controller (not shown) connected to system bus 505. Mass storage device 507 and its associated computer-readable media provide non-volatile storage for electronic device 500. That is, mass storage device 507 may include computer-readable media (not shown) such as hard disk or CD-ROM drive.

[0206] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technologies, CD-ROM, DVD or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 504 and mass storage device 507 described above can be collectively referred to as memory.

[0207] According to various embodiments of this application, the electronic device 500 can also be connected to a remote computer on a network, such as the Internet. That is, the electronic device 500 can be connected to a network 512 via a network interface unit 511 connected to the system bus 505, or the network interface unit 511 can be used to connect to other types of networks or remote computer systems (not shown).

[0208] The aforementioned memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU.

[0209] In some embodiments, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of the image generation method described above. For example, the computer-readable storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0210] It is worth noting that the computer-readable storage medium mentioned in the embodiments of this application can be a non-volatile storage medium, in other words, it can be a non-transient storage medium.

[0211] It should be understood that all or part of the steps of the above embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially in the form of a computer program product. The computer program product includes one or more computer instructions. The computer instructions can be stored in the above-described computer-readable storage medium.

[0212] That is, in some embodiments, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the steps of the image generation method described above.

[0213] It should be understood that "at least one" as mentioned herein refers to one or more, and "multiple" refers to two or more. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, in order to clearly describe the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and the terms "first," "second," etc., are not necessarily different.

[0214] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the sample images, background images, etc. involved in the embodiments of this application were obtained with full authorization.

[0215] The above descriptions are embodiments provided in this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image generation method, characterized in that, The method includes: Based on the first annotation information corresponding to multiple sample images, multiple mask images are determined. The multiple sample images are all images of the target product. The target product in the multiple sample images has defects and the defect type includes the target type. The first annotation information includes the size and position of the defect area of ​​the target product in the corresponding sample image. The multiple mask images all include regions of interest, and the size and / or position of the regions of interest in the multiple mask images are different. Based on the multiple mask images, multiple noise signals, a background image, and the target type, multiple generated images are determined by a diffusion model, such that the target product in the multiple generated images has a defect and the defect type is the target type. The background image is an image of the target product without defects. The regions of interest included in the multiple mask images are used to guide the regions where the defects of the target product are located in the multiple generated images. The multiple noise signals are used to guide the generation of defects in the target product in the multiple generated images.

2. The method as described in claim 1, characterized in that, The determination of multiple mask images based on the first annotation information corresponding to multiple sample images includes: Determine target indication information, which indicates whether a defect of the target type occurs at a random location in the target product; Based on the target indication information and the first annotation information corresponding to the multiple sample images, the multiple mask images are determined.

3. The method as described in claim 2, characterized in that, The step of determining the plurality of mask images based on the target indication information and the first annotation information corresponding to the plurality of sample images includes: Based on the target indication information, multiple candidate locations are determined; Based on the size of the defective region of the target product in the multiple sample images, multiple candidate sizes are determined; Based on the multiple candidate positions and the multiple candidate sizes, multiple mask images are generated such that the position and size of the region of interest in any one of the multiple mask images are any one of the multiple candidate positions and the multiple candidate sizes.

4. The method as described in claim 3, characterized in that, The multiple sample images are of the same size; The determination of multiple candidate locations based on the target indication information includes: If the target indication information indicates that a defect of the target type appears at a random location on the target product, then any multiple locations in the sample image are determined as the multiple candidate locations; or, If the target indication information indicates that the defect of the target type does not appear at a random location on the target product, then the location of the defect area of ​​the target product in the plurality of sample images is determined as the plurality of candidate locations.

5. The method as described in claim 3, characterized in that, The determination of multiple candidate sizes based on the size of the defective region of the target product in the multiple sample images includes: The dimensions of the defective region of the target product in the plurality of sample images are determined as the plurality of candidate dimensions; or... The dimensions of the defective regions of the target product in the plurality of sample images are multiplied by a scaling factor to obtain the plurality of candidate dimensions; or... Based on the dimensions of the defective region of the target product in the multiple sample images, a range of size values ​​is determined, and based on the range of size values, multiple candidate sizes are determined.

6. The method as described in claim 3, characterized in that, The determination of multiple candidate sizes based on the size of the defective region of the target product in the multiple sample images includes: If the target indication information indicates that the defect of the target type does not appear at a random location on the target product, then the size of the area where the defect of the target product is located in the plurality of sample images is determined as the plurality of candidate sizes.

7. The method as described in claim 3, characterized in that, The plurality of candidate sizes includes a plurality of first sizes and a plurality of second sizes; The determination of multiple candidate sizes based on the size of the defective region of the target product in the multiple sample images includes: If the target indication information indicates that a defect of the target type appears at a random location on the target product, then the dimensions of the defect area of ​​the target product in the plurality of sample images are multiplied by a scaling factor to obtain the plurality of first dimensions; based on the dimensions of the defect area of ​​the target product in the plurality of sample images, a range of dimension values ​​is determined, and based on the range of dimension values, the plurality of second dimensions are determined.

8. The method as described in claim 5 or 7, characterized in that, The dimensions of the defective area of ​​the target product include both horizontal and vertical dimensions, and the range of these dimensions includes both horizontal dimension range and aspect ratio range. Determining the size range based on the size of the defective region of the target product in the multiple sample images includes: The maximum value of the horizontal dimension of the defect area of ​​the target product in the plurality of sample images is determined as the upper limit of the horizontal dimension range, and the minimum value of the horizontal dimension of the defect area of ​​the target product in the plurality of sample images is determined as the lower limit of the horizontal dimension range. The maximum value among the aspect ratios corresponding to the defective area of ​​the target product in the plurality of sample images is determined as the upper limit of the aspect ratio range, and the minimum value among the aspect ratios corresponding to the defective area of ​​the target product in the plurality of sample images is determined as the lower limit of the aspect ratio range. The aspect ratio is the ratio of the horizontal dimension to the vertical dimension.

9. The method according to claim 1, characterized in that, The method further includes: Based on the multiple mask images, the multiple noise signals, the multiple background images, and the target type, the second annotation information corresponding to the multiple generated images is determined by the diffusion model. The second annotation information includes the size, location, and defect type of the defect area of ​​the target product in the corresponding generated image.

10. The method as described in claim 9, characterized in that, The diffusion model includes a downsampling network, an intermediate network, and an upsampling network, and each of the downsampling network, the intermediate network, and the upsampling network includes at least one network layer; In this network, the size of the input feature map of any layer in the downsampling network is smaller than the size of the output feature map, the size of the input feature map of any layer in the upsampling network is larger than the size of the output feature map, the input feature map of the first layer in the intermediate network is the output feature map of the last layer in the downsampling network, and the output feature map of the last layer in the intermediate network is the input feature map of the first layer in the upsampling network. The size of the feature map includes the size of the feature map in the horizontal direction and the size of the feature map in the vertical direction.

11. The method as described in claim 10, characterized in that, The step of determining the second annotation information corresponding to each of the multiple generated images based on the multiple mask images, the multiple noise signals, the multiple background images, and the target type using the diffusion model includes: Based on the feature maps output by multiple network layers, a target affinity feature map is determined. The target affinity feature map contains a second region, the boundary of which is the outline of the defect in the target affinity feature map. The target affinity feature map has the same size as the target generated image. The target generated image is any one of the multiple generated images. The multiple network layers include at least two of the downsampling network, the intermediate network, and the upsampling network. The size of the second region is determined as the size of the defect region of the target product in the target generated image, the position of the second region in the target affinity feature map is determined as the position of the defect region of the target product in the target generated image, and the target type is determined as the defect type of the target product in the target generated image.

12. The method as described in claim 11, characterized in that, The determination of the target affinity feature map based on the feature maps output by multiple network layers includes: Image semantic segmentation is performed on the feature maps output by the multiple network layers to obtain multiple pixel affinity feature maps; Based on the plurality of pixel affinity feature maps, the similarity between each pixel affinity feature map in the plurality of pixel affinity feature maps and other pixel affinity feature maps in the plurality of pixel affinity feature maps is determined, thereby obtaining multiple similarity values ​​corresponding to each pixel affinity feature map; The pixel affinity feature map with the highest target similarity among the multiple pixel affinity feature maps is determined as the target affinity feature map, and the target similarity is the average of the multiple similarities corresponding to the pixel affinity feature map.

13. The method as described in claim 1, characterized in that, The method further includes: The generated images are filtered to obtain multiple target images, which are used to train the target defect detection model.

14. The method as described in claim 13, characterized in that, The step of filtering the multiple generated images to obtain multiple target images includes: The multiple generated images are sequentially input into the first defect detection model to obtain the first output results corresponding to the multiple generated images output by the first defect detection model. The first output results indicate whether the target product in the corresponding generated image has a defect. The generated images in which the first output result indicates that the target product has a defect are identified as multiple candidate generated images. Based on the multiple candidate generated images, the multiple target images are determined; Wherein, the number of parameters in the first defect detection model is greater than the number of parameters in the target defect detection model, and / or, the accuracy of the parameters in the first defect detection model is higher than the accuracy of the parameters in the target defect detection model, and / or, the number of network layers in the first defect detection model is greater than the number of network layers in the target defect detection model.

15. The method as described in claim 14, characterized in that, The step of determining multiple target images based on the multiple candidate generated images includes: The plurality of candidate generated images are sequentially input into the second defect detection model to obtain the second output results corresponding to the plurality of candidate generated images output by the second defect detection model. The second output results indicate whether the target product in the corresponding candidate generated image has a defect. If the second output results indicate that the target product in the corresponding candidate generated image has a defect, the second output results also include the defect type of the defect in the corresponding candidate generated image. The generated image in which the second output result indicates that the target product has no defect or the defect type is not the target defect type is determined as the multiple target images; Wherein, the number of parameters in the second defect detection model is less than the number of parameters in the target defect detection model, and / or, the accuracy of the parameters in the second defect detection model is lower than the accuracy of the parameters in the target defect detection model, and / or, the number of network layers in the second defect detection model is less than the number of network layers in the target defect detection model.

16. An image generation apparatus, characterized in that, The device includes: The first determining module is used to determine multiple mask images based on the first annotation information corresponding to multiple sample images respectively. The multiple sample images are all images of a target product. The target product in the multiple sample images has a defect and the defect type includes the target type. The first annotation information includes the size and position of the area where the defect of the target product is located in the corresponding sample image. The multiple mask images all include regions of interest, and the size and / or position of the regions of interest in the multiple mask images are different. The second determining module is used to determine multiple generated images based on the multiple mask images, multiple noise signals, a background image, and the target type using a diffusion model, such that the target product in the multiple generated images has a defect and the defect type is the target type, the background image is an image of the target product without defects, the regions of interest included in the multiple mask images are used to guide the region where the defect of the target product is located in the multiple generated images, and the multiple noise signals are used to guide the target product to generate defects in the multiple generated images.

17. An electronic device, characterized in that, The computer device includes a memory and a processor. The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to implement the steps of the method according to any one of claims 1-15.

18. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in any one of claims 1-15.

Images