Training method of sample image generation model and image generation method and device

Abstract

The embodiment of the invention provides a training method of a sample image generation model and an image generation method and device, and relates to the technical field of machine vision. The training method of the sample image generation model comprises the following steps: acquiring a sample image; training a to-be-trained sample image generation model by using the sample image until the model converges; wherein the model comprises a generator and a discriminator; each training process of the model comprises the following steps: performing mask mark fusion on the first mark graph and the second mark graph to obtain a target mark graph; inputting the target marking graph and the template graph into a generator, so that the generator performs image fusion on the target marking graph and the template graph to obtain a generated image; and inputting the target image and the generated image into a discriminator, so that the discriminator judges whether the generated image is true or false based on the target image and the generated image. Therefore, through the scheme, the stability of the sample image generation model can be improved, so that the quality of the generated sample image is improved.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a training method for a sample image generation model, an image generation method and a device. Background Art

[0002] Generally, in industrial production, the requirements for product yield are high, and using defect detection models to detect industrial defects on product images to determine whether the product is qualified is the main way to ensure product yield. Among them, whether the defect images are sufficient is directly related to the performance of the defect detection model, and then affects the product yield. The so-called defect image is obtained by capturing images of products with product defects and the image content includes defects (blemishes).

[0003] However, due to the strictness of the industrial production environment, it is often not suitable to collect a large number of defective images of defective products. In some cases, due to the high yield rate of the production line, the products are not prone to defects, which also makes it difficult to collect defective images.

[0004] In the related art, in order to obtain a large number of defect images for training a defect detection model, a sample image generation model is usually trained to obtain sample images for training the defect detection model.

[0005] However, in the above-mentioned related technologies, it is necessary to obtain defect-free images for training the sample image generation model. However, since defect-free images are difficult to collect, and when there are slight defects in the defect-free images that do not meet the defect standards but do not affect the product being a good product, it will affect the stability of the sample image generation model, thereby affecting the quality of the generated sample images. Summary of the invention

[0006] The purpose of the embodiments of the present application is to provide a training method, an image generation method and a device for a sample image generation model. The specific technical solution is as follows:

[0007] In a first aspect, an embodiment of the present application provides a training method for a sample image generation model, comprising:

[0008] Acquire a sample image; wherein the sample image includes a template image, a target image, a first mark image, and a second mark image; the first mark image is an image obtained after a predetermined process is performed on the template image, and the second mark image is an image obtained after the predetermined process is performed on the target image, and the predetermined process is a process of setting a mask mark for a defect;

[0009] Using the sample image, the sample image generation model to be trained is continuously trained until the sample image generation model converges; wherein the sample image generation model includes: a generator and a discriminator; each training process of the sample image generation model includes:

[0010] Performing mask mark fusion on the first mark map and the second mark map to obtain a target mark map; wherein the mask mark fusion is used to fuse the mask marks existing in the first mark map and the mask marks existing in the second mark map into the same image, and to represent the mask marks existing in the first mark map in the same image in a preset manner;

[0011] The target label image and the template image are input to the generator, so that the generator performs image fusion on the target label image and the template image to obtain a generated image; wherein the training convergence target of the generator includes: when there is a mask mark represented in the preset manner in the target label image, using the image content of the template image to perform image repair on the target area; the target area is the image area indicated by the mask mark represented in the preset manner;

[0012] The target image and the generated image are input to the discriminator, so that the discriminator makes a true or false judgment on the generated image based on the target image and the generated image.

[0013] In a second aspect, an embodiment of the present application provides an image generation method, including:

[0014] Obtain a pre-collected real product image as an initial image;

[0015] Acquire a mask mark map; wherein the mask mark map is generated by adding mask marks of various types of defects to a blank image of the same size as the initial image, and the mask marks of different types of defects are different;

[0016] Inputting the initial image and the mask label map into a generator of a pre-trained sample image generation model, so that the generator fuses the initial image and the mask label map to obtain a generated image corresponding to the initial image;

[0017] The generated image corresponding to the initial image includes: defects of the type indicated by each mask mark of the mask mark image, and the sample image generation model is trained based on a sample image generation model training method provided in the first aspect.

[0018] In a third aspect, an embodiment of the present application provides a training device for a sample image generation model, comprising:

[0019] A sample image acquisition module, used to acquire a sample image; wherein the sample image includes a template image, a target image, a first mark image, and a second mark image; the first mark image is an image obtained after a predetermined process is performed on the template image, and the second mark image is an image obtained after the predetermined process is performed on the target image, and the predetermined process is a process of setting a mask mark for a defect;

[0020] A model training module, used to train a sample image generation model to be trained using the sample image until the sample image generation model converges; wherein the sample image generation model includes: a generator and a discriminator;

[0021] Each training process of the sample image generation model includes:

[0022] Performing mask mark fusion on the first mark map and the second mark map to obtain a target mark map; wherein the mask mark fusion is used to fuse the mask marks existing in the first mark map and the mask marks existing in the second mark map into the same image, and to represent the mask marks existing in the first mark map in the same image in a preset manner;

[0023] The target label image and the template image are input to the generator, so that the generator performs image fusion on the target label image and the template image to obtain a generated image; wherein the training convergence target of the generator includes: when there is a mask mark represented in the preset manner in the target label image, using the image content of the template image to perform image repair on the target area; the target area is the image area indicated by the mask mark represented in the preset manner;

[0024] The target image and the generated image are input to the discriminator, so that the discriminator makes a true or false judgment on the generated image based on the target image and the generated image.

[0025] In a fourth aspect, an embodiment of the present application provides an image generating device, including:

[0026] An image acquisition module is used to acquire a real product image collected in advance as an initial image;

[0027] A marking acquisition module, used to acquire a mask marking map; wherein the mask marking map is generated by adding mask markings of various types of defects to a blank image of the same size as the initial image, and the mask markings of different types of defects are different;

[0028] An image generation module, used for inputting the initial image and the mask label map into a generator of a pre-trained sample image generation model, so that the generator fuses the initial image and the mask label map to obtain a generated image corresponding to the initial image;

[0029] The generated image corresponding to the initial image includes: defects of the type indicated by each mask mark of the mask mark image, and the sample image generation model is trained based on the training method of the sample image generation model.

[0030] In a fifth aspect, an embodiment of the present application provides an electronic device, including:

[0031] Memory, used to store computer programs;

[0032] The processor is used to implement any sample image generation model training method provided in the first aspect and / or any image generation method provided in the second aspect when executing the program stored in the memory.

[0033] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the training method of any sample image generation model provided in the first aspect and / or any image generation method provided in the second aspect is implemented.

[0034] In the seventh aspect, an embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any training method for the sample image generation model provided in the first aspect above, and / or any image generation method provided in the second aspect above.

[0035] Beneficial effects of the embodiments of the present application:

[0036] As can be seen from the above, a sample image generation model provided by an embodiment of the present application includes a generator and a discriminator. In each training process of the sample image generation model, the mask marks of the first label map and the second label map in the sample image are fused, so that in the obtained target label map, while the mask marks in the first label map and the mask marks in the second label map exist, the mask marks existing in the first label map are characterized in a preset manner; thereby, the target label map and the template map are image-fused by the generator, and when the mask marks characterized in a preset manner exist in the target label map, the image content of the template map is used to perform image repair on the area indicated by such mask marks, so that the generated image can be close to the real defect image (the background part and the defect part are close), so as to achieve the effect of deceiving the discriminator, thereby improving the stability of the sample image generation model and further improving the quality of the generated sample image.

[0037] In addition, an image generation method provided by an embodiment of the present application can generate a large number of sample images using a small number of real product images by means of a sample image generation model obtained through training, for training a defect detection model. In addition, the generated sample images can include not only the image content in the above-mentioned initial image, but also the types of defects corresponding to the mask marks added when generating the mask mark map. Since the initial image is a real product image, when there are defects in the initial image, the generated image corresponding to the initial image generated can include not only the real defects of the product, but also the generated defects. In addition, when making the above-mentioned mask mark map, various types of defects and mask marks corresponding to various types of defects can be added to the blank image, thereby greatly improving the diversity of generated defects in the generated image and enabling the generated defects in the generated image to have a good data distribution. In this way, by applying the solution provided by the embodiment of the present application, sufficient defect images can be obtained as images for training the defect detection model to improve the performance of the defect detection model.

[0038] In addition, in the present application, there is no limitation on whether the template image has defects, so that in the training process and image generation process of the above-mentioned sample image generation model, the template image and the initial image can be defective images or defect-free images. This makes the above-mentioned training process and image generation process have high flexibility in the selection of template images and initial images, avoiding the impact on model training and image generation due to substandard image selection or difficult image acquisition (such as defect-free images are difficult to acquire).

[0039] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0041] Figure 1 A flowchart of a training method for a sample image generation model provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of a flow chart of an image generation method provided in an embodiment of the present application;

[0043] FIG3 (a) is an exemplary training flow chart of a sample image generation model in a specific embodiment provided in an embodiment of the present application;

[0044] FIG3( b ) is another exemplary training flowchart of a sample image generation model in a specific embodiment provided in an embodiment of the present application;

[0045] Figure 4 A flow chart of reasoning of a sample image generation model in a specific embodiment provided in the embodiment of the present application;

[0046] Figure 5 A schematic diagram of the structure of a training device for a sample image generation model provided in an embodiment of the present application;

[0047] Figure 6 A schematic diagram of the structure of an image generating device provided in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0050] To facilitate understanding, first, the professional terms and background technologies involved in this application are described in detail.

[0051] Deep learning is a research direction in the field of machine learning (ML). It is introduced into machine learning to make it closer to its original goal - artificial intelligence (AI). Among them, deep learning is the inherent laws and representation levels of learning sample data. The information obtained in these learning processes is of great help in the interpretation of data such as text, images and sounds. The ultimate goal of deep learning is to enable machines to have analytical learning capabilities like humans, and to be able to recognize data such as text, images and sounds. The applicable directions of the embodiments of the present application mainly include classification, target detection, semantic segmentation, instance segmentation, and text positioning in the field of computer vision.

[0052] Generative Adversarial Network (GAN) is a deep learning neural network model that learns tasks through game theory. The GAN consists of two parts: the generator and the discriminator. The generator and the discriminator learn through game theory during the model training phase. After training, the generator can be used to generate image samples to achieve the purpose of making the fake look real.

[0053] Defect-free image: refers to an image without any flaws in the image. In the defect detection task, it refers to a good image, also known as an OK image. In related technologies, it is also called a template image or a reference image.

[0054] Defective image: refers to an image with one or more flaws in the image. In the defect detection task, it refers to a defective image, also known as an NG image or a defective image.

[0055] Model training: By setting a certain goal, the deep neural network is continuously made to approach the goal, and the parameters of the deep neural network are adjusted in this process.

[0056] Defect detection: refers to the use of deep learning technology to detect defects in industrial products and mark them out, generally using technologies such as object detection, semantic segmentation, and instance segmentation.

[0057] In industrial production, the product yield rate is required to be high. The industrial defect detection task is the main part of ensuring the product yield rate. Whether the defect images are sufficient is directly related to the performance of the defect detection model and affects the yield rate of the output products. Due to the rigor of the industrial production environment, the data used to train the above defect detection model is often not suitable for large-scale collection. In some cases, it is difficult to collect defect samples (i.e. defect images) because they are not easy to generate (normal production lines have a high yield rate); and in some cases, it is difficult to collect "pure" defect-free samples because there are weak defects in good products that do not meet the defect standards. Collecting a large number of defect-free samples and then manually screening them may not only affect normal production but also consume labor costs.

[0058] In the related art, there are three exemplary methods for obtaining defect images as sample images for training defect detection models:

[0059] The first method: through data enhancement means (not relying on neural networks), the original defects in the collected defect images are transformed and then copied and pasted or Poisson fused in other areas. However, this method cannot generate defects with sufficient diversity. Its defect features are still based on the defects in the collected defect images, and the main changes are the positional relationships.

[0060] The second solution is to use a generative adversarial network to generate defects, and then copy and paste or Poisson fusion the defects onto the image. However, this method does not consider using a neural network to generate defect-free areas in the image. Therefore, the pixel values ​​of the defect-free areas are fixed, which can easily cause overfitting in the model training using the generated samples, and the defective and defect-free areas may have a false sense and boundary sense after pasting.

[0061] The third solution is to collect some defect-free images and a small number of defective images, and train the generative adversarial network by pairing the defect-free images with the defective images as template images, and generate defective images based on the defect-free images, so as to generate a certain number of sample images with generated defects. Since the defect-free areas are also generated by the network model (the generator in the generative adversarial network), the pixel diversity of the defect-free areas is also guaranteed. However, this method requires the simultaneous collection of defect-free images and defective images. In some cases, there are some minor defects in the defect-free images that do not meet the defect standard (such as the size is not a certain size, the contrast is not a certain threshold, etc., and the common oral expression is relatively light dirt, relatively small spots, relatively light oil stains, etc.). The minor defects are not labeled. If they are used as defect-free images for training, it will affect the stability of the generative adversarial network training, thereby affecting the quality of the generated sample images. On the other hand, the single sample image generated by this type of method only has generated defects, and cannot contain both real defect features and generated defect features. The diversity of defects in the sample images is insufficient, which can easily cause a large difference in the distribution of training data and the distribution of data in actual reasoning occasions.

[0062] It can be seen from the above that in the related art, when solving the above-mentioned technical problem of how to obtain sufficient defect images as sample images for training defect detection models, it is necessary to collect defect-free images and use the defect-free images as template images, and it is impossible to generate a single sample image that contains both real defect features and generated defect features.

[0063] That is to say, the related technologies have the following problems: 1. If it is difficult to collect a large number of OK images (images without defects), or there are only a small number of NG images (defective images), the above-mentioned related technologies cannot be used. 2. When the OK images (images without defects) have minor defects that do not meet the defect standards but do not affect the quality of the product, it is not conducive to the training effect of the generative adversarial network. If the OK images are to be manually screened, the labor cost is high. 3. The generated single sample image only has generated defects and cannot contain both real defect features and generated defect features. The diversity of defects in the sample images is insufficient, and the model using this sample set is prone to overfitting.

[0064] In order to obtain sufficient defect images as sample images for training a defect detection model, so as to improve the performance of the defect detection model, thereby ensuring the yield rate of the product, and solving the problems existing in the above-mentioned related technologies, firstly, an embodiment of the present application provides a training method for a sample image generation model to improve the stability of the sample image generation model, thereby improving the quality of the generated sample images.

[0065] Among them, the method can be applied to various industrial production scenarios that require training defect detection models to detect whether the products are qualified. For example, automobile manufacturers train defect detection models to detect whether the automobile parts they produce are qualified. For example, mobile phone manufacturers train defect detection models to detect whether the mobile phones they produce are qualified, etc. In addition, the execution subject of the method can be various electronic devices, which are specifically used to train to obtain sample image generation models. In specific applications, the electronic device can be a server, a desktop computer, etc., and the electronic device can be a single electronic device or an electronic device cluster composed of multiple electronic devices. Based on this, the embodiment of the present application does not specifically limit the application scenario and execution subject of the method.

[0066] A training method for a sample image generation model provided in an embodiment of the present application may include the following steps:

[0067] Acquire a sample image; wherein the sample image includes a template image, a target image, a first mark image, and a second mark image; the first mark image is an image obtained after a predetermined process is performed on the template image, and the second mark image is an image obtained after the predetermined process is performed on the target image, and the predetermined process is a process of setting a mask mark for a defect;

[0068] Using the sample image, training a sample image generation model to be trained until the sample image generation model converges; wherein the sample image generation model includes: a generator and a discriminator;

[0069] Each training process of the sample image generation model includes:

[0070] Performing mask mark fusion on the first mark map and the second mark map to obtain a target mark map; wherein the mask mark fusion is used to fuse the mask marks existing in the first mark map and the mask marks existing in the second mark map into the same image, and to represent the mask marks existing in the first mark map in the same image in a preset manner;

[0071] The target label image and the template image are input to the generator, so that the generator performs image fusion on the target label image and the template image to obtain a generated image; wherein the training convergence target of the generator includes: when there is a mask mark represented in the preset manner in the target label image, using the image content of the template image to perform image repair on the target area; the target area is the image area indicated by the mask mark represented in the preset manner;

[0072] The target image and the generated image are input to the discriminator, so that the discriminator makes a true or false judgment on the generated image based on the target image and the generated image.

[0073] As can be seen from the above, a sample image generation model provided by an embodiment of the present application includes a generator and a discriminator. In each training process of the sample image generation model, the mask marks of the first label map and the second label map in the sample image are fused, so that in the obtained target label map, while the mask marks in the first label map and the mask marks in the second label map exist, the mask marks existing in the first label map are characterized in a preset manner; thereby, the target label map and the template map are image-fused by the generator, and when the mask marks characterized in a preset manner exist in the target label map, the image content of the template map is used to repair the area indicated by such mask marks, so that the generated image can be close to the real defect image (the background part and the defect area are close), so as to achieve the effect of deceiving the discriminator, thereby improving the stability of the sample image generation model and further improving the quality of the generated sample image.

[0074] In addition, in the present application, there is no limitation on whether the template image has defects, so that in the training process of the above-mentioned sample image generation model and the subsequent image generation process, the template image and the initial image can be defective images or defect-free images. This allows the above-mentioned training process and the subsequent image generation process to have high flexibility in the selection of template images and initial images, avoiding the impact on model training and image generation due to substandard image selection or difficult image acquisition (such as defect-free images are difficult to acquire).

[0075] Below, in conjunction with the accompanying drawings, a training method for a sample image generation model provided in an embodiment of the present application is specifically described.

[0076] Figure 1 A flow chart of a training method for a sample image generation model provided in an embodiment of the present application, the method may include the following steps:

[0077] S101: Acquire a sample image.

[0078] Among them, the sample image includes a template image, a target image, a first marked image and a second marked image; the first marked image is an image obtained after a predetermined processing is performed on the template image, and the second marked image is an image obtained after a predetermined processing is performed on the target image, and the predetermined processing is a processing of setting a mask mark for defects.

[0079] In the present application, a sample image generation model to be trained including a generator and a discriminator can be pre-constructed, and then a training image for training the sample image generation model is obtained as a sample image. The number of sample images can be multiple, and each sample image includes a set of images: a template image, a target image, a first labeled image, and a second labeled image. It can be understood that in order to ensure that there are differences between the sample images, so as to achieve the purpose of effectively training the sample image generation model using different sample images, for any two sample images, at least one image in the template image and the target image is different.

[0080] The template image and the target image are real product images collected in advance, and the target image is a defect image. The template image can be a defect image or a non-defect image, and there are no defects with overlapping areas between the template image and the target image.

[0081] The first marking image is an image obtained by processing the template image with mask markings for defects, that is, the mask marking image of the template image obtained by marking defects on the template image with mask markings is used as the first marking image. When the above-mentioned mask markings are used to mark defects on the template image, the first marking image can be automatically generated.

[0082] Optionally, when there are defects in the template image, that is, when the template image is a defect image, for each defect in the template image, the area where the defect is located in the first marking image is covered with a graphic of a mask mark of the same type as the defect, then the covered graphic is the mask mark of the marked defect, and the area in the first marking image where the above mask mark is not added is a blank area.

[0083] Correspondingly, the second marked image is an image obtained by processing the target image with mask marks for defects, that is, the mask marked image of the target image obtained by marking defects on the target image with mask marks is used as the second marked image. When the target image is marked with defects with mask marks, the second marked image can be automatically generated.

[0084] Optionally, for each defect in the target image, if the area where the defect is located in the second marking image is covered with a graphic of a mask mark of the same type as the defect, the covered graphic is the mask mark of the marked defect, and the area in the second marking image where the above mask mark is not added is a blank area.

[0085] Among them, for any defect, the above-mentioned covered pattern can be a pattern with the same shape as the defect and a mask mark color preset for the type of defect; it can also be a pattern with the same shape as the defect, of course, it can also be a pattern with a mask mark color preset for the type of defect, and this embodiment of the present application does not make specific limitations. Any method in the related art that can realize setting a mask mark for a defect can be applied to the present application.

[0086] S102: training a sample image generation model to be trained using the sample image until the sample image generation model converges;

[0087] Among them, the sample image generation model includes: a generator and a discriminator;

[0088] Each training process of the sample image generation model may include the following steps:

[0089] Step A1: performing mask label fusion on the first label image and the second label image to obtain a target label image;

[0090] The mask mark fusion is used to fuse the mask marks in the first mark map and the mask marks in the second mark map into the same image, and to characterize the mask marks in the first mark map in the same image in a preset manner;

[0091] Step A2: inputting the target label image and the template image into the generator, so that the generator performs image fusion on the target label image and the template image to obtain a generated image;

[0092] The training convergence objectives of the generator include: when there is a mask mark represented in a preset manner in the target mark image, the target area is inpainted using the image content of the template image; the target area is the image area indicated by the mask mark represented in a preset manner;

[0093] Step A3: Input the target image and the generated image into the discriminator, so that the discriminator makes a true or false judgment on the generated image based on the target image and the generated image.

[0094] In the present application, the sample image generation model to be trained can be a generative adversarial network, which includes a generator and a discriminator. Therefore, when the sample image is input into the sample image generation model for training, the first labeling map and the second labeling map can be firstly fused with mask labels to obtain a target labeling map, and then the obtained target labeling map and the template map are input into the generator, so that the generator performs image fusion on the target labeling map and the template map to obtain a generated image. Exemplarily, the mask mark existing in the first labeling map and the mask mark existing in the second labeling map can be fused into the same blank image. Of course, it is not limited to a blank image. For example, it can also be a pre-set image with a predetermined background. Exemplarily, the predetermined background can be a product line background, etc., which is also reasonable.

[0095] The training goal of the generator is to expect the generated image generated by the generator to deceive the discriminator of the sample image generation model to be trained, and the discriminator is expected to identify the generated image as true. Furthermore, in order to achieve the above training goal, for the generator, it is expected that the generated image is as similar as possible to the real defect image (i.e., the defect image of the product collected), or even the same.

[0096] In order to achieve the above-mentioned purpose, when performing mask marker fusion, the mask marker existing in the first marker map corresponding to the template map and included in the target marker map can be represented in a preset manner, so that the training convergence target of the generator can include: when the target marker map has a mask marker represented in a preset manner, using the image content of the template map to perform image repair on the image area indicated by the mask marker represented in the preset manner.

[0097] The essence of the mask mark fusion process can be understood as: the mask mark of the first mark map and the mask mark of the second mark map are fused into the same image, and the obtained target mark map can simultaneously include the mask mark of the first mark map and the mask mark of the second mark map. Since the template image can be a defective image or a non-defective image, and the target image is a defective image, the target mark map must include the mask mark of the second mark map, and when the template image is a defective image, the target mark map can also include the mask mark of the first mark map; correspondingly, when the template image is a non-defective image, the mask mark of the first mark map can be empty, so that the target mark map can only include the mask mark of the second mark map.

[0098] In order to distinguish the mask marks of the first label map from the mask marks of the second label map, and subsequently determine whether the model has converged, in the above-mentioned mask mark fusion process, the mask marks existing in the first label map are represented in the same image in a preset manner.

[0099] Optionally, in a specific implementation, when both the template image and the target image are defect images, the above step A1, performing mask label fusion on the first label image and the second label image to obtain the target label image, includes the following steps:

[0100] Step A11: On the basis of retaining the mask marks existing in the second label map and characterizing the mask marks existing in the first label map in a preset manner, the mask marks existing in the first label map and the mask marks existing in the second label map are fused into the same image to obtain a target label map.

[0101] In this specific implementation, the purpose of fusing the defects in the first mark image and the defects in the second mark image is achieved by keeping the mask marks in the second mark image unchanged and characterizing the mask marks in the first mark image in a preset manner; of course, in other implementations, when the mask marks in the first mark image are characterized in a preset manner and the positions and types of defects indicated by the mask marks in the second mark image can still be identified in the target mark image, in the process of fusion to the target mark image, the mask marks in the second mark image may also change, for example, the mark color changes, or the mark shape changes, which are all feasible.

[0102] By characterizing the mask marks existing in the first label map in the same image in a preset manner, the trained sample image generation model can be equipped with the ability to repair the image area indicated by the mask marks characterized in a preset manner. That is to say, if the mask marks existing in the first label map are not specially processed (characterized in a preset manner), then the defects in the first label map will be retained in the generated image obtained by the sample image generation model. Therefore, regardless of whether there are defects in the template map, the defects will be retained in the generated image, that is, when the template map is a defect-free image, there will be no defects at the corresponding position of the generated image, and when the template map is a defective image, defects will be retained at the corresponding position of the generated image. This results in the defects in the generated image being defects other than those in the target image, and if the image has defects other than those in the target image, the image is easily judged as a false image.

[0103] Furthermore, if the mask mark in the first label map is not specially processed, then for the entire training process, the existence of the mask mark may make it difficult for the model to converge; in another case, the model may converge, but in one case, the model may randomly patch up the image area indicated by the mask mark in the first label map. That is to say, if the template map corresponding to the first label map has defects, in actual use, when the generated image is obtained, the defects may be randomly patched up or deformed, etc., resulting in the failure to retain the real defects; in another case, when there are no defects in the template map corresponding to the first label map, the model may also add random defects relative to the defects in the second label map, which may also result in the failure to retain the real defects when the image is generated.

[0104] There may be multiple implementations for representing the mask marks existing in the first mark map in the same image in a preset manner.

[0105] Optionally, in a specific implementation, the above-mentioned mask mark existing in the first mark image is represented in the same image in a preset manner, which may include: adjusting the color value of the area indicated by the mask mark existing in the first mark image in the same image to a specified color value.

[0106] In this specific implementation, when performing mask mark fusion, the color value of the area indicated by the mask mark existing in the first mark map can be adjusted to a specified color value in the same image into which the mask mark existing in the first mark map and the mask mark existing in the second mark map are fused. The above-mentioned specified color value can also be called a reserved color value. Exemplarily, the above-mentioned specified color value can be black.

[0107] In this way, through the adjustment of the above-mentioned color values, it is easy to distinguish the mask marks existing in the two marking images in the same image. Therefore, when using the generator to obtain the generated image, by determining the area where the specified color value exists, the area where the image needs to be repaired can be directly determined, so as to improve the efficiency of determining the area to be repaired.

[0108] Optionally, the mask mark existing in the first mark image is characterized in the same image in a preset manner, including: adjusting the area identifier of the area indicated by the mask mark existing in the first mark image in the same image to a predetermined area identifier.

[0109] In this specific implementation, when performing mask mark fusion, the area identifier of the area indicated by the mask mark existing in the first mark map can be adjusted to a predetermined area identifier in the same image into which the mask mark existing in the first mark map and the mask mark existing in the second mark map are fused.

[0110] In this way, by adjusting the above-mentioned area identification, it is easy to distinguish the mask marks existing in the two marking images in the same image. Therefore, when using the generator to obtain the generated image, by determining the area belonging to the predetermined area identification, the area that needs to be repaired can be directly determined, so as to improve the efficiency of determining the area to be repaired.

[0111] It should be noted that the goal of image inpainting is to ensure that the generated image output by the generator can deceive the discriminator as much as possible. Otherwise, when the generated image contains mask marks represented by the above preset method, the generated image can be easily judged as False by the discriminator. Image inpainting can make the adversarial training of the above generator and discriminator proceed normally and stably. At the same time, the above image inpainting can also increase the learning of structural knowledge by the above generator and discriminator, and improve the quality of the final image. Moreover, in the training process of the above sample image generation model, the use of image inpainting can make the above generator and discriminator work stably.

[0112] Optionally, in a specific implementation, in the above step A2, the generator performs image fusion on the target label image and the template image to obtain a generated image, including:

[0113] Step A21: generating image content corresponding to the image content of the reference area of ​​the template image in a designated area other than the existing mask mark in the target mark image, and obtaining an intermediate image;

[0114] Wherein, the reference area is an image area having the same image position as the image position of the specified area;

[0115] Step A22: performing predetermined image processing on the intermediate image to obtain a generated image;

[0116] Wherein, the predetermined image processing includes a first sub-processing and a second sub-processing;

[0117] The first sub-processing includes: when there is a mask mark represented in a preset manner in the target mark map, using the image content of the template map to perform image repair on the target area in the intermediate image;

[0118] The second sub-processing includes: generating defects of the type indicated by other mask marks in the image area indicated by other mask marks in the intermediate image according to the form of other mask marks that are not represented in a preset manner in the target mark map.

[0119] In this specific implementation, after receiving the target mark map and the template map, the generator can generate image content corresponding to the image content of the reference area of ​​the template map in the specified area outside the mask mark existing in the target mark map to obtain an intermediate image. That is, for the specified area, the generator can generate the image content of the specified area of ​​the target mark map according to the image content of the reference area of ​​the template map.

[0120] When generating the image content of the designated area of ​​the target mark map, the generated image content should be as close as possible to the image content of the reference area of ​​the template map, so that the generated image can be as close as possible to the real defect image. The above image content can be understood as the visual pixels of the image.

[0121] Then, a predetermined image processing is performed on the obtained intermediate image to obtain a generated image, wherein the predetermined image processing includes a first sub-processing and a second sub-processing.

[0122] That is, after obtaining the intermediate image, it is determined whether there is a mask mark represented in a preset manner in the target mark map. If so, the image content of the template map is used to perform image repair on the target area in the intermediate image.

[0123] Furthermore, after obtaining the intermediate image, other mask marks that are not represented in a preset manner in the target mark image are determined, and defects of the type indicated by the other mask marks are generated in the image area indicated by the other mask marks in the intermediate image according to the form of the other mask marks.

[0124] Among them, considering that the mask marks originally belonging to the second mark map, that is, other mask marks that are not represented in a preset manner in the target mark map, have a morphology, therefore, the shape, size and other information of the defect to be generated can be determined according to the morphology of the other mask marks, and the type of the defect to be generated can be determined according to the type indicated by the other mask marks. In this way, the generator can generate a defect with a corresponding morphology and type in the image area indicated by the other mask marks in the intermediate image according to the morphology of the other mask marks.

[0125] Optionally, the mask mark may include two attributes, namely, shape and color, wherein different colors correspond to different types of mask marks. Therefore, after determining the shape, size and other information of the defect to be generated, the type of defect to be generated may be determined according to the color of other mask marks, so that the generator may generate a defect of the type having the color of the mask mark in the image area indicated by the other mask mark in the intermediate image according to the shape of the other mask mark. It should be noted that when the mask mark also includes a color, the color value of the color is different from the above-mentioned specified color value.

[0126] Among them, the embodiment of the present application does not limit the execution order of the above-mentioned first sub-processing and the second sub-processing. The first sub-processing can be executed first and then the second sub-processing; or the second sub-processing can be executed first and then the first sub-processing, which is reasonable.

[0127] It should be emphasized that the above template image can be a defective image or a non-defective image. When the above template image is a non-defective image, there is no mask mark belonging to the first mark image in the above target mark image, that is, there is no mask mark characterized in a preset manner. Therefore, after generating the image content of the specified area of ​​the above target mark image, and generating the defect of the corresponding type of the mask mark in the image area indicated by the other mask mark in the intermediate image according to the form of other mask marks, the generator can complete the image generation and obtain the generated image. At this time, the generated image obtained is a defective image.

[0128] When the template image is a defect image, there are mask marks belonging to the first label image in the target label image, that is, there are mask marks characterized in a preset manner, so it is necessary to process the mask marks belonging to the first label image in the target label image. For example, when the template image is a defect image, there are mask marks characterized in a preset manner in the target label image, so the generator needs to process such mask marks.

[0129] Among them, when the above-mentioned template image is a defective image, in order to achieve the training purpose of the generator, it is expected that the defects of the above-mentioned generated image are as similar as possible to, or even the same as, the above-mentioned target image, and considering that there are no defects with overlapping areas between the above-mentioned template image and the target image, the defects of the template image do not exist in the target image. Furthermore, in order to make the defects of the obtained generated image as similar as possible to, or even the same as, the area indicated by the corresponding mask mark represented in a preset manner in the obtained generated image should not have defects, but should be as close as possible to the image content of the above-mentioned target image. Furthermore, considering that the image content of the designated area of ​​the target mark image is generated based on the template image, therefore, when the above-mentioned template image is a defective image and there are mask marks represented in a preset manner, for the mask marks belonging to this type in the target mark image, the image content of the template image can be used to perform image repair on the area where the mask marks of this type are located.

[0130] In this way, the image content of the specified area of ​​the above-mentioned target mark image is generated; according to the form of other mask marks, defects of the corresponding type of the mask mark are generated in the image area indicated by other mask marks in the intermediate image; and after the mask mark represented in a preset manner is repaired, the generator can complete the image generation and obtain a generated image. At this time, the generated image obtained is a defect image.

[0131] Afterwards, after obtaining the above generated image, the above target image and the generated image can be input into the discriminator of the sample image generation model to be trained.

[0132] Among them, the training purpose of the above discriminator is to judge the real defect image as true (True) and the generated defect image as false, that is, the training purpose of the above discriminator is to judge the above target image as true and the above generated image as false.

[0133] Based on this, when the above generated image is obtained, a True label can be added to the above target image, and a False label can be added to the above generated image, so that the target image and the generated image with the label added are input to the discriminator, and the discriminator can first learn the image features of the real defect image and the image features of the generated defect image based on the image content of the acquired target image and the generated image, as well as the labels added to the target image and the generated image. Then, a True label can be added to the above generated image, and the generated image with the True label can be input to the discriminator again, so that the discriminator first makes a true or false judgment on the generated image based on the learned image feature knowledge, and determines whether the judgment result of the discriminator is accurate based on the similarities and differences between the obtained judgment result and the True label added to the above generated image.

[0134] Among them, for the above-mentioned generated image with the True label added, when the judgment result of the discriminator is: the generated image is False, it means that the image generated by the above-mentioned generator fails to deceive the discriminator, and the model parameters of the above-mentioned generator can be adjusted so that the adjusted generator can generate a generated image that is closer to the real defect image (target image); and when the judgment result of the discriminator is: the generated image is True, it means that the image generated by the above-mentioned generator can deceive the discriminator, but the training purpose of the discriminator itself is not achieved. Therefore, the parameters of the above-mentioned discriminator can be adjusted so that the adjusted generator can distinguish the real defect image (target image) and the generated defect image (generated image) as much as possible, so as to judge the real defect image as True and the generated defect image as False.

[0135] As mentioned above, in the embodiment of the present application, the training process of the sample image generation model includes a generator training stage and a discriminator training stage, and the generator training stage and the discriminator training stage appear alternately and influence each other.

[0136] Among them, in the above generator training stage, the generator generates image content corresponding to the image content of the reference area of ​​the template image in the blank area of ​​the target mark image, and generates defects of the type corresponding to the mask mark in the area where the mask mark is located according to the form of the mask mark of the second mark image, and obtains a generated image, and its training purpose is to expect the discriminator to judge the above generated image as true (True). In the above discriminator training stage, its training purpose is to judge the real defect image as True and the generated defect image as False.

[0137] That is, the generator hopes that the output obtained can deceive the discriminator, while the discriminator hopes not to be deceived by the output of the generator, and the judgment result of the discriminator can affect the adjustment of the model parameters of the generator and the discriminator, and the generated image output by the generator can affect the judgment result of the discriminator. Therefore, the training purposes of the generator and the discriminator are contradictory, and the output results of the two affect each other. In this way, the output results and training purposes of the generator and the discriminator can be balanced by alternating the training of the generator and the discriminator, and the model training can be completed.

[0138] In this way, when training the sample image generation model to be trained, the acquired template image, target image, first labeled image and second labeled image can be used to alternately train the generator and discriminator of the sample image generation model to be trained, and finally a balance between the output results of the above generator and discriminator is achieved, for example, Nash equilibrium, so as to finally obtain a trained sample image generation model that meets the needs of actual applications.

[0139] Thus, after obtaining the judgment result of the above-mentioned discriminator for judging the authenticity of the above-mentioned generated image, the loss value of the discriminator can be calculated based on the judgment result of the above-mentioned discriminator, and the loss value of the generator can be calculated based on the judgment result of the above-mentioned discriminator, the target image and the generated image. Furthermore, the model parameters of the discriminator can be adjusted based on the loss value of the discriminator, and the model parameters of the generator can be adjusted based on the loss value of the generator.

[0140] Among them, the loss function used to calculate the loss value of the discriminator and the loss function used to calculate the loss value of the generator can be a conventional generative adversarial network loss function.

[0141] In this way, when the loss value of the above-mentioned discriminator and the loss value of the above-mentioned generator both meet the preset loss conditions, it can be considered that the output results of the above-mentioned generator and discriminator are balanced with the training purpose, and the training of the generator and the discriminator is completed, that is, the training of the sample image generation model to be trained is completed, and a trained sample image generation model is obtained.

[0142] In addition, optionally, in some cases, when the number of iterations of the sample image reaches a set value during the training process, the training is stopped, and the model obtained when the training is stopped is used as the trained sample image generation model.

[0143] As mentioned above, in this specific implementation, the principle of training the generative adversarial network is: the discriminator will judge the difference (loss, loss value) between the predicted image (generated image) and the real image (target image) generated by the generative adversarial network, and the training goal is to make the predicted image (generated image) closer to the real defect image. Among them, when training the generative adversarial network, the defects in the defect image (target image) with defects can be marked with special marks (mask marks), so that the generative adversarial network can repair the defects in the template image. In this way, based on the defect image (target image) and the template image, the predicted defect image (generated image) generated by the generative adversarial network can be close to the background part of the template image and the defect part of the real defect image (target image). In this way, the predicted defect image can be close to the real defect image, achieving the effect of deceiving the discriminator of the generative adversarial network. Therefore, this application does not limit whether the template image has defects, and regardless of whether the template image has defects, the discriminator can confirm the ability of the generative adversarial network to generate defect images without being disturbed by the defects of the template image.

[0144] Based on this, the following problems existing in the prior art can be solved: in order to normally train the generative adversarial network, the template image can only be an OK image and cannot have defects. Otherwise, the predicted image generated by the generative adversarial network will contain the defects in the template image, making it difficult to approach the real image and affecting the work of the discriminator.

[0145] At this point, after completing the training of the above sample image generation model, the image generation method provided by the embodiment of the present application can be used to generate a large number of sample images using the generator in the above trained sample image generation model, and the defect detection model training can be performed using the generated large number of sample images. Finally, the defect detection model obtained by training is used to perform defect detection on the generated product to ensure the yield rate of the generated product.

[0146] As can be seen from the above, a sample image generation model provided by an embodiment of the present application includes a generator and a discriminator. In each training process of the sample image generation model, the mask marks of the first label map and the second label map in the sample image are fused, so that in the obtained target label map, while the mask marks in the first label map and the mask marks in the second label map exist, the mask marks existing in the first label map are characterized in a preset manner; thereby, the target label map and the template map are image-fused by the generator, and when the mask marks characterized in a preset manner exist in the target label map, the image content of the template map is used to repair the area indicated by such mask marks, so that the generated image can be close to the real defect image (the background part and the defect area are close), so as to achieve the effect of deceiving the discriminator, thereby improving the stability of the sample image generation model and further improving the quality of the generated sample image.

[0147] In addition, in the present application, there is no limitation on whether the template image has defects, so that in the training process of the above-mentioned sample image generation model and the subsequent image generation process, the template image and the initial image can be defective images or defect-free images. This allows the above-mentioned training process and the subsequent image generation process to have high flexibility in the selection of template images and initial images, avoiding the impact on model training and image generation due to substandard image selection or difficult image acquisition (such as defect-free images are difficult to acquire).

[0148] Optionally, in a specific implementation, the training method of a sample image generation model provided in an embodiment of the present application may further include the following steps:

[0149] Step B: collecting multiple real product images, and performing predetermined processing on each real product image to obtain a labeling image of each real product image;

[0150] The multiple real product images at least include: defect images.

[0151] Accordingly, the above step S101, obtaining a sample image, may include the following steps:

[0152] Step C: Acquire a template image and a target image from a plurality of real product images, and acquire a labeled image of the template image as a first labeled image, and acquire a labeled image of the target image as a second labeled image, so as to obtain a sample image.

[0153] In this specific implementation, multiple real product images may be collected in advance, and the multiple real product images may at least include defective images, that is, the multiple real product images may all be defective images, or may be partially defective images and partially non-defective images. Furthermore, each real product image may be processed in a predetermined manner to obtain a labeling image of each real product image.

[0154] Among them, for each real product image, when the real product image is a defective image, the marking map of the real product image may include mask marks for covering each defect in the real product image, the shape of each mask mark is the same as the shape of the defect to be covered, and the type of the mask mark can represent the type of the defect to be covered; and when the real product image is a defect-free image, the marking map of the real product image can be a blank image.

[0155] Since the construction of the sample image needs to follow certain construction principles: the target image is a defect image, and the template image can be a defect image or a non-defective image, and the template image and the target image do not have defects with overlapping areas, therefore, after obtaining multiple real product images, the template image and the target image can be obtained from the multiple real product images according to the construction principles, and then, the marking image of the template image can be further obtained as the first marking image, and the marking image of the target image can be obtained as the second marking image, thereby completing the acquisition of the sample image.

[0156] In this specific implementation, the above-mentioned template image can be a defective image or a non-defective image. Among them, the so-called defective image is a product image obtained by image acquisition of a product with product defects and the image content includes defects (flaws). Correspondingly, the so-called non-defective image refers to a product image obtained by image acquisition of a product and the image content does not include defects. That is, a non-defective image refers to an image without any defects in the image. In the defect detection task, it refers to a good product image, also known as an OK image. A defective image is also called a defective image, which refers to an image with one or more defects in the image. In the defect detection task, it refers to a defective image, also known as an NG image. In this way, in the present application, since it is not necessary to limit the above-mentioned template image to a defective image or a non-defective image, the difficulty of constructing the sample image can be reduced and the flexibility of using the template image for model training can be improved.

[0157] Moreover, in actual scenarios, in some cases, good products, i.e. qualified products, may have weak defects that do not meet the defect standards. Therefore, it is difficult to collect "pure" defect-free images. After collecting a large number of defect-free images, manual screening of "pure" defect-free images will not only affect normal production but also consume labor costs. Therefore, when the above template image is a defective image, it is not necessary to collect defect-free images of the product. Thus, it is possible to avoid some defects in the collected defect-free images that do not meet the defect standards from affecting the training of the above sample image generation model, and it is also possible to avoid the labor costs caused by manual selection of "pure" defect-free images and the impact on product generation.

[0158] Optionally, in a specific implementation, the above step C, obtaining the template image and the target image from multiple real product images, may include the following steps:

[0159] Step C1: Randomly select two images from multiple real product images;

[0160] Step C2: when the two extracted images include a defective image and a non-defective image, the extracted non-defective image is determined as a template image, and the extracted defective image is determined as a target image;

[0161] Step C3: When both the extracted images are defective images, determine whether there is an overlapping area between the defects of the two extracted images; if not, execute step C4; if so, execute step C5.

[0162] Step C4: determining one of the extracted images as a template image, and determining another extracted image as a target image;

[0163] Step C5: Ignore the two extracted images and return to the above step C1.

[0164] Step C6: When the two extracted images are both defect-free images, ignore the two extracted images and return to the above step C1.

[0165] In this specific implementation, two images may be randomly extracted from the multiple real product images collected above. Since the multiple real product images may include only defective images or both defective images and non-defective images, the two extracted images may exist in various situations, including: extracting two defective images, extracting two non-defective images, and extracting one defective image and one non-defective image. In response to the above different situations, whether the two extracted images can be used as template images and target images respectively is different.

[0166] Among them, when the two extracted images are both defect-free images, since both images cannot be used as target images, it is necessary to abandon the extraction results and re-extract the images, that is, ignore the two extracted images and return to the step of randomly extracting two images from multiple real product images to re-extract the images.

[0167] In the case where the two extracted images include a defective image and a non-defective image, since there is no defect in the non-defective image, the two extracted images do not have defects with overlapping areas. Therefore, the extracted defective image can be directly determined as the target image, and the extracted non-defective image can be determined as the template image.

[0168] In the case where the two extracted images include two defective images, since both the extracted images have defects, and the target image and the template image finally determined do not have defects with overlapping areas, it is necessary to first determine whether the defects in the two extracted images have overlapping areas.

[0169] For example, the label images corresponding to the two extracted images can be obtained, so as to determine whether the mask marks in the two label images exist: mask marks belonging to the two label images and having overlapping areas. If they exist, it means that the defects in the two extracted images have overlapping areas, and if they do not exist, it means that the defects in the two extracted images do not have overlapping areas.

[0170] In this way, when it is determined that there is no overlapping area between the defects of the two extracted images, the two extracted images can be used as a template image and a target image, respectively. Thus, one of the extracted images can be determined as a template image, and the other extracted image can be determined as a target image. Among them, the two extracted images can be determined as a template image and a target image, respectively, in a variety of ways. For example, one of the extracted images can be randomly determined as a template image, and the other extracted image can be determined as a target image. For another example, the image extracted first can be determined as a template image, and the image extracted later can be determined as a target image, etc.

[0171] Correspondingly, when it is determined that there is an overlapping area between the defects of the two extracted images, the two extracted images cannot be used as the template image and the target image respectively. Therefore, it is necessary to abandon the extraction result and re-extract the image, that is, ignore the two extracted images and return to the step of randomly extracting two images from multiple real product images to re-extract the image.

[0172] Secondly, an embodiment of the present application provides an image generation method to improve the diversity of generated defects in the generated image and make the generated defects in the generated image have a good data distribution. At the same time, sufficient defect images are provided for the training of the defect detection model to improve the performance of the defect detection model.

[0173] Among them, the method can be applied to various industrial production scenarios that require training defect detection models to detect whether the products are qualified. For example, automobile manufacturers train defect detection models to detect whether the automobile parts they produce are qualified. For example, mobile phone manufacturers train defect detection models to detect whether the mobile phones they produce are qualified, etc. In addition, the execution subject of the method can be various electronic devices, which are specifically used to generate images using the sample image generation model obtained by training. In specific applications, the electronic device can be a server, a desktop computer, etc., and the electronic device can be a single electronic device or an electronic device cluster composed of multiple electronic devices. Based on this, the embodiment of the present application does not specifically limit the application scenario and execution subject of the method.

[0174] It should be noted that the electronic device that executes an image generation method provided in an embodiment of the present application and the electronic device that executes the training method of the above-mentioned sample image generation model can be the same device or different devices, and the embodiment of the present application does not make any specific limitations on this.

[0175] An image generation method provided in an embodiment of the present application may include the following steps:

[0176] Obtain a pre-collected real product image as an initial image;

[0177] Acquire a mask mark map; wherein the mask mark map is generated by adding mask marks of various types of defects to a blank image of the same size as the initial image, and the mask marks of different types of defects are different;

[0178] Inputting the initial image and the mask label map into a generator of a pre-trained sample image generation model, so that the generator fuses the initial image and the mask label map to obtain a generated image corresponding to the initial image;

[0179] The generated image corresponding to the initial image includes defects of the type indicated by each mask mark of the mask mark image, and the sample image generation model is trained based on the training method of the sample image generation model.

[0180] As can be seen from the above, an image generation method provided by an embodiment of the present application can generate a large number of sample images using a small number of real product images by means of a sample image generation model obtained through training, for training a defect detection model. Moreover, the generated sample image can include not only the image content in the above-mentioned initial image, but also the types of defects corresponding to the mask marks added when generating the mask mark map. Since the initial image is a real product image, when there are defects in the initial image, the generated image corresponding to the initial image generated can include not only the real defects of the product, but also the generated defects. Moreover, when making the above-mentioned mask mark map, various types of defects and mask marks corresponding to various types of defects can be added to the blank image, thereby greatly improving the diversity of generated defects in the generated image and enabling the generated defects in the generated image to have a good data distribution. In this way, by applying the solution provided by the embodiment of the present application, sufficient defect images can be obtained as images for training the defect detection model to improve the performance of the defect detection model.

[0181] In addition, in the present application, there is no limitation on whether the template image has defects, so that in the training process and image generation process of the above-mentioned sample image generation model, the template image and the initial image can be defective images or defect-free images. This makes the above-mentioned training process and image generation process have high flexibility in the selection of template images and initial images, avoiding the impact on model training and image generation due to substandard image selection or difficult image acquisition (such as defect-free images are difficult to acquire).

[0182] Below, in conjunction with the accompanying drawings, an image generation method provided in an embodiment of the present application is specifically described.

[0183] Figure 2 A schematic diagram of a flow chart of an image generation method provided in an embodiment of the present application, the method may include the following steps:

[0184] S201: Acquire a pre-collected real product image as an initial image.

[0185] In the present application, when a sample image generation model is trained and used to generate a sample image, a pre-collected real product image may be first obtained as an initial image.

[0186] Among them, the above-mentioned initial image can be a defective image or a non-defective image. As mentioned above, the so-called defective image is a product image obtained by image acquisition of a product with product defects and the image content includes defects (flaws). Correspondingly, the so-called non-defective image refers to a product image obtained by image acquisition of a product and the image content does not include defects. That is, a non-defective image refers to an image without any defects in the image. In the defect detection task, it refers to a good product image, also known as an OK image. A defective image is also called a defective image, which refers to an image with one or more defects in the image. In the defect detection task, it refers to a defective image, also known as an NG image. In this way, in the present application, since it is not necessary to limit the above-mentioned initial image to a defective image or a non-defective image, the flexibility of using the initial image for image generation can be improved.

[0187] Moreover, in actual scenarios, in some cases, since good products, i.e. qualified products, may have weak defects that do not meet the defect standards, it is also difficult to collect "pure" defect-free images. After collecting a large number of defect-free images, manually screening the "pure" defect-free images will not only affect normal production but also consume labor costs. Therefore, when the above-mentioned initial image is a defective image, it is not necessary to collect defect-free images of the product, thereby avoiding some defects in the collected defect-free images that do not meet the defect standards from affecting the training of the above-mentioned sample image generation model and the above-mentioned image generation, and also avoiding the labor costs caused by manually selecting "pure" defect-free images and the impact on product generation.

[0188] S202: Obtain a mask marking image.

[0189] The mask mark image is generated by adding mask marks of various types of defects to a blank image with the same size as the initial image, and the mask marks of different types of defects are different.

[0190] In the present application, when the sample image generation model obtained by the training is used to generate the sample image, a mask mark map needs to be prepared, and the mask mark map is generated by adding mask marks of various types of defects to a blank image of the same size as the initial image. That is, a blank image of the same size as the initial image can be first obtained, and then mask marks of various types of defects can be added to the blank image, and different types of defects have different mask marks.

[0191] Among them, various types of defects that may exist in the product can be determined in advance, and mask marks for each type of defect can be set. In order to distinguish different types of defects, the mask marks for different types of defects are different. For example, the mask mark for scratch defects is mark 1, and the mask mark for pit defects is mark 2, etc. The process of setting marks for defects in the image, that is, using mask marks to mark the image for defects, can be understood as using mask marks to cover the area where the defects are located in the image. At this time, the shape of the mask mark is the same as the shape of the defect in the image, and the type is the type of the defect in the preset image.

[0192] Optionally, use color to distinguish mask marks of different types of defects. For example, the color of the mask mark for scratch defects is red, the color of the mask mark for pit defects is yellow, the color of the mask mark for convex defects is green, and so on. Generally, a mask mark can be understood as a graphic with a specific shape and color used to cover defects in an image, so that when the mask mark is used to cover the area where the defect is located in the image, the shape of the mask mark is the same as the shape of the defect in the image, and the color is the color of the mask mark of the type to which the defect belongs in the preset image. Optionally, when the mask mark is used to mark defects in an image, the edge line of the mask mark is outlined along the edge of the area where the defect is located to obtain the graphic of the mask mark, and then the corresponding color is filled in the outlined graphic according to the type of defect.

[0193] As mentioned above, the above-mentioned mask mark map is obtained by directly adding mask marks to a blank image, and there are no defects in the blank image. Therefore, the mask marks added to the blank image are not used to cover the defects in the image, but to mark the generated defects that are desired to exist in the generated image. That is, for each mask mark in the mask mark map, in the sample image generated based on the mask mark map, there may be defects with the same position and shape as the mask mark and the type of the mask mark. The above-mentioned shape may include information such as shape and size. For example, assuming that the mask color of the convex defect is mark 1, when there is a circular mask mark with image position A as the center and size B as the radius and mark 1 in the mask mark map, then there will be a convex defect with image position A as the center and size B as the radius in the image generated based on the mask mark map.

[0194] Based on this, when generating the above-mentioned mask mark map, the position, form, type and other information of the generated defects that are expected to exist in the target sample image used for the model training task can be determined according to the needs of the actual application, the actual production situation of the product, etc., so that according to the information of the generated defects that are expected to exist in the above-mentioned target sample image, the corresponding mask mark is added to the blank image to obtain the mask mark map. Among them, one or more mask marks of one type of defect can be added to the above-mentioned mask mark map, and multiple mask marks of multiple types of defects can be added, and the number of mask marks of each type of defect can be one or more, and the mask mark can be added at any position of the above-mentioned mask mark map, and the form of each added mask mark can be arbitrary. That is, when generating the above-mentioned mask mark map, the position, number, form and type of the mask mark added in the mask mark map are not limited. In this way, a large number of mask mark maps with different added mask marks can be generated, thereby greatly improving the richness of the mask mark map. Furthermore, when using the above-mentioned mask marking image to generate sample images, a large number of sample images with different defects can also be generated. Therefore, the diversity of generated defects in the generated sample images can be greatly improved, and the generated defects in the generated sample images can have a good data distribution, so as to ensure that sufficient defect images can be finally obtained as sample images for training the defect detection model, so as to improve the performance of the defect detection model.

[0195] The mask mark image can be generated in a variety of ways. For example, using a computer peripheral such as a mouse, in a blank image of the same size as the initial image, using preset mask marks of various types of defects (or the colors of mask marks), draw one or more, one or more types of mask marks of defects; for another example, using a function to generate mask marks such as curves, ellipses, polygons, etc. in a blank image of the same size as the initial image.

[0196] S203: Inputting the initial image and the mask label image into the generator of the pre-trained sample image generation model, so that the generator fuses the initial image and the mask label image to obtain a generated image corresponding to the initial image.

[0197] Among them, the generated image corresponding to the initial image includes: defects of the type indicated by each mask mark of the mask mark image, and the sample image generation model is trained based on a sample image generation model training method provided in an embodiment of the present application.

[0198] In the present application, after obtaining the above-mentioned initial image and mask marking map, the above-mentioned initial image and mask marking map can be input into the generator of the pre-trained sample image generation model, so that the generator can fuse the initial image and the mask marking map to obtain a generated image corresponding to the initial image.

[0199] It should be noted that the generated image corresponding to the above initial image is the target sample image used as the subsequent model training task, for example, the target sample image used as the training task of the defect detection model.

[0200] The image area of ​​the mask mark map is divided into two parts according to whether the mask mark is added or not. The area without the mask mark is the blank area, and the other area is the area with the mask mark added. The image area in the initial image whose image position is the same as the image position of the blank area is used as the target reference area.

[0201] In this way, after obtaining the above-mentioned initial image and the mask mark map, the generator can generate image content corresponding to the image content of the target reference area of ​​the initial image in the blank area in the mask mark map. That is, for the blank area in the mask mark map, the generator can generate the image content of the blank area according to the image content of the target reference area of ​​the initial image. The above-mentioned image content can be understood as the visual pixels of the image.

[0202] Among them, under ideal conditions, the image content of the blank area generated above is consistent with the image content of the target reference area of ​​the initial image, and in actual applications, the image content of the blank area generated above has a high similarity with the image content of the target reference area of ​​the initial image, and the image content of the blank area generated above can retain the significant features in the image content of the target reference area of ​​the initial image to a great extent. Therefore, when there is a defect in the target reference area of ​​the initial image, the defect feature of the defect can be retained in the generated image to a great extent. Since the initial image is a real product image, the defect in the initial image is a real defect, so the real defect feature can exist in the generated image.

[0203] Furthermore, for each mask mark in the mask mark diagram, since the mask mark has a shape and a type, the shape, size and other information of the defect to be generated can be determined according to the shape of the mask mark, and the type of the defect to be generated can be determined according to the type of the mask mark. Thus, according to the shape of each mask mark, a defect of the type indicated by the mask mark is generated in the area where the mask mark is located in the mask mark diagram.

[0204] Optionally, different types of mask marks can be characterized by different colors, so that the type of defect to be generated can be determined according to the color of the mask mark, and according to the shape of each mask mark, in the area where the mask mark is located in the mask mark map, a defect of the type with the color of the mask mark is generated. For example, assuming that the mask color of a convex defect is green, when there is a green circular mask mark with image position A as the center and size B as the radius in the mask mark map, then, in the mask mark map, the area where the green circular mask mark is located generates a convex defect with image position A as the circle and size B as the radius. Different types of mask marks can also be characterized by different visual elements, and visual elements can be understood as graphic elements with a smaller size. The mask mark can be composed of several visual elements. For example, the mask mark of a type A defect is composed of multiple first-type visual elements, and the mask mark of a type B defect is composed of multiple second-type visual elements. It should be noted that different colors are used to distinguish different types of mask marks. Different colors can be understood as multiple colors with different hues, for example, gray, red, green, etc.; they can also be understood as different variants of the same color, for example, gray variants can include titanium gray, pink gray, lead gray, etc. with different grayscales, which are all reasonable.

[0205] Among them, as mentioned above, the mask mark added in the mask mark image represents: the defect that is expected to exist in the generated image. Therefore, the defect generated for each mask mark is not the defect originally existing in the initial image, but a generated defect. Therefore, the generated image may also have generated defect features.

[0206] That is, the generated image may include not only real defect features but also generated defect features, thereby further improving the diversity of the generated sample images and ensuring that the sample images have good data distribution.

[0207] As can be seen from the above, an image generation method provided by an embodiment of the present application can generate a large number of sample images using a small number of real product images by means of a sample image generation model obtained through training, for training a defect detection model. Moreover, the generated sample image can include not only the image content in the above-mentioned initial image, but also the types of defects corresponding to the mask marks added when generating the mask mark map. Since the initial image is a real product image, when there are defects in the initial image, the generated image corresponding to the initial image generated can include not only the real defects of the product, but also the generated defects. Moreover, when making the above-mentioned mask mark map, various types of defects and mask marks corresponding to various types of defects can be added to the blank image, thereby greatly improving the diversity of generated defects in the generated image and enabling the generated defects in the generated image to have a good data distribution. In this way, by applying the solution provided by the embodiment of the present application, sufficient defect images can be obtained as images for training the defect detection model to improve the performance of the defect detection model.

[0208] In addition, in the present application, there is no limitation on whether the template image has defects, so that in the training process and image generation process of the above-mentioned sample image generation model, the template image and the initial image can be defective images or defect-free images. This makes the above-mentioned training process and image generation process have high flexibility in the selection of template images and initial images, avoiding the impact on model training and image generation due to substandard image selection or difficult image acquisition (such as defect-free images are difficult to acquire).

[0209] Optionally, in a specific implementation manner, an image generation method provided in an embodiment of the present application may further include the following steps:

[0210] Step D: Based on the mask marks of the defects in the initial image and the mask marks existing in the mask mark map, defect labels are added to the defects in the generated image corresponding to the initial image, and based on the defect labels, defects in the generated image corresponding to the initial image that do not meet the preset defect conditions are filtered out.

[0211] In this specific implementation, since the label map of each real product image collected can be obtained, and the initial image is extracted from the multiple real product images collected, the mask label map of the initial image can be obtained, and the label map of the initial image can be annotated with the mask label of the real defect in the initial image. In addition, as mentioned above, the mask label map can be annotated with the mask label of the generated defect that is expected to be generated in the target sample image used for the model training task.

[0212] Based on this, for the target sample image generated by the generator of the above-mentioned sample image generation model, the morphology and type of the generated defects and real defects in the target sample image are known. Therefore, the labeling map and mask labeling map of the initial image can be used to add defect labels to the defects in the target sample image generated corresponding to the initial image.

[0213] The defect labels added above can be represented in a variety of ways, for example, using the minimum circumscribed rectangular box of the defect as a label, or using a polygonal multi-point contour map of the defect as a label.

[0214] Furthermore, depending on the product type, product generation requirements, product applicable products and other scenarios, when performing defect detection on products, the requirements for the form and type of defects marked in the target sample image are also different. Therefore, before using the above target sample image to train the defect detection model, defect conditions can be set in advance according to the actual needs of product defect detection as preset defect conditions to retain defects that meet the actual needs of product defect detection for use in defect detection model training. For example, the defect size, defect shape, and image contrast of the defect area can be set.

[0215] Thus, the defects in the target sample image are filtered using the defect labels added in the target sample image, that is, the defect labels of the defects in the target sample image that do not meet the preset defect conditions are removed. In this way, when the defect detection model is trained using the target sample image after the defect filtering, the defects marked by the filtered defect labels will not be used as defects for image feature learning by the defect detection model, but will only be used as image features for image regions where there are no defects in the target sample image. Therefore, the defect detection model will not learn the image features of the defects marked by the filtered defect labels, and further, the trained defect detection model will not regard the regional defects in the image whose image features are similar to the image features of the defects marked by the filtered defect labels as defects.

[0216] In order to facilitate understanding of an image generation method provided in an embodiment of the present application, and a training method for a sample image generation model as described above, the following can be seen from FIG. 3 (a), FIG. 3 (b) and Figure 4 The specific embodiment shown is described in detail.

[0217] Among them, Figure 3 (a) is an exemplary training flowchart of the sample image generation model, and Figure 3 (b) is another exemplary flow chart of the sample image generation model, wherein the real defect image is the target image, the defect image mask marking is the mask marking image (second marking image) about the target image, the template image mask marking is the mask marking image (first marking image) about the template image, the fused mask marking is the target marking image, the generated defect image is the generated image, and the generator (also called GAN generator) and the discriminator (also called GAN discriminator) are the generator and the discriminator of the sample image generation model.

[0218] As shown in Figure 3 (a), the real defect image has defects 01, 02 and 03, and the defect image mask mark has mask mark 011 corresponding to defect 01, mask mark 021 corresponding to defect 02, and mask mark 031 corresponding to defect 03; the template image has defect 04, and the template image mask mark has mask mark 041 corresponding to defect 04; the fused mask marks have mask mark 012 corresponding to defect 01, mask mark 022 corresponding to defect 02, mask mark 032 corresponding to defect 03, and mask mark 042 corresponding to defect 04 (the mask mark 042 has been adjusted to the specified color, black); the generated defect image has defect 013 corresponding to defect 01, defect 023 corresponding to defect 02, and defect 033 corresponding to defect 03.

[0219] As shown in Figure 3 (b), the real defect image has defects 05, 06 and 07, and the defect image mask mark has mask mark 051 corresponding to defect 05, mask mark 061 corresponding to defect 06, and mask mark 071 corresponding to defect 07; the template image has defect 08, and the template image mask mark has mask mark 081 corresponding to defect 08; the fused mask marks have mask mark 052 corresponding to defect 05, mask mark 062 corresponding to defect 06, mask mark 072 corresponding to defect 07, and mask mark 082 corresponding to defect 08 (the mask mark 082 has been adjusted to the specified color, black); the generated defect image has defect 053 corresponding to defect 05, defect 063 corresponding to defect 06, and defect 073 corresponding to defect 07.

[0220] Figure 4 It is the inference flowchart of the sample image generation model, that is, the generation flowchart of the target sample image (the generated image corresponding to the initial image), wherein the defect mask mark to be generated is the mask mark map, the template map is the initial image, the template map mask mark is the mark map of the initial image, the generated defect map is the target sample image (that is, the generated image corresponding to the initial image), and the generator is the generator of the sample image generation model.

[0221] like Figure 4As shown, the mask marks of the defect map to be generated include mask mark 111, mask mark 121 and mask mark 131; the template map has defect 14; the template map mask mark includes mask mark 141; the generated defect map includes defect 112, defect 122, defect 132 and defect 142.

[0222] Furthermore, in the above-mentioned Figures 3(a), 3(b) and Figure 4 In the specific embodiment shown, the sample image generation model is a generative adversarial network.

[0223] As shown in Figure 3(a), Figure 3(b) and Figure 4 As shown, the training process of the sample image generation model may include the following Step 1-Step 7, and the generation process of the target sample image may include the following Step 8-Step 12. Specifically:

[0224] Step 1: Sample collection. In this specific embodiment, only a small amount of defect images need to be collected, and the real defects in the collected defect images are masked and annotated to obtain a labeling map of each collected defect image. Among them, different types of defects are annotated with masks of different colors (for example, mask mark 011, mask mark 021 and mask mark 031 in the mask mark of the defect image in Figure 3 (a) are three mask marks with different gray levels, and mask mark 051, mask mark 061 and mask mark 071 in the mask mark of the defect image in Figure 3 (b) are three mask marks with different gray levels, that is, Figures 3 (a) and 3 (b) are characterized by different gray levels. Three mask marks), and the collected defect image can be called a defect sample, and the annotated mask is called a mask mark. Among them, in order to improve the training effect of the sample image generation model, when the real defects in the collected defect image are masked, it is usually necessary to carefully annotate, so that the above mask annotation can also be called mask precision annotation.

[0225] Step 2: Training image pairing. Randomly select two images from the defect images collected in Step 1 to form an image pair, one of which is called the template image and the other is called the real defect image (target image). Finally, four input quantities are obtained, namely the real defect image (target image), the defect image mask mark (the mask mark image or the second mark image about the target image), the template image and the template image mask mark (the mask mark image or the first mark image about the template image), which are the four leftmost input quantities shown in Figure 3 (a) and Figure 3 (b).

[0226] Step 3: Process through the filtering module. Input the defect image mask mark and the template image mask mark (input into the filtering module to determine whether the defect image mask mark (target image mask mark map) and the template image mask mark (template image mask mark map) have overlapping positions of defects. If there is overlap, repeat Step 2 and randomly extract a pair of images again. If there is no overlap of defect positions, proceed to Step 4.

[0227] Step 4: Processing by fusion module. Through the fusion module, the defect image mask mark and the template image mask mark are fused. Specifically: retain the color pixel value of each mask mark of the defect image mask mark (for example, in FIG3 (a), the color pixel value of mask mark 012 is the same as that of mask mark 011, the color pixel value of mask mark 022 is the same as that of mask mark 021, and the color pixel value of mask mark 032 is the same as that of mask mark 031; in FIG3 (b), the color pixel value of mask mark 052 is the same as that of mask mark 051, the color pixel value of mask mark 062 is the same as that of mask mark 061, and the color pixel value of mask mark 072 is the same as that of mask mark 071), and convert each mask mark of the template image mask mark into a specified color, for example, black (for example, the pixel value of mask mark 042 in FIG3 (a) and the pixel value of mask mark 082 in FIG3 (b) are both black). After fusion, the mask labels of all defects in the defect image mask label and the template image mask label are fused into a label image to obtain the fused mask label (target label image). For defects in the template image, at the location of the defect in the fused mask mark (target mark map), a mask mark of a specified color is used to mark it. For example, the black triangle in Figure 3 (a) and Figure 3 (b) is the mask mark of the template image mask mark (template image mask mark map). For defects in the real defect image (target image), at the location of the defect in the fused mask mark (target mark map), a mask mark of a color set for the type of defect is used to mark it. For example, the curves and irregular figures composed of the curves in Figure 3 (a) and Figure 3 (b) are the mask marks in the defect image mask mark (target image mask mark map) (for example, mask mark 011, mask mark 021 and mask mark 031 in Figure 3 (a); mask mark 051, mask mark 061 and mask mark 071 in Figure 3 (b)). The colors set for the mask marks of different types of defects are different from the above-mentioned designated colors to avoid the reserved colors (the reserved colors are the designated colors, exemplified by the colors of the triangles in FIG. 3 (a) and FIG. 3 (b)).

[0228] Step 5: Generator training. The template image and the fused mask mark (target mark image) are input together into the GAN generator of the generative adversarial network for training the generative adversarial network. The generation goal of the GAN generator is: the blank area of ​​the fused mask mark generates the visual pixels corresponding to the reference area of ​​the template image, the colored mask mark part of the fused mask mark (the mask mark of the defect in the real defect image) generates the corresponding type of defect, and the black mask mark part of the fused mask mark (target mark image) performs image repair to restore the visual pixels corresponding to the template image (for example, there are defects 013, 023 and 033 in the generated defect image in Figure 3 (a), and there are defects 053, 063 and 073 in the generated defect image in Figure 3 (b)). In this way, the generated defect image (generated image) is generated and output, and the training goal of the GAN generator is to expect the generated defect image (generated image) to deceive the discriminator in Step 6, and it is expected that the discriminator will discriminate the generated defect image (generated image) as True.

[0229] It should be noted that in the above step 5, the purpose of image repairing the black mask marked part is: to ensure that the generated defect image (generated image) output by the generator can deceive the discriminator as much as possible, and the area responsible for the black mask marked part can be easily judged as a fake image by the discriminator. This design is to ensure that the adversarial training of the generator and the discriminator can proceed normally and stably, while enhancing the learning of structural knowledge and improving the quality of data generation.

[0230] Step 6: Discriminator training. The real defect image (target image) and the generated defect image (generated image) output from Step 5 are input to the GAN discriminator. The training goal of the GAN discriminator is to discriminate the real defect image (target image) as true (True) and the generated defect image (generated image) as false (False); that is, the discriminator outputs the true / false discrimination result.

[0231] As mentioned above, in the above Step 5 and Step 6, the GAN generator generates visual pixels corresponding to the reference area of ​​the template image in the blank area of ​​the fused mask mark, generates defects of the corresponding type in the colored mask mark part of the fused mask mark, and performs image repair on the black mask mark part of the fused mask mark to restore the visual pixels corresponding to the template image, and obtains the generated defect image, and its training purpose is that the discriminator will judge the generated defect image as True. The training purpose of the GAN discriminator is to judge the real defect image as True and the generated defect image as False.

[0232] Step7: Perform iterative training according to Step2-Step6. When the number of iterations reaches the set value, stop training and output the best model, that is, obtain the trained sample image generation model.

[0233] Step 8: Reasoning template image selection. Randomly select a picture from the defect image collected in Step 1 as the template image (initial image) for generating the target sample image, and obtain the labeling picture of the template image (initial image) for generating the target sample image as the template image labeling mask of the template image (initial image) for generating the target sample image.

[0234] Step 9: Defect mask image generation. To generate a sample image, a mask mark of the defect image to be generated (mask mark image) needs to be prepared. In the training process and Figure 4 In the inference process of the sample image generation model shown, the functions of the fused mask mark (target mark map) in the training process and the defect mask mark (mask mark map) to be generated in the inference process are the same. When preparing the defect mask mark (mask mark map) to be generated, the color of the defect mark mask (mask mark) can be pre-agreed according to the type of defect (for example, Figure 4 In the figure, the mask mark 111, the mask mark 121 and the mask mark 131 are predetermined colors). Therefore, as long as the type of defect is determined, and then the color of the mask mark in the defect mask mark to be generated (mask mark map) is determined, the defect mask mark to be generated (mask mark map) can be generated in a variety of ways. For example, using a computer peripheral such as a mouse, a mask mark of a certain shape can be drawn in a blank image with each determined color, and one or more, one or more can be drawn. For another example, a mask mark of a defect is generated by drawing a function, such as a curve, an ellipse, a polygon, etc.

[0235] Step 10: Generator reasoning. The template image (initial image) used to generate the target sample image and the mask mark of the defect image to be generated (mask mark image) are input into the GAN generator of the generative adversarial network together, so that the GAN generator can output the generated defect image (target sample image).

[0236] The difference between Step 10 and Step 5 in the training process of the above sample images is that the black mask mark is missing in the mask mark of the defect image to be generated (mask mark map). According to Step 5, in Step 10, there is no area in the mask mark of the defect image to be generated (mask mark map) that needs to be repaired, but only the defects need to be generated at the corresponding positions of the colored mask mark parts, and the visual pixels corresponding to the reference area of ​​the template map need to be generated in the blank area. Therefore, the real defect features of the template map (initial image) used to generate the target sample image will still be retained and generated, and new generated defects are additionally generated on this basis. Figure 4As shown in the generated defect map (target sample image) in , the generated single image can contain both real defect features and generated defect features, that is, defect 112, defect 122, defect 132 and defect 142, which improves the sample diversity and ensures that the defects in the sample image have good data distribution.

[0237] Step 11: Label generation. So far, the known information includes: the mask mark of the defect image to be generated (mask mark map) and the template map mask mark (mark map) of the template map (initial image) used to generate the target sample image. Therefore, on the generated defect image (target sample image) output by Step 10, the mask marks of all defects are known, so the code can automatically generate label information to add defect labels to each defect on the generated defect image (target sample image), thereby outputting the target sample image with defect labels. The generated defect label can be a minimum circumscribed regular rectangular box or a polygonal multi-point outline of the defect. Figure 4 As shown in the rightmost image in , the generated defect label is the minimum circumscribed regular rectangular box.

[0238] Step 12: Label filtering. Label filtering is to filter the defect labels of defects that do not meet the defect standards (preset defect conditions) in the sample images according to the pre-conditions (preset defect conditions) of different projects (actual needs of product defect detection). For example, setting the defect size, setting the defect contrast, etc.

[0239] By performing repeated sample image generation model inference according to Step 8-Step 12, a certain number of "generated defect images" can be generated. The "generated defect images" are automatically labeled, and a certain number of sample images are obtained.

[0240] It should be noted that the solution provided by the embodiment of the present application is compatible with the scenario where the template image in the above training process and reasoning process is a defect-free image, that is, in the above training process and reasoning process, if a defect-free image is used as the template image, there is no need to change the solution provided by the embodiment of the present application. Therefore, the minimum condition to be achieved by the solution provided by the embodiment of the present application is: only a small number of defective images are collected.

[0241] As mentioned above, in the solution provided in the embodiment of the present application, a generative adversarial network (a generative adversarial network for training by pairing a template image and a defect image) is used. Among them, the real defect sample (target image) and the mask mark corresponding to the real defect (target image mask mark map), as well as the template sample (template image) and the corresponding defect mask mark in the template sample (template image mask mark map) are paired and input into the generative adversarial network for training, and before training the generative adversarial network, it is necessary to fuse the mask mark corresponding to the real defect (target image mask mark map) and the corresponding defect mask mark in the template sample (template image mask mark map) to obtain the fused mask mark (target mark map), and the fused mask mark (target mark map) will participate in the training process of the above-mentioned generative adversarial network. Among them, the above-mentioned mask mark fusion process can be regarded as a preprocessing process before model training. In addition, the above-mentioned template image is not limited to defect-free images, but can also be defective images (defective images). In the model inference stage, the constructed defect mask (mask labeling map), template sample (template map), and the corresponding defect mask (template map mask labeling map) in the template sample are simultaneously input into the sample image generation model, and new defects are generated on the template sample (template map), without changing the visual features of the original defects on the template sample (template map), thereby generating a large number of defect samples (sample images). The newly generated data can automatically generate defect labels without additional manual annotation. The generated defect labels and defect samples (sample images) can be used for specific defect detection training tasks, such as semantic segmentation, object detection, instance segmentation, etc.

[0242] Among them, the template image mentioned in the solution provided in the embodiment of the present application supports defective images (defective images). By marking the mask marks of the template image and pairing the input to generate the adversarial network, it is possible to avoid the slight defects in some defect-free images that do not meet the degree standard affecting the training results, thereby ensuring the quality of the generated data (sample image). At the same time, the generated single sample image contains both real sample features and generated sample features, and the data diversity of the defects of the sample image is better. In addition, in terms of resource requirements, the workload of collecting a large number of defect-free images and manually selecting "pure" defect-free images in related methods is avoided.

[0243] Corresponding to the training method of a sample image generation model provided in the above-mentioned embodiment of the present application, the embodiment of the present application also provides a training device for a sample image generation model.

[0244] Figure 5 A schematic diagram of the structure of a training device for a sample image generation model provided in an embodiment of the present application, such as Figure 5 As shown, the device comprises:

[0245] The sample image acquisition module 510 is used to acquire a sample image; wherein the sample image includes a template image, a target image, a first mark image, and a second mark image; the first mark image is an image obtained after a predetermined process is performed on the template image, and the second mark image is an image obtained after the predetermined process is performed on the target image, and the predetermined process is a process of setting a mask mark for a defect;

[0246] The model training module 520 is used to train the sample image generation model to be trained using the sample image until the sample image generation model converges; wherein the sample image generation model includes: a generator and a discriminator;

[0247] Each training process of the sample image generation model includes:

[0248] Performing mask mark fusion on the first mark map and the second mark map to obtain a target mark map; wherein the mask mark fusion is used to fuse the mask marks existing in the first mark map and the mask marks existing in the second mark map into the same image, and to represent the mask marks existing in the first mark map in the same image in a preset manner;

[0249] The target label image and the template image are input to the generator, so that the generator performs image fusion on the target label image and the template image to obtain a generated image; wherein the training convergence target of the generator includes: when there is a mask mark represented in the preset manner in the target label image, using the image content of the template image to perform image repair on the target area; the target area is the image area indicated by the mask mark represented in the preset manner;

[0250] The target image and the generated image are input to the discriminator, so that the discriminator makes a true or false judgment on the generated image based on the target image and the generated image.

[0251] Optionally, in a specific implementation, the generator performs image fusion on the target label image and the template image to obtain a generated image, including:

[0252] In a designated area other than the existing mask mark in the target mark image, an image content corresponding to the image content of the reference area of ​​the template image is generated to obtain an intermediate image; wherein the reference area is an image area having the same image position as the designated area;

[0253] Performing a predetermined image processing on the intermediate image to obtain a generated image; wherein the predetermined image processing includes a first sub-processing and a second sub-processing;

[0254] The first sub-processing includes: when there is a mask mark characterized in the preset manner in the target mark map, using the image content of the template map to perform image repair on the target area in the intermediate image;

[0255] The second sub-processing includes: generating a defect of a type indicated by an identification of the mask mark of the second mark map in an image area indicated by the mask mark of the second mark map in the intermediate image according to the form of the mask mark of the second mark map.

[0256] Optionally, in a specific implementation, the template image and the target image are both defect images;

[0257] The step of performing mask label fusion on the first label image and the second label image to obtain a target label image includes:

[0258] On the basis that the mask marks existing in the second label map are retained and the mask marks existing in the first label map are characterized in the preset manner, the mask marks existing in the first label map and the mask marks existing in the second label map are fused into the same image to obtain a target label map.

[0259] Optionally, in a specific implementation, the device further includes:

[0260] An image annotation module is used to collect multiple real product images and perform the predetermined processing on each real product image to obtain a labeled image of each real product image; wherein the multiple real product images at least include: defect images;

[0261] The sample image acquisition module 510 includes:

[0262] The image acquisition submodule is used to acquire a template image and a target image from the multiple real product images, and acquire a labeled image of the template image as a first labeled image, and acquire a labeled image of the target image as a second labeled image, so as to obtain a sample image.

[0263] Optionally, in a specific implementation, the image acquisition submodule is specifically used to:

[0264] Randomly extract two images from the plurality of real product images;

[0265] In the case where the two extracted images include: a defective image and a non-defective image, the extracted non-defective image is determined as a template image, and the extracted defective image is determined as a target image;

[0266] In the case that both the extracted two images are defective images, it is determined whether there is an overlapping area of ​​defects in the two extracted images; if not, one of the extracted images is determined as a template image, and the other extracted image is determined as a target image; otherwise, the two extracted images are ignored, and the process returns to the step of randomly extracting two images from the multiple real product images;

[0267] In the case that both the extracted two images are defect-free images, the two extracted images are ignored, and the process returns to the step of randomly extracting two images from the multiple real product images.

[0268] Optionally, in a specific implementation, the mask mark existing in the first mark map is characterized in the same image in a preset manner, including:

[0269] The color value of the area indicated by the mask mark existing in the first mark map in the same image is adjusted to a specified color value.

[0270] Corresponding to the image generating method provided in the above-mentioned embodiment of the present application, the embodiment of the present application also provides an image generating device. Figure 6 A schematic diagram of the structure of an image generating device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device comprises:

[0271] An image acquisition module 610 is used to acquire a pre-collected real product image as an initial image;

[0272] The marking acquisition module 620 is used to acquire a mask marking map; wherein the mask marking map is generated by adding mask markings of various types of defects to a blank image of the same size as the initial image, and the mask markings of different types of defects are different;

[0273] An image generation module 630 is used to input the initial image and the mask label map into a generator of a pre-trained sample image generation model, so that the generator fuses the initial image and the mask label map to obtain a generated image corresponding to the initial image;

[0274] The generated image corresponding to the initial image includes: defects of the type indicated by each mask mark of the mask mark image, and the sample image generation model is trained based on a sample image generation model training method provided in an embodiment of the present application.

[0275] Optionally, in a specific implementation, the device further includes:

[0276] A label adding module is used to add defect labels to defects in the generated image corresponding to the initial image based on the mask marks of the defects in the initial image and the mask marks existing in the mask mark map, and based on the defect labels, filter defects in the generated image corresponding to the initial image that do not meet preset defect conditions.

[0277] The present application also provides an electronic device, such as Figure 7 As shown, including:

[0278] Memory 701, used for storing computer programs;

[0279] The processor 702 is used to implement the steps of the training method of any sample image generation model and / or the sample image generation method provided in the above-mentioned embodiments of the present application when executing the program stored in the memory 701.

[0280] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 702, the communication interface, and the memory 701 communicate with each other via the communication bus.

[0281] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0282] The communication interface is used for communication between the above electronic device and other devices.

[0283] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0284] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0285] In another embodiment provided in the present application, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, it implements the training method of any sample image generation model and / or the steps of the sample image generation method provided in the above-mentioned embodiments of the present application.

[0286] In another embodiment provided by the present application, a computer program product comprising instructions is also provided. When the computer program product is run on a computer, the computer executes the steps of the training method of any sample image generation model and / or the sample image generation method provided in the above-mentioned embodiments of the present application.

[0287] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0288] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0289] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, electronic device embodiment, computer-readable storage medium embodiment, and computer program product embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0290] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A training method for a sample image generation model, characterized in that: The method comprises: Acquire a sample image; wherein the sample image includes a template image, a target image, a first mark image, and a second mark image; the first mark image is an image obtained after a predetermined process is performed on the template image, and the second mark image is an image obtained after the predetermined process is performed on the target image, and the predetermined process is a process of setting a mask mark for a defect; Using the sample image, training a sample image generation model to be trained until the sample image generation model converges; wherein the sample image generation model includes: a generator and a discriminator; Each training process of the sample image generation model includes: Performing mask mark fusion on the first mark map and the second mark map to obtain a target mark map; wherein the mask mark fusion is used to fuse the mask marks existing in the first mark map and the mask marks existing in the second mark map into the same image, and to represent the mask marks existing in the first mark map in the same image in a preset manner; Inputting the target label image and the template image into the generator, so that the generator performs image fusion on the target label image and the template image to obtain a generated image; The training convergence target of the generator includes: when there is a mask mark represented in the preset manner in the target mark map, using the image content of the template map to perform image repair on the target area; the target area is the image area indicated by the mask mark represented in the preset manner; The target image and the generated image are input to the discriminator, so that the discriminator makes a true or false judgment on the generated image based on the target image and the generated image.

2. The method according to claim 1, characterized in that The generator performs image fusion on the target label image and the template image to obtain a generated image, including: In a designated area other than the existing mask mark in the target mark image, an image content corresponding to the image content of the reference area of ​​the template image is generated to obtain an intermediate image; wherein the reference area is an image area having the same image position as the designated area; Performing a predetermined image processing on the intermediate image to obtain a generated image; wherein the predetermined image processing includes a first sub-processing and a second sub-processing; The first sub-processing includes: when there is a mask mark characterized in the preset manner in the target mark map, using the image content of the template map to perform image repair on the target area in the intermediate image; The second sub-processing includes: generating defects of the type indicated by the other mask marks in the image area indicated by the other mask marks in the intermediate image according to the forms of the other mask marks that are not represented in the preset manner in the target mark map.

3. The method according to claim 1, characterized in that The template image and the target image are both defect images; The step of performing mask label fusion on the first label image and the second label image to obtain a target label image includes: On the basis that the mask marks existing in the second label map are retained and the mask marks existing in the first label map are characterized in the preset manner, the mask marks existing in the first label map and the mask marks existing in the second label map are fused into the same image to obtain a target label map.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Collect multiple real product images, and perform the predetermined processing on each real product image to obtain a labeling image of each real product image; wherein the multiple real product images at least include: defect images; The obtaining of the sample image comprises: A template image and a target image are obtained from the multiple real product images, and a labeled image of the template image is obtained as a first labeled image, and a labeled image of the target image is obtained as a second labeled image to obtain a sample image.

5. The method according to claim 4, characterized in that The acquiring the template image and the target image from the plurality of real product images comprises: Randomly extract two images from the plurality of real product images; In the case where the two extracted images include: a defective image and a non-defective image, the extracted non-defective image is determined as a template image, and the extracted defective image is determined as a target image; In the case that both the extracted two images are defective images, it is determined whether there is an overlapping area of ​​defects in the two extracted images; if not, one of the extracted images is determined as a template image, and the other extracted image is determined as a target image; otherwise, the two extracted images are ignored, and the process returns to the step of randomly extracting two images from the multiple real product images; In the case that both the extracted two images are defect-free images, the two extracted images are ignored, and the process returns to the step of randomly extracting two images from the multiple real product images.

6. The method according to any one of claims 1 to 3, characterized in that: The mask marks existing in the first mark map are characterized in the same image in a predetermined manner, including: The color value of the area indicated by the mask mark existing in the first mark map in the same image is adjusted to a specified color value.

7. An image generation method, characterized in that: The method comprises: Obtain a pre-collected real product image as an initial image; Acquire a mask mark map; wherein the mask mark map is generated by adding mask marks of various types of defects to a blank image of the same size as the initial image, and the mask marks of different types of defects are different; Inputting the initial image and the mask label map into a generator of a pre-trained sample image generation model, so that the generator fuses the initial image and the mask label map to obtain a generated image corresponding to the initial image; The generated image corresponding to the initial image includes: defects of the type indicated by each mask mark of the mask mark image, and the sample image generation model is trained based on the training method of the sample image generation model according to any one of claims 1-6.

8. The method according to claim 7, characterized in that The method further comprises: Based on the mask marks of the defects in the initial image and the mask marks existing in the mask mark map, defect labels are added to the defects in the generated image corresponding to the initial image, and based on the defect labels, defects that do not meet preset defect conditions in the generated image corresponding to the initial image are filtered out.

9. A training device for a sample image generation model, characterized in that: The device comprises: A sample image acquisition module, used to acquire a sample image; wherein the sample image includes a template image, a target image, a first mark image, and a second mark image; the first mark image is an image obtained after a predetermined process is performed on the template image, and the second mark image is an image obtained after the predetermined process is performed on the target image, and the predetermined process is a process of setting a mask mark for a defect; A model training module, used to train a sample image generation model to be trained using the sample image until the sample image generation model converges; wherein the sample image generation model includes: a generator and a discriminator; Each training process of the sample image generation model includes: Performing mask mark fusion on the first mark map and the second mark map to obtain a target mark map; wherein the mask mark fusion is used to fuse the mask marks existing in the first mark map and the mask marks existing in the second mark map into the same image, and to represent the mask marks existing in the first mark map in the same image in a preset manner; The target label image and the template image are input to the generator, so that the generator performs image fusion on the target label image and the template image to obtain a generated image; wherein the training convergence target of the generator includes: when there is a mask mark represented in the preset manner in the target label image, using the image content of the template image to perform image repair on the target area; the target area is the image area indicated by the mask mark represented in the preset manner; The target image and the generated image are input to the discriminator, so that the discriminator makes a true or false judgment on the generated image based on the target image and the generated image.

10. An image generating device, characterized in that: The device comprises: An image acquisition module is used to acquire a real product image collected in advance as an initial image; A marking acquisition module, used to acquire a mask marking map; wherein the mask marking map is generated by adding mask markings of various types of defects to a blank image of the same size as the initial image, and the mask markings of different types of defects are different; An image generation module, used for inputting the initial image and the mask label map into a generator of a pre-trained sample image generation model, so that the generator fuses the initial image and the mask label map to obtain a generated image corresponding to the initial image; The generated image corresponding to the initial image includes: defects of the type indicated by each mask mark of the mask mark image, and the sample image generation model is trained based on the training method of the sample image generation model according to any one of claims 1-6.

11. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-6 and / or claims 7-8 when executing a program stored in a memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 6 and / or claims 7 to 8 is implemented.

Images