A defect image generation method, device, electronic device and storage medium

By fusion of defect masks and close-up images generated by adversarial training, the problem of lack of real defect images is solved. The generated defect images are used to train defect detection models, improving training efficiency.

CN114972268BActive Publication Date: 2024-12-06HANGZHOU HIKVISION DIGITAL TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210606077.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-31
Publication Date
2024-12-06
Estimated Expiration
2042-05-31

AI Technical Summary

Technical Problem

In the prior art, the defect detection model requires a large number of defect image samples and high acquisition cost, and there is a lack of effective defect image generation method.

Method used

Through pre-trained defect mask generation model and close-up image generation model, adversarial training is used to generate target defect masks and close-up images of target defects corresponding to target noise images, and fuse them into the real image to generate target defect images with real visual effects .

Benefits of technology

The problem of lack of real defect image samples is solved. The generated defect images can be used to train defect detection models, reducing the need for real defect image acquisition and improving the training efficiency of defect detection models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114972268B_ABST
    Figure CN114972268B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention provides a defect image generation method, device, electronic device and storage medium, the method comprising: inputting a target noise image into a pre-trained defect mask generation model to generate a target defect mask corresponding to the target noise image; inputting the target defect mask into a pre-trained close-up image generation model to generate a target defect close-up image corresponding to the target defect mask; acquiring a real image including a target to be simulated, and fusing the target defect close-up image into the real image to obtain a target defect image. The defect mask generation model is obtained based on adversarial training of a mask corresponding to a real defect image and a mask corresponding to a random noise image, and can generate a target defect mask with a realistic appearance based on the target noise image, and then the close-up image generation model can generate a target defect close-up image with a realistic appearance, and the target defect close-up image is fused onto the target to be simulated in the real image to obtain a target defect image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image generation and simulation, and in particular to a method, device, electronic equipment and storage medium for generating a defect image. Background Art

[0002] Defect detection using computer vision technology is a widely used technology with extremely high practical application significance. Defects can include cracks and potholes on the road surface, as well as scratches, cracks, and defects on the surface of workpieces.

[0003] The commonly used defect detection method is to use a defect detection model to detect whether an object has defects. The number of defect image samples required to train the defect detection model is very large, but the number of real defect image samples that can be obtained is often small and the acquisition cost is high. Therefore, a method that can generate defect images is urgently needed to solve this problem. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a method, device, electronic device and storage medium for generating a defect image, so as to generate a defect image. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present invention provides a method for generating a defect image, the method comprising:

[0006] Inputting a target noise image into a pre-trained defect mask generation model to generate a target defect mask corresponding to the target noise image, wherein the target noise image is a noise image corresponding to a defect of a preset category, and the defect mask generation model is obtained based on adversarial training of a first defect mask corresponding to a real defect image and a second defect mask corresponding to a random noise image;

[0007] Inputting the target defect mask into a pre-trained close-up image generation model to generate a target defect close-up image corresponding to the target defect mask, wherein the close-up image generation model is obtained based on adversarial training between the real defect image and a first defect close-up image corresponding to the first defect mask;

[0008] A real image including a target to be simulated is acquired, and the target defect close-up image is fused into the real image to obtain a target defect image, wherein the target to be simulated is a target with defects to be simulated.

[0009] Optionally, the target defect mask has corresponding defect category information;

[0010] After the step of fusing the target defect close-up image with the real image to obtain the target defect image, the method further includes:

[0011] Based on the fusion position of the target defect close-up image in the real image and the defect category information, a defect label corresponding to the target defect image is determined.

[0012] Optionally, the defect mask generation model includes a first generation network and a first discrimination network;

[0013] The training method of the defect mask generation model includes:

[0014] Acquire a real defect image, and encode the real defect image based on defects included in the real defect image to obtain a first defect mask;

[0015] Inputting the random noise image into the first generation network, and generating a second defect mask based on the noise characteristics of the random noise image;

[0016] The first defect mask and the second defect mask are input into the first discrimination network, and based on the difference between the image features of the first defect mask and the image features of the second defect mask, the model parameters of the defect mask generation model are adjusted until the defect mask generation model converges.

[0017] Optionally, the step of encoding the real defect image based on the defects included in the real defect image includes:

[0018] Identifying a defective area in the real defect image;

[0019] For each pixel in the real defect image, determining whether the pixel belongs to the defect area;

[0020] If yes, setting the pixel value of the pixel point in the first defect mask having the same pixel position as the pixel point as the first pixel value;

[0021] If not, the pixel value of the pixel point in the first defect mask having the same pixel position as the pixel point is set to the second pixel value.

[0022] Optionally, the step of determining, for each pixel in the real defect image, whether the pixel belongs to the defect area includes:

[0023] For each pixel point in the real defect image, generating a connecting line segment between the pixel point and an image edge of the real defect image;

[0024] Determining the number of intersections between the connecting line segment and the boundary of the defect area;

[0025] If the pixel point is located at the boundary of the defect area or the number is an odd number, it is determined that the pixel point belongs to the defect area;

[0026] If the pixel point is not located at the boundary of the defective area and the number is an even number, it is determined that the pixel point does not belong to the defective area.

[0027] Optionally, the close-up image generation model includes a second generation network and a second discriminant network;

[0028] The training method of the close-up image generation model includes:

[0029] Inputting the first defect mask into the second generation network, and generating a first defect close-up image based on image features of the first defect mask;

[0030] The real defect image and the first defect close-up image are input into the second discriminant network, and based on the difference between the image features of the real defect image and the image features of the first defect close-up image, the model parameters of the close-up image generation model are adjusted until the close-up image generation model converges.

[0031] Optionally, the step of fusing the target defect close-up image into the real image includes:

[0032] Determine a target fusion area corresponding to the target to be simulated in the real image;

[0033] Taking the position in the target fusion area corresponding to the defect in the target defect mask as the position to be fused;

[0034] Based on the position to be fused, the defect in the target defect close-up image is fused into the real image.

[0035] Optionally, the step of fusing the defect in the target defect close-up image into the real image based on the position to be fused includes:

[0036] Determining whether there is a defect at the position to be fused;

[0037] If there is no defect, the pixel value of the pixel point at the position to be fused is replaced with the pixel value of the pixel point at the corresponding position in the close-up image of the target defect;

[0038] If there is a defect, return to the step of determining the target fusion area corresponding to the target to be simulated in the real image.

[0039] Optionally, the step of replacing the pixel value of the pixel point at the position to be fused with the pixel value of the pixel point at the corresponding position in the target defect close-up image includes:

[0040] Based on the pixel values ​​of the pixel points at the positions to be fused, histogram matching is performed on the target defect close-up image to obtain a matched target defect close-up image;

[0041] The pixel value of the pixel point at the position to be fused is replaced by the pixel value of the pixel point at the corresponding position of the matched target defect close-up image.

[0042] Optionally, the step of determining whether there is a defect at the position to be fused includes:

[0043] Encoding the real image to obtain a mask corresponding to the real image;

[0044] Determine whether a pixel value of each pixel point in the mask corresponding to the position to be fused is the same as a pixel value of a pixel point included in a defect area in the target defect mask;

[0045] If they are not the same, it is determined that there is no defect at the position to be fused;

[0046] If there are pixels with the same pixel value, it is determined that there is a defect at the position to be fused.

[0047] Optionally, after the step of replacing the pixel value of the pixel point at the position to be fused with the pixel value of the pixel point at the corresponding position in the target defect close-up image, the method further includes:

[0048] Return to the step of determining the target fusion area corresponding to the target to be simulated in the real image until the number of return operations reaches a preset number.

[0049] Optionally, the method further includes:

[0050] Based on the category of each defect in the target defect image, determining a connected region corresponding to each category of defects;

[0051] For each category of defects, the defect label corresponding to the target defect image is determined based on the boundary and category of the connected region corresponding to the defect of the category.

[0052] Optionally, the step of determining, for each category of defects, a defect label corresponding to the target defect image based on a boundary and a category of a connected region corresponding to the defect of that category, includes:

[0053] For each category of defects, the bounding box of the connected area corresponding to the defect of that category is calculated according to the following formula:

[0054]

[0055] Among them, E xThe connected area corresponding to the defect of this category includes the coordinates of the pixels in the x-axis direction of the target defect image, E y The connected area corresponding to the defect of this category includes the coordinates of the pixels in the y-axis direction of the target defect image, (x min ,y min ) is the coordinate of the upper left corner vertex of the bounding box, (x max ,y max ) are the coordinates of the lower right corner vertex of the bounding box;

[0056] The bounding box and the category corresponding to the defect of the category are determined as the defect label corresponding to the defect of the category in the target defect image.

[0057] In a second aspect, an embodiment of the present invention provides a device for generating a defect image, the device comprising:

[0058] A defect mask generation module, used for inputting a target noise image into a pre-trained defect mask generation model to generate a target defect mask corresponding to the target noise image, wherein the target noise image is a noise image corresponding to a defect of a preset category, and the defect mask generation model is obtained based on adversarial training of a first defect mask corresponding to a real defect image and a second defect mask corresponding to a random noise image;

[0059] a defect close-up image generation module, configured to input the target defect mask into a pre-trained close-up image generation model to generate a target defect close-up image corresponding to the target defect mask, wherein the close-up image generation model is obtained based on adversarial training between the real defect image and a first defect close-up image corresponding to the first defect mask;

[0060] The image fusion module is used to obtain a real image including a target to be simulated, and fuse the target defect close-up image into the real image to obtain a target defect image, wherein the target to be simulated is a target with defects to be simulated.

[0061] Optionally, the target defect mask has corresponding defect category information;

[0062] The device also includes:

[0063] The defect label generation module is used to determine the defect label corresponding to the target defect image based on the fusion position of the target defect close-up image in the real image and the defect category information.

[0064] Optionally, the defect mask generation model includes a first generation network and a first discrimination network;

[0065] The defect mask generation module comprises:

[0066] A first defect mask generating submodule is used to obtain a real defect image, and encode the real defect image based on defects included in the real defect image to obtain a first defect mask;

[0067] A second defect mask generation submodule, configured to input a random noise image into the first generation network, and generate a second defect mask based on noise characteristics of the random noise image;

[0068] The first parameter adjustment submodule is used to input the first defect mask and the second defect mask into the first discriminant network, and adjust the model parameters of the defect mask generation model based on the difference between the image features of the first defect mask and the image features of the second defect mask until the defect mask generation model converges.

[0069] Optionally, the first defect mask generating submodule includes:

[0070] A defect area recognition unit, used to recognize the defect area in the real defect image;

[0071] A pixel point judgment unit, used for judging, for each pixel point in the real defect image, whether the pixel point belongs to the defect area;

[0072] a first pixel value setting unit, configured to, if yes, set the pixel value of the pixel point in the first defect mask having the same pixel position as the pixel point to the first pixel value;

[0073] The second pixel value setting unit is configured to set the pixel value of the pixel point having the same pixel position as the pixel point in the first defect mask to the second pixel value if no.

[0074] Optionally, the pixel point determination unit includes:

[0075] A connecting line segment generating subunit, used for generating, for each pixel point in the real defect image, a connecting line segment between the pixel point and the image edge of the real defect image;

[0076] An intersection number determination subunit, used to determine the number of intersections between the connecting line segment and the boundary of the defect area;

[0077] A first judgment subunit, configured to determine that the pixel point belongs to the defect area if the pixel point is located at a boundary of the defect area or the number is an odd number;

[0078] The second judgment subunit is used to determine that the pixel does not belong to the defect area if the pixel is not located at the boundary of the defect area and the number of times is an even number.

[0079] Optionally, the close-up image generation model includes a second generation network and a second discriminant network;

[0080] The defect close-up image generation module comprises:

[0081] a first defect close-up image generating submodule, configured to input the first defect mask into the second generating network, and generate a first defect close-up image based on image features of the first defect mask;

[0082] The second parameter adjustment submodule is used to input the real defect image and the first defect close-up image into the second discriminant network, and adjust the model parameters of the close-up image generation model based on the difference between the image features of the real defect image and the image features of the first defect close-up image until the close-up image generation model converges.

[0083] Optionally, the image fusion module includes:

[0084] A target fusion region determination submodule is used to determine a target fusion region corresponding to the target to be simulated in the real image;

[0085] A submodule for determining a position to be fused, used for taking a position in the target fusion area corresponding to a defect in the target defect mask as a position to be fused;

[0086] The image fusion submodule is used to fuse the defect in the target defect close-up image into the real image based on the position to be fused.

[0087] Optionally, the image fusion submodule includes:

[0088] A defect judgment unit, used to judge whether there is a defect at the position to be fused;

[0089] A pixel value replacement unit, used for replacing the pixel value of the pixel point at the position to be fused with the pixel value of the pixel point at the corresponding position in the target defect close-up image if there is no defect;

[0090] The first returning unit is used to return to the step of determining the target fusion area corresponding to the target to be simulated in the real image if there is a defect.

[0091] Optionally, the pixel value replacement unit includes:

[0092] A histogram matching subunit, configured to perform histogram matching on the target defect close-up image based on the pixel values ​​of the pixel points at the position to be fused, so as to obtain a matched target defect close-up image;

[0093] The pixel value replacement subunit is used to replace the pixel value of the pixel point at the position to be fused with the pixel value of the pixel point at the corresponding position of the matched target defect close-up image.

[0094] Optionally, the defect judgment unit includes:

[0095] A mask generation subunit, used for encoding the real image to obtain a mask corresponding to the real image;

[0096] A pixel value judgment subunit, used to judge whether the pixel value of each pixel point corresponding to the position to be fused in the mask is the same as the pixel value of the pixel point included in the defect area in the target defect mask;

[0097] A first defect judgment subunit is used to determine that there is no defect at the position to be fused if they are not the same;

[0098] The second defect judgment subunit is used to determine that there is a defect at the position to be fused if there are pixel points with the same pixel value.

[0099] Optionally, the device further comprises:

[0100] A second return unit is used to return to the step of determining the target fusion area corresponding to the target to be simulated in the real image after the step of replacing the pixel value of the pixel point at the position to be fused with the pixel value of the pixel point at the corresponding position in the target defect close-up image, until the number of return operations reaches a preset number.

[0101] Optionally, the device further comprises:

[0102] A connected region determining unit, configured to determine, based on the category of each defect in the target defect image, a connected region corresponding to each category of defects;

[0103] The defect label generating unit is used to determine the defect label corresponding to the target defect image based on the boundary and category of the connected area corresponding to the defect of each category.

[0104] Optionally, the defect label generating unit includes:

[0105] The bounding box determination subunit is used to calculate the bounding box of the connected area corresponding to each category of defects according to the following formula:

[0106]

[0107] Among them, E x The connected area corresponding to the defect of this category includes the coordinates of the pixels in the x-axis direction of the target defect image, E yThe connected area corresponding to the defect of this category includes the coordinates of the pixels in the y-axis direction of the target defect image, (x min ,y min ) is the coordinate of the upper left corner vertex of the bounding box, (x max ,y max ) are the coordinates of the lower right corner vertex of the bounding box;

[0108] The defect label generating subunit is used to determine the bounding box and category corresponding to the defect of the category as the defect label corresponding to the defect of the category in the target defect image.

[0109] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0110] Memory, used to store computer programs;

[0111] The processor is used to implement any method step described in the first aspect when executing the program stored in the memory.

[0112] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of the first aspects are implemented.

[0113] Beneficial effects of the embodiments of the present invention:

[0114] In the solution provided by an embodiment of the present invention, the electronic device can input a target noise image into a pre-trained defect mask generation model to generate a target defect mask corresponding to the target noise image, wherein the target noise image is a noise image corresponding to a defect of a preset category, and the defect mask generation model is obtained through adversarial training of a first defect mask corresponding to a real defect image and a second defect mask corresponding to a random noise image; the target defect mask can be input into a pre-trained close-up image generation model to generate a target defect close-up image corresponding to the target defect mask, wherein the close-up image generation model is obtained through adversarial training of the real defect image and the first defect close-up image corresponding to the first defect mask; and then a real image including a target to be simulated is obtained, and the target defect close-up image is fused into the real image to obtain a target defect image, wherein the target to be simulated is a target with defects to be simulated. Since the first defect mask corresponds to the real defect image, and the second defect mask image corresponds to the random noise image, the defect mask generation model obtained through adversarial training can generate a target defect mask with a realistic appearance based on the target noise image. Similarly, the close-up image generation model obtained through adversarial training can generate a target defect close-up image with a realistic appearance based on the target defect mask, and then fuse the target defect close-up image to the target to be simulated in the real image to obtain the target defect image. The above method can generate a defect image. Of course, implementing any product or method of the present invention does not necessarily require all of the advantages described above to be achieved at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0115] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0116] Figure 1 A flowchart of a method for generating a defect image provided by an embodiment of the present invention;

[0117] Figure 2(a) is based on Figure 1 A schematic diagram of a target defect mask of the illustrated embodiment;

[0118] Figure 2(b) is based on Figure 1 A schematic diagram of another target defect mask of the illustrated embodiment;

[0119] FIG2(c) is a schematic diagram of a close-up image of a target defect corresponding to the target defect mask shown in FIG2(a);

[0120] FIG2(d) is a schematic diagram of a close-up image of a target defect corresponding to the target defect mask shown in FIG2(b);

[0121] Figure 3(a) is based on Figure 1 A schematic diagram of a real image of the illustrated embodiment;

[0122] Figure 3(b) is based on Figure 1 A schematic diagram of a target defect image of the illustrated embodiment;

[0123] Figure 4 Based on Figure 1 A schematic diagram of a process of obtaining a target defect image, a defect label, and a target defect mask in the illustrated embodiment;

[0124] Figure 5 Based on Figure 1 A specific flow chart of a method for training a defect mask generation model in the illustrated embodiment;

[0125] Figure 6 Based on Figure 5 A schematic diagram of the structure of a deep convolutional network of the illustrated embodiment;

[0126] Figure 7 Based on Figure 6 A schematic diagram of the structure of a self-attention mechanism of the illustrated embodiment;

[0127] Figure 8 Based on Figure 5 A specific flow chart of step S501 in the illustrated embodiment;

[0128] Figure 9(a) is based on Figure 8 A schematic diagram of a real defect image of the illustrated embodiment;

[0129] FIG9( b ) is a schematic diagram of a first defect mask obtained by encoding the real defect image shown in FIG9( a );

[0130] Figure 9(c) is based on Figure 8 A schematic diagram of another real defect image of the illustrated embodiment;

[0131] FIG9( d ) is a schematic diagram of a first defect mask obtained by encoding the real defect image shown in FIG9( c );

[0132] Fig.10 Based on Figure 8 A specific flow chart of step S802 in the illustrated embodiment;

[0133] Fig.11 Based on Figure 1 A specific flow chart of a training method for a close-up image generation model in the illustrated embodiment;

[0134] Fig.12 Based on Figure 1A specific flow chart of step S103 in the illustrated embodiment;

[0135] Fig.13 Based on Fig.12 A specific flow chart of step S1203 in the illustrated embodiment;

[0136] Fig.14 Based on Fig.13 A specific flow chart of step S1302 in the illustrated embodiment;

[0137] Fig.15 Based on Fig.13 A specific flow chart of step S1301 in the illustrated embodiment;

[0138] Figure 16(a) is based on Fig.13 A schematic diagram of a target defect image fused with multiple defects according to the embodiment shown;

[0139] FIG16( b ) is a schematic diagram of an original image corresponding to the target defect image shown in FIG16( a );

[0140] FIG. 16( c ) is a schematic diagram of a mask corresponding to the target defect image shown in FIG. 16( a );

[0141] Fig.17 Based on Fig.13 A flow chart of determining a defect label corresponding to a target defect image according to the embodiment shown;

[0142] Fig.18 A schematic diagram of the structure of a defect image generating device provided by an embodiment of the present invention;

[0143] Fig.19 A schematic structural diagram of another defect image generating device provided by an embodiment of the present invention;

[0144] Fig. 20 The present invention is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0145] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on the present invention belong to the scope of protection of the present invention.

[0146] In order to generate a defect image, an embodiment of the present invention provides a defect image generation method, device, electronic device, computer-readable storage medium and computer program product. The following first introduces a defect image generation method provided by an embodiment of the present invention.

[0147] like Figure 1 As shown, a method for generating a defect image, the method comprising:

[0148] S101, inputting a target noise image into a pre-trained defect mask generation model to generate a target defect mask corresponding to the target noise image;

[0149] The target noise image is a noise image corresponding to a defect of a preset category, and the defect mask generation model is obtained through adversarial training between a first defect mask corresponding to a real defect image and a second defect mask corresponding to a random noise image.

[0150] S102, inputting the target defect mask into a pre-trained close-up image generation model to generate a target defect close-up image corresponding to the target defect mask;

[0151] The close-up image generation model is obtained based on adversarial training between the real defect image and the first defect close-up image corresponding to the first defect mask.

[0152] S103, acquiring a real image including the target to be simulated, and fusing the target defect close-up image into the real image to obtain a target defect image;

[0153] The target to be simulated is a target that needs to be simulated but has defects.

[0154] It can be seen that in the scheme provided by the embodiment of the present invention, the electronic device can input the target noise image into the pre-trained defect mask generation model to generate a target defect mask corresponding to the target noise image, wherein the target noise image is a noise image corresponding to a defect of a preset category, and the defect mask generation model is obtained through adversarial training of a first defect mask corresponding to the real defect image and a second defect mask corresponding to the random noise image; the target defect mask can be input into the pre-trained close-up image generation model to generate a target defect close-up image corresponding to the target defect mask, wherein the close-up image generation model is obtained through adversarial training of the real defect image and the first defect close-up image corresponding to the first defect mask; and then a real image including the target to be simulated is obtained, and the target defect close-up image is fused into the real image to obtain a target defect image, wherein the target to be simulated is a target with defects to be simulated. Since the first defect mask corresponds to the real defect image, and the second defect mask image corresponds to the random noise image, the defect mask generation model obtained through adversarial training can generate a realistic target defect mask based on the target noise image. Similarly, the close-up image generation model obtained through adversarial training can generate a realistic target defect close-up image based on the target defect mask, and then fuse the target defect close-up image to the target to be simulated in the real image to obtain the target defect image. The above method can generate a defect image.

[0155] Using a defect detection model to detect whether an object has defects is a commonly used defect detection method. However, the number of defect image samples required to train the defect detection model is very large. In order to generate a defect image, an embodiment of the present invention provides a defect image generation method. In step S101, the electronic device can input the target noise image into a pre-trained defect mask generation model to generate a target defect mask corresponding to the target noise image.

[0156] Different objects are usually located in different environments and usage scenarios, so the categories of defects can be diverse. Taking highway defects as an example, the defect categories may include road cracks, potholes, etc.; taking workpiece defects as an example, the defect categories may include scratches, cracks, and defects on the workpiece surface. In an embodiment of the present invention, different categories of defects may correspond to different noise images, and the electronic device may obtain noise images corresponding to defects of preset categories, that is, target noise images. For example, the target noise image may be a noise image corresponding to road cracks, or a noise image corresponding to scratches on the workpiece surface, etc. The target noise image may be determined based on actual needs such as the application scenario of the defect image to be generated, and is not specifically limited here. As for the generation method of the target noise image, the target noise image may be generated by the electronic device, or may be generated in other models or devices and then obtained by the electronic device, and is not limited here.

[0157] The defect mask generation model can be a generative adversarial network model. During the training of the defect mask generation model, the input of the defect mask generation model can be a first defect mask corresponding to a real defect image and a second defect mask corresponding to a random noise image. Since the first defect mask is generated based on a real defect image, the authenticity and accuracy of the defects therein are relatively high. The second defect mask is generated based on a random noise image. When the defect mask generation model is not trained, the authenticity and accuracy of the defects in the second defect mask are relatively low compared to the defects in the first defect mask. Through adversarial training, it is simulated that the first defect mask with higher authenticity and accuracy is used to guide the training of the second defect mask, so that the second defect mask generated based on the random noise image can gradually approach the first defect mask, and the authenticity and accuracy of the defects in the second defect mask are continuously improved. Thus, the defect mask generation model obtained by training can generate a corresponding target defect mask based on the target noise image. Although the target defect mask is a mask of a virtual defect image, it looks more realistic.

[0158] Through the above method, the target defect mask generated can be a single-channel black-and-white image. For example, based on the target noise image corresponding to the defect category of road potholes, the schematic diagram of the generated target defect mask can be shown in Figure 2(a), where the defect area 201 is a virtual mask of road potholes, and the non-defect area 202 is a virtual mask of normal road surface; based on the target noise image corresponding to the defect category of scratches on the workpiece surface, the schematic diagram of the generated target defect mask can be shown in Figure 2(b), where the defect area 203 is a virtual mask of scratches on the workpiece surface, and the non-defect area 204 is a virtual mask of the normal surface of the workpiece.

[0159] In the above step S102, the electronic device may input the target defect mask into a pre-trained close-up image generation model to generate a target defect close-up image corresponding to the target defect mask.

[0160] The close-up image generation model can be a conditional adversarial generative network model. During the training of the close-up image generation model, the input of the close-up image generation model can be a real defect image and a first defect close-up image corresponding to the first defect mask. The real defect image can be an artificially collected defect image, in which the image of the defect is clear, real, and consistent with the style of the background. Although the first defect mask is generated based on the real defect image, the authenticity and accuracy are both high, but the first defect mask has lost part of the image information compared with the real defect image, and the first defect close-up image is generated based on the first defect mask that has lost part of the image information. When the close-up image generation model is not trained, compared with the image of the defect in the real defect image, the image of the defect in the first defect close-up image may not be clear enough, less authentic, or not consistent with the style of the background. Through the adversarial training method, it is simulated to use the real defect image to guide the training of the first defect close-up image, so that the first defect close-up image generated based on the random noise image can gradually approach the real defect image, and continuously improve the clarity, authenticity and style consistency of the defect image in the first defect close-up image with the background. The close-up image generation model thus trained can generate a corresponding target defect close-up image based on the target defect mask. Although the target defect close-up image is a virtual defect image, it looks more realistic and is consistent with the style of the background.

[0161] Through the above method, the generated target defect close-up image can be a multi-channel image with the same length and width as the corresponding target defect mask. For example, the target defect mask shown in Figure 2(a) is input into the trained close-up image generation model, and the schematic diagram of the generated target defect close-up image can be shown in Figure 2(c), wherein the image of the defect area 205 is an image of a virtual pothole on the road surface, and the image of the non-defect area 206 is an image of a virtual normal road surface; the target defect mask shown in Figure 2(b) is input into the trained close-up image generation model, and the schematic diagram of the generated target defect close-up image can be shown in Figure 2(d), wherein the image of the defect area 207 is an image of a virtual scratch on the surface of the workpiece, and the image of the non-defect area 208 is an image of a virtual normal surface of the workpiece.

[0162] Furthermore, in the above step S103, the electronic device may acquire a real image including the target to be simulated, and fuse the target defect close-up image into the real image to obtain the target defect image.

[0163] In the above method, the electronic device has obtained a close-up image of the target defect that looks more realistic and consistent with the style of the background. However, the target defect close-up image is only a close-up image of a virtual defect, which should be a part of the image of the target with defects to be simulated, and may not be suitable for direct training of the defect detection model. The electronic device can fuse the target defect close-up image with the real image to obtain the target defect image, and the generated target defect image can be used to train the defect detection model. Among them, the target defect close-up image can be fused with the real image using a variety of image fusion methods commonly used in digital image processing, which are not specifically limited here.

[0164] The target with defects to be simulated is the target to be simulated. For example, when the close-up image of the target defect is an image of a pothole on the road, the target to be simulated can be a cement road, asphalt road or other road surface; when the close-up image of the target defect is an image of scratches on the surface of a workpiece, the target to be simulated can be a workpiece such as a bearing or a wheel, which is not specifically limited here. The real image acquired by the electronic device can include the target to be simulated, so when the close-up image of the target defect is fused into the real image, the close-up image of the target defect can be fused into the area where the target to be simulated is located. Taking the generation of a road defect image as an example, after the electronic device obtains a virtual close-up image of a road defect, it can obtain a real image, which can be shown in Figure 3(a), where the target to be simulated is a road surface, and the electronic device can fuse the close-up image of the road defect into the area where the road surface is located in the real image, and the obtained target defect image can be shown in Figure 3(b).

[0165] By adopting the scheme provided by the embodiment of the present invention, the electronic device can input the target noise image into the pre-trained defect mask generation model to generate a target defect mask corresponding to the target noise image. The target defect mask is input into the pre-trained close-up image generation model to generate a target defect close-up image corresponding to the target defect mask. Then, a real image including the target to be simulated is obtained, and the target defect close-up image is fused into the real image to obtain a target defect image. Compared with the common method of manually collecting defect images and defect-free images and using grayscale images to fuse the defect images into defect-free images, the scheme provided by the embodiment of the present invention can be based on the defect mask generation model and the close-up image generation model obtained by training, and when generating defect images, there is no need to collect real defect images again, and target defect images of various defect categories can be generated.

[0166] As an implementation method of an embodiment of the present invention, the target defect mask may have corresponding defect category information; after the step of fusing the target defect close-up image with the real image to obtain the target defect image, the method may further include:

[0167] Based on the fusion position of the target defect close-up image in the real image and the defect category information, a defect label corresponding to the target defect image is determined.

[0168] Since the target noise image is a noise image corresponding to a preset category of defect, the electronic device inputs the target noise image into a pre-trained defect mask generation model, and the target defect mask corresponding to the generated target noise image may have defect category information corresponding to the target noise image. The target defect mask is input into a pre-trained close-up image generation model, and the target defect close-up image corresponding to the generated target defect mask may also have corresponding defect category information.

[0169] Furthermore, after fusing the target defect close-up image to the real image, the electronic device can obtain the fusion position of the target defect close-up image in the real image, and can also obtain the defect category information of the target defect close-up image, which is the same as the defect category information of the corresponding target defect mask. The electronic device can determine the defect label corresponding to the target defect image based on the fusion position of the target defect close-up image in the real image and the defect category information.

[0170] In one embodiment, the electronic device can obtain the target defect image, defect label and target defect mask in the above manner. The schematic diagram of the process of obtaining the above data can be as follows: Figure 4 As shown, the electronic device can input the target noise image into the pre-trained defect mask generation model to obtain the corresponding target defect mask and defect category information; then input the target defect mask into the pre-trained close-up image generation model to obtain the corresponding target defect close-up image and defect category information; and then the target defect close-up image can be fused into the real image. In the process of generating a defect image once, the target defect image, defect label and target defect mask can be obtained. The obtained target defect mask and defect label can be used to expand the data set of training samples of the defect detection model, and can also be applied to tasks such as image segmentation or target detection.

[0171] In this embodiment, the electronic device can determine the defect label corresponding to the target defect image based on the fusion position of the target defect close-up image in the real image and the defect category information. Therefore, after the target defect image is generated, the target defect image and the defect label can be obtained, and there is no need to manually calibrate the target defect image, thereby improving the calibration efficiency of the target defect image.

[0172] As an implementation method of an embodiment of the present invention, the above-mentioned defect mask generation model may include a first generation network and a first discrimination network; Figure 5 As shown, the training method of the above defect mask generation model may include:

[0173] S501, acquiring a real defect image, and encoding the real defect image based on defects included in the real defect image to obtain a first defect mask.

[0174] The electronic device can obtain a real defect image, which may include defective and non-defective parts. Based on the defects included in the real defect image, the electronic device can encode the real defect image to obtain a first defect mask.

[0175] In one embodiment, the electronic device can identify defects in a real defect image, and when generating a first defect mask corresponding to the real defect image, the pixel values ​​of the pixel points in the areas corresponding to the defect and non-defective parts are set to different pixel values. For example, the pixel value of the area corresponding to the defect can be set to 1, and the pixel value of the pixel points in the area corresponding to the non-defective part can be set to 0. No specific limitation is made here.

[0176] S502: Input the random noise image into the first generation network, and generate a second defect mask based on the noise characteristics of the random noise image.

[0177] The random noise image may be a randomly generated noise image having corresponding noise characteristics. For example, the random noise image N may be a normally distributed matrix with a mean of 0 and a variance of 1, where N = {n1, ...n n}, where the random noise n n It can be a vector of 1×1×512. The first generation network of the defect mask generation model can generate a second defect mask based on the noise feature of the random noise image.

[0178] In one embodiment, the first generation network may include a deep convolutional network and a self-attention mechanism. For example, a schematic diagram of the deep convolutional network may be as follows: Figure 6 As shown in the figure, the Transpose Conv layer is a transposed convolution layer, which can perform transposed convolution upsampling on the input random noise image. The deep convolution network also includes a self-attention mechanism SelfAttention. The schematic diagram of the self-attention mechanism can be shown as follows Figure 7 As shown in the figure, the input layer is the input, the output is the output, and the self-attention mechanism includes three branches. The Conv 1×1 layer of each branch represents a convolution layer with a convolution kernel size of 1×1, and the Transpose layer represents the transposition of the input. Represents matrix multiplication, non-corresponding elements are multiplied, and the Softmax layer can be used for classification tasks.

[0179] For example, the electronic device can encode the real defect image to obtain the first defect mask M. After the random noise image N is input into the first generation network, the first generation network can transform the random noise distribution to obtain the single channel matrix D corresponding to the random noise image N = {D1,...,D n}, the single-channel matrix D is the second defect mask corresponding to the random noise image, and the second defect mask D can have the same length and width as the first defect mask M.

[0180] S503: Input the first defect mask and the second defect mask into the first discrimination network, and adjust the model parameters of the defect mask generation model based on the difference between the image features of the first defect mask and the image features of the second defect mask until the defect mask generation model converges.

[0181] The first defect mask is generated based on a real defect image, and the second defect mask is generated based on a random noise image, and the difference between the image features of the two may be large. The first discriminant network can discriminate the difference between the image features, and then based on the difference between the image features of the first defect mask and the image features of the second defect mask, the electronic device can adjust the model parameters of the defect mask generation model until the defect mask generation model converges.

[0182] In one embodiment, the first discriminant network may be a fully convolutional discriminant network, which may output a scalar corresponding to an input. For example, the second defect mask D corresponding to the input random noise image, the output of the first discriminant network may be F1={f1,...,f n}, f n represents the image features corresponding to the second defect mask; the first defect mask M corresponding to the real defect image is input, and the output of the first discriminant network is T1 = {t,...,t n}, t n Represents the image features corresponding to the first defect mask. Then, the mean square error loss can be used as the loss function for training the first discriminant network, and the loss function of the defect mask generation model can be:

[0183]

[0184] Among them, mean means taking the mean of the corresponding data, Loss dis1 Represents the loss of the first discriminant network, which is used to update the model parameters of the first discriminant network. adv1 Represents the first adversarial loss, which is used to update the model parameters of the first generative network.

[0185] Thus, the electronic device can continuously reduce the difference between the image features of the second defect mask and the first defect mask by alternately updating the model parameters of the first discriminant network and the model parameters of the first generation network, until the number of iterations of the first defect mask and the second defect mask reaches a preset number, or the accuracy of the output result of the defect mask generation model is greater than the preset accuracy, and it can be determined that the defect mask generation model converges. The specific method of adjusting the model parameters of the defect mask generation model can be a gradient descent algorithm, a stochastic gradient descent algorithm, etc., which is not specifically limited here.

[0186] In this embodiment, the electronic device can encode the acquired real defect image to obtain a first defect mask, and input the random noise image into the first generation network to generate a second defect mask, and then input the first defect mask and the second defect mask into the first discrimination network, and adjust the model parameters of the defect mask generation model based on the difference between the image features of the first defect mask and the image features of the second defect mask until the defect mask generation model converges. The defect mask generation model obtained by training can generate a target defect mask with a realistic appearance based on the target noise image.

[0187] As an implementation method of the embodiment of the present invention, Figure 8 As shown, the step of encoding the real defect image based on the defects included in the real defect image may include:

[0188] S801, identifying a defect area in the real defect image.

[0189] A real defect image may include defective and non-defective parts. Based on target recognition, feature detection and other methods commonly used in digital image processing, electronic devices can identify defective areas in real defect images.

[0190] In one embodiment, the electronic device may acquire a certain number of real defect images I = {i1, ..., i n}, and then identify the defect area in the real defect image, and mark the outline of the defect area with a polygon. The outline can be represented by the vertex set C = {c0, c1, ..., c n-1} means, where c i =(x i ,y i ), which is the vertex c i The coordinates of i = 0, 1, 2, ..., n-1. Two adjacent vertices can form a boundary e i , thus we get the boundary set E = (e0, e1, ..., e n-1 ), the boundary set E can be represented by vertices in the following form:

[0191]

[0192] S802, for each pixel in the real defect image, determine whether the pixel belongs to the defect area; if yes, execute step S803; if no, execute step S804.

[0193] The electronic device has identified the defect area in the real defect image, and can then determine, for each pixel in the real defect image, whether the pixel belongs to the defect area.

[0194] S803: Set the pixel value of the pixel point in the first defect mask having the same pixel position as the pixel point as the first pixel value.

[0195] If the pixel point belongs to the defective area, the electronic device can set the pixel value of the pixel point in the first defect mask with the same pixel position as the pixel point to the first pixel value. The first pixel value is used to identify that the pixel point at the corresponding position in the real defect image belongs to the defective area.

[0196] S804: Set the pixel value of the pixel point in the first defect mask having the same pixel position as the pixel point as the second pixel value.

[0197] If the pixel does not belong to the defective area, the electronic device may set the pixel value of the pixel at the same pixel position as the pixel in the first defect mask to a second pixel value. The second pixel value is used to identify that the pixel at the corresponding position in the real defective image does not belong to the defective area, and the second pixel value is different from the first pixel value. For example, the first pixel value may be set to 1, and the second pixel value may be set to 0, which is not specifically limited here.

[0198] For example, the real defect image may be shown in FIG9(a). When it is a road surface defect, the first defect mask obtained by encoding in the above manner may be shown in FIG9(b); for another example, the real defect image may be shown in FIG9(c). When it is a workpiece surface defect, the first defect mask obtained by encoding in the above manner may be shown in FIG9(d). The first defect mask shown in FIG9(b) may be consistent with the length and width of the real defect image shown in FIG9(a), and the first defect mask shown in FIG9(d) may be consistent with the length and width of the real defect image shown in FIG9(c).

[0199] In this embodiment, the electronic device can identify the defective area in the real defective image, and for each pixel in the real defective image, determine whether the pixel belongs to the defective area. If it belongs to the defective area, the electronic device can set the pixel value of the pixel at the same pixel position as the pixel in the first defect mask to the first pixel value; if it does not belong to the defective area, the electronic device can set the pixel value of the pixel at the same pixel position as the pixel in the first defect mask to the second pixel value. In this way, the defective area and the non-defective area in the first defect mask obtained by encoding can be clearly distinguished.

[0200] As an implementation method of the embodiment of the present invention, Fig.10 As shown, the above step of determining, for each pixel in the real defect image, whether the pixel belongs to the defect area may include:

[0201] S1001 , for each pixel point in the real defect image, generating a connecting line segment between the pixel point and an image edge of the real defect image.

[0202] The electronic device can use the ray method to determine whether the pixel point belongs to the defect area, that is, to determine whether the pixel point belongs to the defect area by the number of intersections between the connecting line segment between the pixel point and the image edge and the boundary of the defect area. Therefore, for each pixel point in the real defect image, the electronic device can generate a connecting line segment between the pixel point and the image edge of the real defect image.

[0203] For example, the height of the real defect image is h, the width is w, and for the pixel to be tested (x i ,y k ),x i ∈[0,h),y k ∈[0,w), the electronic device can generate a line segment l from the pixel to be tested to the edge of the real defect image ik =((x0,y k ),(x i ,y k )), line segment l ik is the connecting line segment between the pixel point and the image edge of the real defect image.

[0204] S1002, determine the number of intersections between the connecting line segment and the boundary of the defect area; if the pixel point is located at the boundary of the defect area or the number is an odd number, execute step S1003; if the pixel point is not located at the boundary of the defect area and the number is an even number, execute step S1004.

[0205] The electronic device may determine the number of intersections between the connecting line segment and the boundary of the defective area. For example, the electronic device obtains a polygon boundary set E=(e0, e1, ..., e n-1 ), we can calculate the line segment l ik The number of crossings of the polygon boundary set E is the number of intersections between the connecting line segment and the boundary of the defect area.

[0206] S1003: Determine whether the pixel point belongs to the defective area.

[0207] If the pixel is located at the boundary of the defect area, it can be considered that the pixel belongs to the defect area; if the number of intersections between the connecting line segment and the boundary of the defect area is an odd number, such as 1, 3, 5, etc., it can be understood that the line segment is connected from the defect area to the image edge of the real defect image, that is, the pixel belongs to the defect area.

[0208] S1004: Determine that the pixel does not belong to the defective area.

[0209] If the pixel is not located at the boundary of the defect area and the number is an even number, such as 2, 4, 6, etc., it can be understood that the line segment is connected from the outside of the defect area to the image edge of the real defect image, that is, the pixel does not belong to the defect area.

[0210] In this embodiment, the electronic device can generate a connecting line segment between each pixel in the real defect image and the image edge of the real defect image, and determine whether the pixel belongs to the defect area based on the number of intersections between the connecting line segment and the boundary of the defect area. Thus, by calculation, it can be accurately determined whether each pixel in the real defect image belongs to the defect area.

[0211] As an implementation of an embodiment of the present invention, the close-up image generation model may include a second generation network and a second discrimination network; Fig.11 As shown, the training method of the close-up image generation model may include:

[0212] S1101, inputting the first defect mask into the second generation network, and generating a first defect close-up image based on image features of the first defect mask.

[0213] The first defect mask is obtained by encoding a real defect image by an electronic device, and has image features corresponding to the real defect image. The second generation network of the close-up image generation model can generate a first defect close-up image based on the image features of the first defect mask.

[0214] In one embodiment, the second generation network may include a deep convolutional neural network and a pixel-to-pixel high-definition image generation network, which may generate a corresponding multi-channel first defect close-up image based on an input single-channel first defect mask, and the generated first defect close-up image may have the same length and width as the first defect mask.

[0215] S1102, inputting the real defect image and the first defect close-up image into the second discriminant network, and adjusting the model parameters of the close-up image generation model based on the difference between the image features of the real defect image and the image features of the first defect close-up image until the close-up image generation model converges.

[0216] Although the first defect mask is generated based on the real defect image, and has high authenticity and accuracy, the first defect mask has lost some image information compared to the real defect image, and the first defect close-up image is generated based on the first defect mask that has lost some image information, and the difference between the image features of the real defect image and the image features of the real defect image may be large. The second discriminant network can discriminate the difference between the image features, and then based on the difference between the image features of the first defect close-up image and the image features of the real defect image, the electronic device can adjust the model parameters of the close-up image generation model until the close-up image generation model converges.

[0217] In one embodiment, the second discriminant network may be a discriminant network based on a deep convolutional neural network, which may output a scalar corresponding to an input. For example, when a first defect close-up image corresponding to a first defect mask is input, the output of the second discriminant network is F2={f1,...,f n}, f n represents the image features corresponding to the first defect close-up image; if a real defect image is input, the output of the first discriminant network is T2 = {t,...,t n}, t n Represents the image features corresponding to the real defect image. Then the mean square error loss can be used as the loss function for the second discriminant network training, and the loss function of the close-up image generation model can be:

[0218]

[0219] Among them, mean means taking the mean of the corresponding data, Loss dis2 Represents the loss of the second discriminant network, which is used to update the model parameters of the second discriminant network. adv2 Represents the second adversarial loss, which is used to update the model parameters of the second generation network.

[0220] Thus, the electronic device can continuously reduce the difference between the image features of the first defect close-up image and the real defect image by alternately updating the model parameters of the second discriminant network and the model parameters of the second generation network, until the number of iterations of the first defect close-up image and the real defect image reaches a preset number, or the accuracy of the output result of the close-up image generation model is greater than the preset accuracy, and it can be determined that the close-up image generation model converges. The specific method of adjusting the model parameters of the close-up image generation model can be a gradient descent algorithm, a stochastic gradient descent algorithm, etc., which is not specifically limited here.

[0221] In this embodiment, the electronic device can input the first defect mask into the second generation network, generate the first defect close-up image based on the image features of the first defect mask, and then input the real defect image and the first defect close-up image into the second discriminant network, and then adjust the model parameters of the close-up image generation model based on the difference between the image features of the real defect image and the image features of the first defect close-up image until the close-up image generation model converges. The trained close-up image generation model can generate a realistic-looking target defect close-up image based on the target defect mask.

[0222] As an implementation method of the embodiment of the present invention, Fig.12 As shown, the step of fusing the target defect close-up image into the real image may include:

[0223] S1201, determining a target fusion area corresponding to the target to be simulated in the real image.

[0224] The real image acquired by the electronic device includes a target to be simulated, and the target to be simulated is a target with defects to be simulated, so the close-up image of the target defect can be fused into the position of the target to be simulated in the real image. The electronic device can determine the target fusion area corresponding to the target to be simulated in the real image, and the target fusion area is the area in the real image used to fuse the close-up image of the target defect.

[0225] In one implementation, the length and width of the target fusion area may be the same as the target defect close-up image. For example, the target defect close-up image is a rectangle with a length of h and a width of w. The electronic device may randomly select a partial area from the location of the target to be simulated in the real image to determine the target fusion area. The coordinates of the upper left corner of the target fusion area may be determined to be p1=(x1, y1) and the coordinates of the lower right corner may be determined to be p2=(x2, y2), thereby satisfying:

[0226]

[0227] Make the length and width of the target fusion area the same as the target defect close-up image.

[0228] S1202, taking the position in the target fusion area corresponding to the defect in the target defect mask as the position to be fused.

[0229] The target defect close-up image includes both defective part images and non-defective part images. In the process of fusing the target defect close-up image into the real image, only the defective part image needs to be fused into the real image, and the non-defective part image may not be fused into the real image. Therefore, the electronic device can use the position in the target fusion area corresponding to the defect in the target defect mask as the position to be fused. The position to be fused is the position of the defective part image in the real image used to fuse the target defect close-up image.

[0230] S1203: Based on the position to be fused, the defect in the target defect close-up image is fused into the real image.

[0231] Then, the electronic device can fuse the defect in the target defect close-up image to the position to be fused in the real image to obtain the target defect image.

[0232] In this embodiment, the electronic device can determine the target fusion area corresponding to the target to be simulated in the real image, and use the position in the target fusion area corresponding to the defect in the target defect mask as the position to be fused, and then fuse the defect in the target defect close-up image into the real image based on the position to be fused. By accurately determining the position of defect fusion, the fusion effect of the defect in the target defect close-up image and the target to be simulated and the authenticity of the target defect image are improved.

[0233] As an implementation method of the embodiment of the present invention, Fig.13 As shown, the step of fusing the defect in the target defect close-up image into the real image based on the position to be fused may include:

[0234] S1301, determine whether there is a defect at the position to be fused, if there is no defect, execute step S1302; if there is a defect, execute step S1303.

[0235] The real image acquired by the electronic device includes the target to be simulated, and the target to be simulated in the real image may already have defects. When the target to be simulated has defects, in the process of fusing the defects in the target defect close-up image to the position to be fused in the real image, if it is not determined whether there are defects at the position to be fused, there may be multiple defects superimposed at some positions of the target defect image. At this time, the label of the target defect image is difficult to determine, and the accuracy and authenticity of the target defect image will also be reduced, so the electronic device can determine whether there are defects at the position to be fused.

[0236] S1302: Replace the pixel value of the pixel point at the position to be fused with the pixel value of the pixel point at the corresponding position in the target defect close-up image.

[0237] If there is no defect at the position to be fused, it means that if the defect in the target defect close-up image is fused into the real image, there will be no superposition of multiple defects, and the target defect image obtained by fusion has high accuracy and authenticity, and the defect in the target defect close-up image can be fused into the real image. The electronic device can replace the pixel value of the pixel point at the position to be fused with the pixel value of the pixel point at the corresponding position in the target defect close-up image.

[0238] For example, the target defect close-up image is i x , close-up image of target defect i x The corresponding target defect mask is m x , the real image is g y , the real image g y The target fusion area corresponding to the target to be simulated in is area s, and area s can be processed as follows:

[0239]

[0240] in, Represents the target defect mask m x The position where the defect is located in the region s is the position to be fused. The processing method can replace the pixel value of the pixel point at the position to be fused with the pixel value of the pixel point at the corresponding position in the close-up image of the target defect.

[0241] S1303, returning to the step of determining the target fusion area corresponding to the target to be simulated in the real image.

[0242] If there is a defect at the position to be fused, it means that if the defect in the target defect close-up image is fused into the real image, multiple defects may be superimposed, and the accuracy and authenticity of the fused target defect image are low. At this time, the position to be fused is not suitable for fusing the defect in the target defect close-up image. The electronic device can return to the step of determining the target fusion area corresponding to the target to be simulated in the real image, and reselect the target fusion area.

[0243] In this embodiment, the electronic device can determine whether there is a defect at the position to be fused. If there is no defect, the pixel value of the pixel point at the position to be fused can be replaced with the pixel value of the pixel point at the corresponding position in the target defect close-up image; if there is a defect, the electronic device can return to the step of determining the target fusion area corresponding to the target to be simulated in the real image, and reselect the target fusion area. This avoids the phenomenon of defect superposition in the target defect image and improves the accuracy and authenticity of the target defect image.

[0244] As an implementation method of the embodiment of the present invention, Fig.14 As shown, the step of replacing the pixel value of the pixel point at the position to be fused with the pixel value of the pixel point at the corresponding position in the close-up image of the target defect may include:

[0245] S1401, performing histogram matching on the target defect close-up image based on the pixel values ​​of the pixel points at the position to be fused, to obtain a matched target defect close-up image.

[0246] Although the target defect close-up image generated based on the target defect mask looks more realistic, it is still different from the real image. The image style of the target defect close-up image and the real image may not be consistent. The electronic device can perform histogram matching on the target defect close-up image based on the pixel value of the pixel point at the position to be fused to obtain the matched target defect close-up image. Compared with the original target defect close-up image, the matched target defect close-up image has a more consistent image style with the real image and a better fusion effect.

[0247] For example, the target defect close-up image is i x , the real image is g y , the real image g y The target fusion area corresponding to the target to be simulated in is area s. The process of histogram matching of the target defect close-up image is to transform the pixel values ​​of the pixels in area s as follows:

[0248]

[0249]

[0250] z q =G -1 (s k )

[0251] Among them, r is the real image g y The pixel value of the pixel point in the region s, z is the close-up image of the target defect i x , s is the pixel value of the pixel point in the intermediate image, and p is the probability density function.

[0252] S1402: Replace the pixel value of the pixel point at the position to be fused with the pixel value of the pixel point at the corresponding position of the matched target defect close-up image.

[0253] Furthermore, the electronic device can replace the pixel value of the pixel point at the position to be fused with the pixel value of the pixel point at the corresponding position of the matched target defect close-up image, and fuse the defect in the target defect close-up image to the position to be fused in the real image.

[0254] In this embodiment, the electronic device can perform histogram matching on the target defect close-up image based on the pixel values ​​of the pixel points at the position to be fused, obtain the matched target defect close-up image, and then replace the pixel values ​​of the pixel points at the position to be fused with the pixel values ​​of the pixel points at the corresponding position of the matched target defect close-up image. The image style of the matched target defect close-up image and the real image is more unified, so that the fusion effect is better, the fused target defect image looks more realistic, and the detection errors caused by image fusion can be avoided.

[0255] As an implementation method of the embodiment of the present invention, Fig.15 As shown, the above step of judging whether there is a defect at the position to be fused may include:

[0256] S1501, encode the real image to obtain a mask corresponding to the real image.

[0257] Since the defect area in the mask is usually more accurate and more obvious, and the target defect close-up image also has a corresponding target defect mask, by comparing the mask corresponding to the real image with the target defect mask, it can be determined whether there is a defect at the position to be fused. After the electronic device obtains the real image, it can encode the real image to obtain the mask corresponding to the real image.

[0258] In one embodiment, the electronic device can determine whether there is a defective area in the real image, and then for each pixel in the real image, it can be determined whether the pixel belongs to the defective area. If the pixel belongs to the defective area, the pixel value of the pixel at the same pixel position as the pixel in the mask can be set to the first pixel value; if the pixel does not belong to the defective area, the pixel value of the pixel at the same pixel position as the pixel in the first defect mask can be set to the second pixel value. Thus, a mask corresponding to the real image can be obtained. The second pixel value is different from the first pixel value. For example, the first pixel value can be set to 1, and the second pixel value can be set to 0. The pixel value setting method can correspond to the pixel value setting method in the target defect mask, and no specific limitation is made here.

[0259] S1502, determine whether the pixel value of each pixel point in the mask corresponding to the position to be fused is the same as the pixel value of the pixel points included in the defect area in the target defect mask; if they are not the same, execute step S1503; if there are pixel points with the same pixel value, execute step S1504.

[0260] After obtaining the mask corresponding to the real image, for each pixel point in the mask corresponding to the position to be fused, the electronic device can determine whether the pixel value of the pixel point is the same as the pixel value of the pixel point included in the defect area in the target defect mask.

[0261] S1503, determining that there is no defect at the position to be fused.

[0262] If the pixel values ​​of each pixel are different, it means that the pixels included in the defect area in the target defect mask are identified as the defect area, and each pixel corresponding to the position to be fused in the corresponding mask is identified as the non-defect area, that is, there is no defect at the position to be fused, and the defect in the target defect close-up image can be fused to the position to be fused.

[0263] S1504, determining that there is a defect at the position to be fused.

[0264] If there are pixels with the same pixel value, it means that the pixels included in the defect area in the target defect mask are marked as defect areas, and among each pixel corresponding to the position to be fused in the corresponding mask, there are pixels marked as defect areas, that is, there is a defect in the position to be fused. At this time, the position to be fused is not suitable for fusing the defect in the target defect close-up image, and the electronic device can return to the step of determining the target fusion area corresponding to the target to be simulated in the real image, and reselect the target fusion area.

[0265] In one embodiment, the number of defect category information is preset to distinguish defects of different categories, and a corresponding relationship between defect category information and pixel values ​​can also be established. After the step of replacing the pixel value of the pixel point at the position to be fused with the pixel value of the pixel point at the corresponding position in the target defect close-up image, for the defect area in mask A corresponding to the target defect image after fusion processing, the electronic device can replace the pixel value of the pixel point in the defect area in mask A with the pixel value corresponding to the defect category information, and can perform the following processing on mask A:

[0266]

[0267] in, Represents the target defect mask m x The location of the defect in the image, c is the target defect mask m x The pixel value corresponding to the defect category information of the fused defect can be obtained by merging the target defect image into a mask corresponding to the defect category information of the fused defect.

[0268] In this embodiment, the electronic device can encode the real image to obtain the mask corresponding to the real image, and then determine whether the pixel value of each pixel point corresponding to the position to be fused in the mask is the same as the pixel value of the pixel point included in the defect area in the target defect mask. If they are not the same, it can be determined that there is no defect at the position to be fused; if there are pixels with the same pixel value, it can be determined that there is a defect at the position to be fused. By comparing the mask corresponding to the real image with the target defect mask, the electronic device can quickly and accurately determine whether there is a defect at the position to be fused.

[0269] As an implementation of an embodiment of the present invention, after the step of replacing the pixel value of the pixel point at the position to be fused with the pixel value of the pixel point at the corresponding position in the close-up image of the target defect, the method may further include:

[0270] Return to the step of determining the target fusion area corresponding to the target to be simulated in the real image until the number of return operations reaches a preset number.

[0271] Based on the above scheme, defects in the target defect close-up image can be fused into the real image, and by fusing defects multiple times, the fused target defect image can include multiple defects, which can simulate the situation where the target in the real scene has multiple defects or multiple types of defects.

[0272] After the step of replacing the pixel value of the pixel point at the position to be fused with the pixel value of the pixel point at the corresponding position in the close-up image of the target defect, the electronic device can return to the step of determining the target fusion area corresponding to the target to be simulated in the real image until the number of return operations reaches a preset number. Correspondingly, if there is a defect at the position to be fused, the electronic device can also return to the step of determining the target fusion area corresponding to the target to be simulated in the real image. The preset number of times can be set based on the actual situation such as the size of the target to be fused and the close-up image of the target defect, application requirements, etc., and is not specifically limited here.

[0273] For example, the preset number of times may be 5 times. After the defect in the target defect close-up image is first fused into the real image, or after it is determined that there is a defect at the position to be fused, the electronic device may return to the step of determining the target fusion area corresponding to the target to be simulated in the real image, and then determine the new target fusion area again, and determine whether there is a defect at the position to be fused. When the number of return operations reaches 5 times, that is, the sum of the number of operations of fusion of the defect in the target defect close-up image into the real image and the number of operations of determining that there is a defect at the position to be fused reaches 5 times, the electronic device may stop determining the target fusion area corresponding to the target to be simulated in the real image. Thus, the electronic device may select up to 5 target fusion areas, fuse up to 5 defects in the target defect close-up images into the real image, and determine that the step of defect fusion of the real image has been completed; accordingly, the electronic device may determine whether there is a defect at the position to be fused up to 5 times. If it is determined that there is a defect at the position to be fused each time, it may also be determined that the step of defect fusion of the real image has been completed.

[0274] In the above manner, a schematic diagram of a target defect image obtained by fusing defects in multiple target defect close-up images can be shown in FIG16(a), wherein the target defect image includes defects in five target defect close-up images; a schematic diagram of an original image corresponding to the target defect image can be shown in FIG16(b); and a schematic diagram of a mask corresponding to the target defect image can be shown in FIG16(c).

[0275] In this embodiment, after replacing the pixel values ​​of the pixels at the positions to be fused with the pixel values ​​of the pixels at the corresponding positions in the close-up image of the target defect, the electronic device can return to the step of determining the target fusion area corresponding to the target to be simulated in the real image until the number of return operations reaches a preset number. Thus, the defects in multiple close-up images of target defects can be fused into the real image, and multiple defects will not be superimposed in the fused target defect image, thereby improving the authenticity of the target defect image.

[0276] As an implementation method of the embodiment of the present invention, Fig.17 As shown, the above method may also include:

[0277] S1701, based on the category of each defect in the target defect image, determine the connected area corresponding to each category of defects.

[0278] In the scheme of the embodiment of the present invention, the electronic device can fuse defects in multiple target defect close-up images into a real image, and the categories of the fused multiple defects may be different. Therefore, when calibrating the defect labels of the target defect image, due to the existence of defects of multiple defect categories, the defect labels obtained by unified calibration may not be accurate, which is not conducive to the subsequent training of the defect detection model. Therefore, each category of defects in the target defect image can be calibrated separately.

[0279] The electronic device can determine the connected area corresponding to each defect category in the target defect image based on the category of each defect in the target defect image. The connected area can be an area corresponding to a single defect in the target defect image, or a defect area formed by multiple defects of the same category being connected to each other.

[0280] In one embodiment, the number of defect category information is pre-set to distinguish defects of different categories, and a correspondence between defect category information and pixel values ​​can also be established. For mask A corresponding to the target defect image after fusion processing, the electronic device can replace the pixel values ​​of the pixel points in the defect area with the pixel values ​​corresponding to the defect category information. Therefore, in mask A, the pixel values ​​of the pixel points in defect areas of different categories are different. The electronic device can search mask A in a breadth-first search manner to obtain defect areas of different categories corresponding to pixel points with different pixel values. The electronic device can construct a connected area set E = {e1,...,e2,...,e3,...,e4,...,e5,...,e6,...,e7,...,e8,...,e9,...,e10,...,e11,...,e12,...,e13,...,e14,...,e15,...,e16,...,e17,...,e18,...,e19,...,e20,...,e11,...,e12,...,e13,...,e14,...,e15,...,e16,...,e17,...,e18,...,e19,...,e19,...,e20,...,e19,...,e20,...,e1 n}, where n represents the number of connected regions, e i Represents a connected area, and each connected area can include a coordinate set of all pixel points in the connected area.

[0281] Through the above-mentioned method for constructing the connected area of ​​the mask of the target defect image, the electronic device can determine the connected area corresponding to each category of defects based on the connected area of ​​the mask of the target defect image and the category of each defect in the target defect image.

[0282] S1702, for each category of defects, based on the boundary and category of the connected region corresponding to the defect of that category, determine the defect label corresponding to the target defect image.

[0283] For each category of defects, the electronic device can determine the boundaries of the connected area corresponding to the defect of that category. The category of the connected area corresponding to the defect of that category is the defect category of the fused target defect close-up image, that is, the defect category of the target defect mask. Based on the boundaries and categories of the connected area corresponding to the defect of that category, the electronic device can determine the defect label corresponding to the target defect image. The defect label can include a connected area corresponding to a defect category, or multiple connected areas with different defect categories, where each connected area has corresponding defect category information.

[0284] In this embodiment, the electronic device can determine the connected area corresponding to each category of defects based on the category of each defect in the target defect image, and then determine the defect label corresponding to the target defect image based on the boundary and category of the connected area corresponding to the defect of each category. When there are multiple categories of defects in the target defect image, the boundary and category of the connected area corresponding to each category of defects can be determined respectively, so that the defect label corresponding to the target defect image is more accurate.

[0285] As an implementation manner of an embodiment of the present invention, the above step of determining the defect label corresponding to the target defect image based on the boundary and category of the connected region corresponding to the defect of each category may include:

[0286] For each category of defects, the bounding box of the connected area corresponding to the defect of that category is calculated according to the following formula:

[0287]

[0288] Among them, E x The connected area corresponding to the defect of this category includes the coordinates of the pixels in the x-axis direction of the target defect image, E y The connected area corresponding to the defect of this category includes the coordinates of the pixels in the y-axis direction of the target defect image, (x min ,y min ) is the coordinate of the upper left corner vertex of the bounding box, (x max ,y max ) are the coordinates of the lower right corner vertex of the bounding box.

[0289] The bounding box and the category corresponding to the defect of the category are determined as the defect label corresponding to the defect of the category in the target defect image.

[0290] For each category of defects, based on the x-axis coordinates of the pixels included in the connected area corresponding to the defect of that category in the target defect image, the electronic device can determine that the maximum value of the x-axis coordinates is x max , the minimum value is xmin Similarly, based on the y-axis coordinates of the pixels included in the connected area corresponding to the defect of this category in the target defect image, the electronic device can determine that the maximum value of the y-axis coordinate is y max , the minimum value is y min . Then (x min ,y min ) as the coordinates of the upper left corner of the bounding box, (x max ,y max ) as the coordinates of the lower right corner vertex of the bounding box to determine the range of the bounding box corresponding to the defect of this category.

[0291] In this embodiment, the electronic device can determine the maximum and minimum values ​​of the x-axis coordinates and the y-axis coordinates for each category of defects based on the coordinates of the pixels in the connected area corresponding to the category of defects in the target defect image, and then determine the coordinates of the upper left corner vertex and the lower right corner vertex of the bounding box corresponding to the category of defects. The range of the bounding box corresponding to the category of defects can be accurately determined, and the defect label corresponding to the target defect image can be obtained.

[0292] Corresponding to the above-mentioned defect image generation method, an embodiment of the present invention further provides a defect image generation device. The defect image generation device provided by an embodiment of the present invention is introduced below.

[0293] like Fig.18 As shown, a device for generating a defect image, the device comprising:

[0294] The defect mask generation module 1801 is used to input the target noise image into the pre-trained defect mask generation model to generate a target defect mask corresponding to the target noise image.

[0295] The target noise image is a noise image corresponding to a defect of a preset category, and the defect mask generation model is obtained through adversarial training between a first defect mask corresponding to a real defect image and a second defect mask corresponding to a random noise image.

[0296] The defect close-up image generation module 1802 is used to input the target defect mask into a pre-trained close-up image generation model to generate a target defect close-up image corresponding to the target defect mask.

[0297] The close-up image generation model is obtained based on adversarial training between the real defect image and the first defect close-up image corresponding to the first defect mask.

[0298] The image fusion module 1803 is used to obtain a real image including the target to be simulated, and fuse the target defect close-up image into the real image to obtain a target defect image.

[0299] The target to be simulated is a target that needs to be simulated but has defects.

[0300] It can be seen that in the scheme provided by the embodiment of the present invention, the electronic device can input the target noise image into the pre-trained defect mask generation model to generate a target defect mask corresponding to the target noise image, wherein the target noise image is a noise image corresponding to a defect of a preset category, and the defect mask generation model is obtained through adversarial training of a first defect mask corresponding to the real defect image and a second defect mask corresponding to the random noise image; the target defect mask can be input into the pre-trained close-up image generation model to generate a target defect close-up image corresponding to the target defect mask, wherein the close-up image generation model is obtained through adversarial training of the real defect image and the first defect close-up image corresponding to the first defect mask; and then a real image including the target to be simulated is obtained, and the target defect close-up image is fused into the real image to obtain a target defect image, wherein the target to be simulated is a target with defects to be simulated. Since the first defect mask corresponds to the real defect image, and the second defect mask image corresponds to the random noise image, the defect mask generation model obtained through adversarial training can generate a realistic target defect mask based on the target noise image. Similarly, the close-up image generation model obtained through adversarial training can generate a realistic target defect close-up image based on the target defect mask, and then fuse the target defect close-up image to the target to be simulated in the real image to obtain the target defect image. The above method can generate a defect image.

[0301] As an implementation method of the embodiment of the present invention, Fig.19 As shown, the target defect mask may have corresponding defect category information; the device may further include:

[0302] The defect label generating module 1901 is used to determine the defect label corresponding to the target defect image based on the fusion position of the target defect close-up image in the real image and the defect category information.

[0303] As an implementation of an embodiment of the present invention, the defect mask generation model may include a first generation network and a first discrimination network; the defect mask generation module 1801 may include:

[0304] The first defect mask generating submodule is used to obtain a real defect image, and encode the real defect image based on defects included in the real defect image to obtain a first defect mask.

[0305] The second defect mask generating submodule is used to input the random noise image into the first generating network, and generate a second defect mask based on the noise characteristics of the random noise image.

[0306] The first parameter adjustment submodule is used to input the first defect mask and the second defect mask into the first discriminant network, and adjust the model parameters of the defect mask generation model based on the difference between the image features of the first defect mask and the image features of the second defect mask until the defect mask generation model converges.

[0307] As an implementation manner of an embodiment of the present invention, the first defect mask generating submodule may include:

[0308] The defect area recognition unit is used to recognize the defect area in the real defect image.

[0309] The pixel point judgment unit is used to judge, for each pixel point in the real defect image, whether the pixel point belongs to the defect area.

[0310] The first pixel value setting unit is configured to set the pixel value of the pixel point having the same pixel position as the pixel point in the first defect mask to the first pixel value if yes.

[0311] The second pixel value setting unit is configured to set the pixel value of the pixel point having the same pixel position as the pixel point in the first defect mask to the second pixel value if no.

[0312] As an implementation manner of an embodiment of the present invention, the above-mentioned pixel point determination unit may include:

[0313] The connecting line segment generating subunit is used to generate, for each pixel point in the real defect image, a connecting line segment between the pixel point and the image edge of the real defect image.

[0314] The intersection number determination subunit is used to determine the number of intersections between the connecting line segment and the boundary of the defect area.

[0315] The first judgment subunit is used to determine that the pixel point belongs to the defect area if the pixel point is located at the boundary of the defect area or the number is an odd number.

[0316] The second judgment subunit is used to determine that the pixel does not belong to the defect area if the pixel is not located at the boundary of the defect area and the number of times is an even number.

[0317] As an implementation of an embodiment of the present invention, the close-up image generation model may include a second generation network and a second discrimination network; the defect close-up image generation module 1802 may include:

[0318] The first defect close-up image generating submodule is used to input the first defect mask into the second generating network, and generate a first defect close-up image based on the image features of the first defect mask.

[0319] The second parameter adjustment submodule is used to input the real defect image and the first defect close-up image into the second discriminant network, and adjust the model parameters of the close-up image generation model based on the difference between the image features of the real defect image and the image features of the first defect close-up image until the close-up image generation model converges.

[0320] As an implementation of the embodiment of the present invention, the image fusion module 1803 may include:

[0321] The target fusion region determination submodule is used to determine the target fusion region corresponding to the target to be simulated in the real image.

[0322] The submodule for determining the position to be fused is used to take the position in the target fusion area corresponding to the defect in the target defect mask as the position to be fused.

[0323] The image fusion submodule is used to fuse the defect in the target defect close-up image into the real image based on the position to be fused.

[0324] As an implementation manner of the embodiment of the present invention, the above-mentioned image fusion submodule may include:

[0325] The defect judgment unit is used to judge whether there is a defect at the position to be fused.

[0326] The pixel value replacement unit is used to replace the pixel value of the pixel point at the position to be fused with the pixel value of the pixel point at the corresponding position in the target defect close-up image if there is no defect.

[0327] The first returning unit is used to return to the step of determining the target fusion area corresponding to the target to be simulated in the real image if there is a defect.

[0328] As an implementation manner of an embodiment of the present invention, the above-mentioned pixel value replacement unit may include:

[0329] The histogram matching subunit is used to perform histogram matching on the target defect close-up image based on the pixel values ​​of the pixel points at the position to be fused, so as to obtain a matched target defect close-up image.

[0330] The pixel value replacement subunit is used to replace the pixel value of the pixel point at the position to be fused with the pixel value of the pixel point at the corresponding position of the matched target defect close-up image.

[0331] As an implementation manner of an embodiment of the present invention, the defect judgment unit may include:

[0332] The mask generation subunit is used to encode the real image to obtain a mask corresponding to the real image.

[0333] The pixel value judgment subunit is used to judge whether the pixel value of each pixel point corresponding to the position to be fused in the mask is the same as the pixel value of the pixel point included in the defect area in the target defect mask.

[0334] The first defect judgment subunit is used to determine that there is no defect at the to-be-fused position if they are all different.

[0335] The second defect judgment subunit is used to determine that there is a defect at the position to be fused if there are pixel points with the same pixel value.

[0336] As an implementation of the embodiment of the present invention, the above device may further include:

[0337] A second return unit is used to return to the step of determining the target fusion area corresponding to the target to be simulated in the real image after the step of replacing the pixel value of the pixel point at the position to be fused with the pixel value of the pixel point at the corresponding position in the target defect close-up image, until the number of return operations reaches a preset number.

[0338] As an implementation of the embodiment of the present invention, the above device may further include:

[0339] The connected region determining unit is used to determine the connected region corresponding to each category of defects based on the category of each defect in the target defect image.

[0340] The defect label generating unit is used to determine the defect label corresponding to the target defect image based on the boundary and category of the connected area corresponding to the defect of each category.

[0341] As an implementation manner of an embodiment of the present invention, the defect label generating unit may include:

[0342] The bounding box determination subunit is used to calculate the bounding box of the connected area corresponding to each category of defects according to the following formula:

[0343]

[0344] Among them, E x The connected area corresponding to the defect of this category includes the coordinates of the pixels in the x-axis direction of the target defect image, E y The connected area corresponding to the defect of this category includes the coordinates of the pixels in the y-axis direction of the target defect image, (x min ,y min) is the coordinate of the upper left corner vertex of the bounding box, (x max ,y max ) are the coordinates of the lower right corner vertex of the bounding box.

[0345] The defect label generating subunit is used to determine the bounding box and category corresponding to the defect of the category as the defect label corresponding to the defect of the category in the target defect image.

[0346] The embodiment of the present invention further provides an electronic device, such as Fig. 20 As shown, it includes a processor 2001, a communication interface 2002, a memory 2003 and a communication bus 2004, wherein the processor 2001, the communication interface 2002, and the memory 2003 communicate with each other through the communication bus 2004.

[0347] Memory 2003, used for storing computer programs;

[0348] The processor 2001 is used to implement the method steps described in any of the above embodiments when executing the program stored in the memory 2003.

[0349] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0350] The communication interface is used for communication between the above electronic device and other devices.

[0351] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0352] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0353] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0354] In another embodiment of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute the method steps described in any of the above embodiments.

[0355] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0356] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0357] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0358] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for generating a defect image, characterized in that: The method comprises: Inputting a target noise image into a pre-trained defect mask generation model to generate a target defect mask corresponding to the target noise image, wherein the target noise image is a noise image corresponding to a defect of a preset category, and the defect mask generation model is obtained based on adversarial training of a first defect mask corresponding to a real defect image and a second defect mask corresponding to a random noise image; Inputting the target defect mask into a pre-trained close-up image generation model to generate a target defect close-up image corresponding to the target defect mask, wherein the close-up image generation model is obtained based on adversarial training between the real defect image and a first defect close-up image corresponding to the first defect mask; A real image including a target to be simulated is acquired, and the target defect close-up image is fused into the real image to obtain a target defect image, wherein the target to be simulated is a target with defects to be simulated.

2. The method according to claim 1, characterized in that: The target defect mask has corresponding defect category information; After the step of fusing the target defect close-up image with the real image to obtain the target defect image, the method further includes: Based on the fusion position of the target defect close-up image in the real image and the defect category information, a defect label corresponding to the target defect image is determined.

3. The method according to claim 1, characterized in that: The defect mask generation model includes a first generation network and a first discrimination network; The training method of the defect mask generation model includes: Acquire a real defect image, and encode the real defect image based on defects included in the real defect image to obtain a first defect mask; Inputting the random noise image into the first generation network, and generating a second defect mask based on the noise characteristics of the random noise image; The first defect mask and the second defect mask are input into the first discrimination network, and based on the difference between the image features of the first defect mask and the image features of the second defect mask, the model parameters of the defect mask generation model are adjusted until the defect mask generation model converges.

4. The method according to claim 3, characterized in that: The step of encoding the real defect image based on the defects included in the real defect image comprises: Identifying a defective area in the real defect image; For each pixel in the real defect image, determining whether the pixel belongs to the defect area; If yes, setting the pixel value of the pixel point in the first defect mask having the same pixel position as the pixel point as the first pixel value; If not, the pixel value of the pixel point in the first defect mask having the same pixel position as the pixel point is set to the second pixel value.

5. The method according to claim 4, characterized in that: The step of determining, for each pixel in the real defect image, whether the pixel belongs to the defect area comprises: For each pixel point in the real defect image, generating a connecting line segment between the pixel point and an image edge of the real defect image; Determining the number of intersections between the connecting line segment and the boundary of the defect area; If the pixel point is located at the boundary of the defect area or the number is an odd number, it is determined that the pixel point belongs to the defect area; If the pixel point is not located at the boundary of the defective area and the number is an even number, it is determined that the pixel point does not belong to the defective area.

6. The method according to claim 1, characterized in that: The close-up image generation model includes a second generation network and a second discriminant network; The training method of the close-up image generation model includes: Inputting the first defect mask into the second generation network, and generating a first defect close-up image based on image features of the first defect mask; The real defect image and the first defect close-up image are input into the second discriminant network, and based on the difference between the image features of the real defect image and the image features of the first defect close-up image, the model parameters of the close-up image generation model are adjusted until the close-up image generation model converges.

7. The method according to any one of claims 1 to 6, characterized in that: The step of fusing the target defect close-up image into the real image comprises: Determine a target fusion area corresponding to the target to be simulated in the real image; Taking the position in the target fusion area corresponding to the defect in the target defect mask as the position to be fused; Based on the position to be fused, the defect in the target defect close-up image is fused into the real image.

8. The method according to claim 7, characterized in that: The step of fusing the defect in the target defect close-up image into the real image based on the position to be fused includes: Determining whether there is a defect at the position to be fused; If there is no defect, the pixel value of the pixel point at the position to be fused is replaced with the pixel value of the pixel point at the corresponding position in the close-up image of the target defect; If there is a defect, return to the step of determining the target fusion area corresponding to the target to be simulated in the real image.

9. The method according to claim 8, characterized in that: The step of replacing the pixel value of the pixel point at the position to be fused with the pixel value of the pixel point at the corresponding position in the target defect close-up image comprises: Based on the pixel values ​​of the pixel points at the positions to be fused, histogram matching is performed on the target defect close-up image to obtain a matched target defect close-up image; The pixel value of the pixel point at the position to be fused is replaced by the pixel value of the pixel point at the corresponding position of the matched target defect close-up image.

10. The method according to claim 8, characterized in that: The step of judging whether there is a defect at the position to be fused comprises: Encoding the real image to obtain a mask corresponding to the real image; Determine whether a pixel value of each pixel point in the mask corresponding to the position to be fused is the same as a pixel value of a pixel point included in a defect area in the target defect mask; If they are not the same, it is determined that there is no defect at the position to be fused; If there are pixels with the same pixel value, it is determined that there is a defect at the position to be fused.

11. The method according to claim 8, characterized in that: After the step of replacing the pixel values ​​of the pixels at the positions to be fused with the pixel values ​​of the pixels at the corresponding positions in the close-up image of the target defect, the method further includes: Return to the step of determining the target fusion area corresponding to the target to be simulated in the real image until the number of return operations reaches a preset number.

12. The method according to claim 11, characterized in that: The method further comprises: Based on the category of each defect in the target defect image, determining a connected region corresponding to each category of defects; For each category of defects, the defect label corresponding to the target defect image is determined based on the boundary and category of the connected region corresponding to the defect of the category.

13. The method according to claim 12, characterized in that: The step of determining the defect label corresponding to the target defect image based on the boundary and category of the connected region corresponding to the defect of each category includes: For each category of defects, the bounding box of the connected area corresponding to the defect of that category is calculated according to the following formula: Among them, E x The connected area corresponding to the defect of this category includes the coordinates of the pixels in the x-axis direction of the target defect image, E y The connected area corresponding to the defect of this category includes the coordinates of the pixels in the y-axis direction of the target defect image, (x min ,y min ) is the coordinate of the upper left corner vertex of the bounding box, (x max ,y max ) are the coordinates of the lower right corner vertex of the bounding box; The bounding box and the category corresponding to the defect of the category are determined as the defect label corresponding to the defect of the category in the target defect image.

14. A defect image generating device, characterized in that: The device comprises: A defect mask generation module, used for inputting a target noise image into a pre-trained defect mask generation model to generate a target defect mask corresponding to the target noise image, wherein the target noise image is a noise image corresponding to a defect of a preset category, and the defect mask generation model is obtained based on adversarial training of a first defect mask corresponding to a real defect image and a second defect mask corresponding to a random noise image; a defect close-up image generation module, configured to input the target defect mask into a pre-trained close-up image generation model to generate a target defect close-up image corresponding to the target defect mask, wherein the close-up image generation model is obtained based on adversarial training between the real defect image and a first defect close-up image corresponding to the first defect mask; The image fusion module is used to obtain a real image including a target to be simulated, and fuse the target defect close-up image into the real image to obtain a target defect image, wherein the target to be simulated is a target with defects to be simulated.

15. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1-13 when executing a program stored in a memory.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 13 are implemented.