Methods for generating defect images and electronic devices and storage media

Abstract

This invention discloses a method for generating defect images, as well as an electronic device and storage medium. In the method, a first defect image is input into a preset target network model, and the target network model outputs a second defect image. The second defect image includes the first defect image and an expanded image, which is obtained by expanding the background of the first defect image. Compared to existing technologies, this application generates a new second defect image with higher resolution by expanding the background of the defect image while retaining the defect portion of the first defect image. Furthermore, because the second defect image retains the defect portion of the first defect image, it inherently carries the defect label of the first defect image, eliminating the need for additional labeling and reducing training costs.

Description

Technical Field

[0001] This application relates to image processing technology, including but not limited to a method for generating defective images, an electronic device, and a storage medium. Background Technology

[0002] In production and daily life, it is inevitable to inspect defective products to facilitate subsequent isolation and targeted treatment. For example, in industrial production, a certain number of defective products are usually generated at various stages of production. Defect detection is performed after the products are manufactured to reduce or even eliminate defects. Similarly, in the field of medical imaging, tumor images often need to be detected and segmented. While detection models are typically used for defective product inspection, the number of defective images for these products is relatively small. Training a detection model with a limited number of defective images often results in insufficient detection diversity. Summary of the Invention

[0003] In view of this, the defect image generation method, electronic device, and storage medium provided in the embodiments of this application can generate new defect images with higher resolution based on the original defect images, thereby expanding the defect images. When using the expanded defect image set to train the neural network, the recognition diversity of the neural network can be improved.

[0004] In a first aspect, embodiments of this application provide a method for generating a defect image, the method comprising: obtaining a first defect image; inputting the first defect image into a preset target network model, and outputting a second defect image through the target network model, the second defect image comprising the first defect image and an expanded image, the expanded image being obtained by expanding the background of the first defect image; wherein the preset target network model is obtained by training an initial network model using a training image set through at least two different training processes, the training image set comprising a sample image set and a sample mask set, and the sample mask sets used in the at least two different training processes being different.

[0005] In the aforementioned method for generating defect images, a first defect image is input into a preset target network model, and the target network model outputs a second defect image. The second defect image includes the first defect image and an expanded image, which is obtained by expanding the background of the first defect image. Compared to existing technologies, this application generates a new second defect image with higher resolution by expanding the background of the defect image while retaining the defect portion of the first defect image. Furthermore, because the second defect image retains the defect portion of the first defect image, it inherently carries the defect label of the first defect image, eliminating the need for additional labeling and reducing training costs.

[0006] In some embodiments, outputting the second defect image through the target network model includes: determining a target mask from a preset mask set through the target network model, and then outputting the second defect image based on the first defect image and the target mask.

[0007] In some embodiments, the target network model includes an encoding module and a decoding module. The encoding module is used to obtain a target encoding sequence based on the target mask, the first defect image, and preset noise information. The decoding module is used to obtain the second defect image based on the target encoding sequence and the decoding mechanism corresponding to the target network model.

[0008] In some embodiments, the sample image set includes a first sample image set and a second sample image set, wherein the sample images included in the first sample image set and the second sample image set are different, and the sample mask set includes a first sample mask set and a second sample mask set. Before inputting the first defect image into the preset target network model, the method further includes: training the initial network model according to the first sample image set and the first sample mask set to obtain a first model; and training the first model according to the second sample image set and the second sample mask set to obtain the target network model.

[0009] In some embodiments, the initial network model includes an encoding module and a decoding module connected to the encoding module. The step of training the first model based on the sample image set and the first sample mask set includes: determining a first sample mask image from the first sample mask set, the first sample mask image having a first region to be repaired, the first region to be repaired being located within the first sample mask image; using the first sample image set and the first sample mask image as conditional information for the first model, and inputting the conditional information and preset noise information into the encoding module of the initial network model for training; and when the loss value of the decoding result output by the decoding module converges, using the current initial network model as the first model.

[0010] In some embodiments, training the first model based on the sample image set and the second sample mask set to obtain the target network model includes: determining a second sample mask image from the second sample mask set, the second sample mask image having a second region to be repaired, and at least one boundary line of the second region to be repaired coinciding with a corresponding boundary line of the second sample mask image; using the second sample image set and the second sample mask image as conditional information of the first model, and inputting the conditional information and preset noise information into the encoding module of the first model for training, and when the loss value of the decoding result output by the decoding module of the first model converges, using the current first model as the target network model.

[0011] In some embodiments, before using the second sample image set and the second sample mask image as conditional information of the first model, and inputting the conditional information and preset noise information into the encoder of the first model for training, the method further includes: obtaining a preset sample defect image set; cropping each sample defect image in the sample defect image set to obtain the second sample image set.

[0012] In some embodiments, the second defect image is obtained by expanding the background of the first defect image multiple times through the target network model, and the size of the second defect image is greater than or equal to a preset size.

[0013] Secondly, an electronic device is provided in the embodiments of this application. The electronic device includes: a defect image acquisition module for acquiring a first defect image; and a defect image augmentation module for inputting the first defect image into a preset target network model and outputting a second defect image through the target network model. The second defect image includes the first defect image and an augmented image, wherein the augmented image is obtained by augmenting the background of the first defect image. The preset target network model is obtained by training an initial network model using a training image set through at least two different training processes. The training image set includes a sample image set and a sample mask set, and the sample mask sets used in the at least two different training processes are different.

[0014] Thirdly, the computer device provided in the embodiments of this application includes a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement the method described in the embodiments of this application.

[0015] Fourthly, the computer-readable storage medium provided in the embodiments of this application stores a computer program thereon, which, when executed by a processor, implements the method described in the embodiments of this application.

[0016] It should be understood that the second to fourth aspects of the embodiments of this application are consistent with the technical solutions of the first aspect of the embodiments of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be described again. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0018] Figure 1 Example diagram of the structure of a defect generation model in the prior art;

[0019] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0020] Figure 3 A schematic flowchart illustrating the method for generating defect images provided in an embodiment of this application;

[0021] Figure 4 A schematic diagram of the background augmentation result of the first defect image provided in the embodiments of this application;

[0022] Figure 5 A schematic diagram of the background augmentation process of the first defect image provided in the embodiments of this application;

[0023] Figure 6This is a schematic diagram of the structure of the target network model provided in the embodiments of this application;

[0024] Figure 7 A schematic flowchart illustrating the method for generating defect images provided in an embodiment of this application;

[0025] Figure 8 A schematic diagram of the training process of the initial network model provided in the embodiments of this application;

[0026] Figure 9 A schematic diagram of the second sample mask image provided in the embodiments of this application;

[0027] Figure 10A This application provides an embodiment of a second region to be repaired.

[0028] Figure 10B This is a schematic image of another second area to be repaired provided in an embodiment of this application;

[0029] Figure 11 A schematic diagram of the training process of the first model provided in an embodiment of this application;

[0030] Figure 12 A schematic diagram of the second mask image provided in an embodiment of this application;

[0031] Figure 13 A schematic flowchart illustrating the method for generating defect images provided in an embodiment of this application;

[0032] Figure 14 A schematic diagram of the second defect image generated for the target model provided in the embodiments of this application;

[0033] Figure 15 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0034] Figure 16 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0037] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0038] It should be noted that the terms "first, second, third" used in the embodiments of this application are used to distinguish similar or different objects and do not represent a specific order of objects. It can be understood that "first, second, third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0039] Existing technologies typically employ defect generation models to augment the background of defective images to address the problem of insufficient defect samples. Currently, mainstream defect generation models primarily rely on cyclically comparing defective and non-defective images between two generation models, thereby constructing a cycle consistency loss between the two data domains. One generation model generates defects, and the other removes them. After training, a defect generation model composed of the two models is used to generate the corresponding defective image. For example, as... Figure 1 As shown, the defect generation model includes a generator F, a generator G, a discriminator D1, and a discriminator D2. First, the defective image b is input into the generator F to generate a new image F(b). Then, discriminator D2 determines whether the generated sample F(b) is a real sample without defects, and discriminator D1 determines whether the generated sample F(b) is a real sample with defects. Similarly, the defect-free image g is input into the generator G to generate a new image G(g). Then, discriminator D1 determines whether the generated sample G(g) is a real sample with defects, and discriminator D2 determines whether the generated sample G(g) is a real sample without defects, and this process is repeated. Finally, because the data in the two domains are not paired (i.e., sample g and sample b), to ensure the consistency of the model's input and output structure, the cycle consistency loss principle is used to calculate the model's total loss value during training. The training of the model ends when the total loss value converges.

[0040] While the above-mentioned schemes can expand the defect image dataset, these methods have the following drawbacks: First, since the generated defects are regenerated, the defect parts are not labeled, requiring manual labeling after the model generates the defects; second, the generated defect images have small image areas for the defect parts, making it easy for the model to fail to perceive the area, resulting in weak defect texture information; third, since only limited defect and background data are used, the generated defects may lack diversity, leading to pattern collapse; finally, since existing background data is used, the final synthesized defects may lack diversity, causing the detection model to get stuck in a local optimum.

[0041] In view of this, embodiments of this application provide a method for generating defective images. This method is applied to an electronic device, which can be various types of devices with information processing capabilities. For example, the electronic device may include a personal computer, laptop, PDA, or server; the electronic device may also be a mobile terminal, such as a mobile phone, in-vehicle computer, tablet computer, or projector. The functions implemented by this method can be achieved by a processor in the electronic device calling program code. Of course, the program code can be stored in a computer storage medium. Therefore, the electronic device includes at least a processor and a storage medium.

[0042] For example, such as Figure 2 As shown, the electronic device 10 may include a processor 101, an external memory interface 102, an internal memory 103, a universal serial bus (USB) interface 104, a camera module 105, a display screen 106, and buttons 107.

[0043] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 10. In other embodiments of this application, the electronic device 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0044] Processor 101 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. For example, processor 101 may be a smart terminal CPU, such as a Snapdragon series processor. In some embodiments, processor 101 may include one or more interfaces. Interfaces may include inter-integrated circuit (I2C) interfaces, inter-integrated circuit sound (I2S) interfaces, pulse code modulation (PCM) interfaces, universal asynchronous receiver / transmitter (UART) interfaces, mobile industry processor interfaces (MIPI) interfaces, general-purpose input / output (GPIO) interfaces, subscriber identity module (SIM) interfaces, and / or universal serial bus (USB) interfaces, etc.

[0045] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 10. In other embodiments of this application, the electronic device 10 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0046] Electronic device 10 implements display functions through a GPU, display screen 106, and application processor. The GPU is a microprocessor for image processing, connecting the display screen 106 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 101 may include one or more GPUs, which execute program instructions to generate or modify display information. Display screen 106 is used to display images, videos, etc.

[0047] Electronic device 10 can perform shooting functions through an ISP, a shooting module 105, a video codec, a GPU, a display screen 106, and an application processor. The ISP processes data fed back by the shooting module 105. The shooting module 105 captures still images or videos. An object is projected onto a photosensitive element by an optical image generated through a lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, and other formats. The digital signal processor processes digital signals, including digital image signals and other digital signals. In some embodiments, the shooting module 105 can be a camera.

[0048] The external storage interface 102 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 10. The external memory card communicates with the processor 101 through the external storage interface 102 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0049] The internal memory 103 can be used to store computer executable program code, which includes instructions. The internal memory 103 may include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the electronic device 10 (such as audio data, phonebook, etc.).

[0050] Buttons 107 include a power button, volume buttons, etc. Buttons 107 can be mechanical buttons or touch-sensitive buttons. The electronic device 10 can receive button input and generate key signal inputs related to user settings and function control of the electronic device 10.

[0051] In addition, the electronic devices involved in the embodiments of this application may also be equipped with an operating system, on which applications can be installed and run. The embodiments of this application do not limit this.

[0052] Figure 3 This is a schematic flowchart illustrating the method for generating defect images provided in an embodiment of this application. Figure 3 As shown, the method may include the following steps:

[0053] Step S101: Obtain the first defect image;

[0054] Step S102: Input the first defect image into the preset target network model, and output the second defect image through the target network model. The second defect image includes the first defect image and the expanded image. The expanded image is obtained by expanding the background of the first defect image.

[0055] The preset target network model is obtained by training the initial network model using a training image set through at least two different training processes. The training image set includes a sample image set and a sample mask set, and the sample mask sets used in the at least two different training processes are different.

[0056] It should be understood that because the number of defect images is relatively small, directly using the sample image set corresponding to the defect images for training may cause the model to get stuck in a local optimum, where only the defect types of the training sample defect images exist, resulting in a lack of diversity in the generated defect images. Therefore, in this application, different sample mask sets are used to perform at least two different training processes on the initial network model, which can enable the target network model to gradually expand the defect images, thereby reducing the overfitting problem caused by the small number of sample images during model training.

[0057] Different sample mask sets have different regions to be repaired. During at least two different training processes of the target network model, the initial network model can be trained first using the first sample mask set with the first region to be repaired to obtain the first network model, so that the first network model has the ability to repair the corresponding position of the first region to be repaired. Then, the first network model can be trained first using the second sample mask set with the second region to be repaired to obtain the second network model, so that the second network model has the ability to repair the corresponding position of the second region to be repaired. This process is repeated until the target network model with background augmentation capability is finally obtained.

[0058] It should be understood that the aforementioned first defect image is an image including the defective area. This defective area is an image feature different from other parts of the image. For example, when a normally smooth target object surface has defects such as discoloration spots, scratches, defects, dust spots, pitting, dirt, or bubbles, an image including these defects will be formed during image acquisition. Because the presence of these defects obstructs a portion of the target object, the area with the defect becomes the defective area of ​​the target object, and the image including the defective area is the defect image. In some embodiments, the aforementioned defect image can be an industrial defect image, a medical tumor image, etc.

[0059] Optionally, the aforementioned expanded image can be an image obtained by expanding the background of the first defect image in any one of the four directions. Any one of the four directions can be the top, bottom, left, or right side of the first defect image. The expansion direction can be randomly determined by the target network model or it can be a fixed direction. The specific design can be made as needed, and this application does not impose any restrictions.

[0060] For example, such as Figure 4 As shown, when the expansion direction is randomly determined by the target network model, after obtaining the first defect image, the target network model can generate a second defect image A that expands to the right or a second defect image B that expands to the left, and the same applies to other directions; when the expansion direction is fixed to the right, the target network model will only generate a second defect image A that expands to the right after obtaining the first defect image.

[0061] Optionally, the above method for determining the expansion direction of the expanded image can be to first determine the target mask in the corresponding direction, and then the target network model determines the second defect image based on the target mask and the first defect image.

[0062] Alternatively, the target mask can be determined by the target network model or predetermined by the electronic device. It can be designed as needed, and this application does not impose any restrictions.

[0063] Optionally, the sample mask set used in at least two different training processes may include regions to be repaired at different locations. By changing the position of the region to be repaired in the sample mask set, the target network model can have a complete region repair function. The position of the region to be repaired may be within the boundary area of ​​the sample mask image, outside the boundary area, or on the boundary line, etc. The specific design can be made as needed, and this application does not impose any restrictions.

[0064] Optionally, the sample image sets used in the above-mentioned at least two different training processes can be the same or different, and can be designed as needed. This application does not impose any restrictions.

[0065] Optionally, the second defect image is obtained by expanding the background of the first defect image multiple times through the target network model, and the size of the second defect image is greater than or equal to a preset size.

[0066] Optionally, the aforementioned preset size can be set by those skilled in the art according to actual conditions, and this application does not impose any restrictions. In some embodiments, the preset size can be stored as a preset size by receiving size information input by the user, for example, by providing the user with a size input control so that the user can determine the preset size through the size input control.

[0067] Optionally, the aforementioned preset size can be pre-input into the target network model so that the target network model determines the number of times the first defect image is expanded based on the preset size, for example, as... Figure 5 As shown, if the preset size of the second defect image is A, and the size of the defect image output after one expansion is B, and A = 3B, then in order to obtain the second defect image of size A, the first defect image needs to be expanded 3 times. The size of the defect image output after 3 expansions is A, so that the size of the output second defect image is greater than or equal to the preset size.

[0068] It is understandable that by performing background augmentation on the first defect image multiple times at once, the number of steps required to repeatedly input the output second defect image into the target network model can be reduced, so that the target network model can finally output a defect image of a size greater than or equal to the preset size, thereby improving the efficiency of defect image generation.

[0069] In the aforementioned method for generating defect images, a first defect image is input into a preset target network model, and the target network model outputs a second defect image. The second defect image includes the first defect image and an expanded image, which is obtained by expanding the background of the first defect image. Compared to existing technologies, this application generates a new second defect image with higher resolution by expanding the background of the defect image while retaining the defect portion of the first defect image. Furthermore, because the second defect image retains the defect portion of the first defect image, it inherently carries the defect label of the first defect image, eliminating the need for additional labeling and reducing training costs.

[0070] Figure 6 This is a schematic diagram of the structure of a target network model provided in an embodiment of this application. Figure 6As shown, the target network model may include an encoding module 1011 and a decoding module 1012. The encoding module 1011 is used to obtain a target encoding sequence based on the target mask, the first defect image and preset noise information. The decoding module 1012 is used to obtain a second defect image based on the target encoding sequence and the decoding mechanism corresponding to the target network model.

[0071] It should be understood that the decoding mechanism corresponding to the above target network model is trained based on the sample image set during the training process. During the training process, the encoding module 1011 converts the target mask, the first defect image, and the noise information into an encoding sequence, and then transmits the encoding sequence to the decoding module 1012. The decoding module 1012 obtains the conditional distribution of the first target vector corresponding to the second defect image based on the decoding mechanism and the encoding sequence, and then samples the first target vector from the conditional distribution. The first target vector is then input into the decoding module 1012 again. The decoding module 1012 starts to define the conditional distribution of the second target vector corresponding to the second defect image based on the first target vector. The above process is repeated until the last target vector is sampled, completing the sampling process of the target encoding sequence, and outputting the second defect image corresponding to all target vectors.

[0072] Optionally, the sampling process for each target vector described above can be represented using a recursive formula:

[0073]

[0074]

[0075] Among them, the initial term z of the recursive formula T It conforms to the standard normal distribution, i.e., z T ~N(0,I), and finally z0 can be obtained recursively, where z0 is the final output second defect image.

[0076] Optionally, the target mask can be pre-set in the electronic device. When the first defect image is input to the target network model, the electronic device will also input the target mask into the target network model. Alternatively, the target mask can be generated by the target network model when it receives the first defect image. The specific design can be customized as needed, and this application does not impose any restrictions.

[0077] Accordingly, when the target mask is generated by the target network model when it receives the first defect image, the target network model may also include a mask determination module. The mask determination module is used to determine the target mask from a preset mask set, so that the target network model can realize the target mask determination process.

[0078] Optionally, the target mask mentioned above may include a region to be repaired and a specified region. The specified region is used to indicate the image basis for image repair, and the region to be repaired is used to indicate the corresponding image region to be repaired based on the specified region. For example, ... Figure 7 As shown, taking the target mask as an example, which includes a black designated area on the left and a white area to be repaired on the right, the black designated area can cover part or all of the image area of ​​the first defective image, and instruct the target network model to repair the white area to be repaired on the right based on the black designated area, so as to complete the background expansion of the first defective image.

[0079] In the aforementioned target network model, by setting an encoding module that encodes the target mask, the first defect image, and preset noise information, and a decoding module that executes a corresponding decoding mechanism, the target network model can achieve background augmentation of the first defect image.

[0080] Figure 7 A flowchart illustrating a method for generating a defect image provided in an embodiment of this application is shown below. Figure 7 As shown, the method may include the following steps:

[0081] Step S201: Train the initial network model based on the first sample image set and the first sample mask set to obtain the first model;

[0082] Step S202: Train the first model based on the second sample image set and the second sample mask set to obtain the target network model;

[0083] Step S203: Obtain the first defect image;

[0084] Step S204: Input the first defect image into the target network model, and output the second defect image through the target network model. The second defect image includes the first defect image and an expanded image. The expanded image is obtained by expanding the background of the first defect image.

[0085] It should be noted that the execution methods of steps S303 to S304 can be the same as those of steps S101 to S102, and will not be repeated here.

[0086] It should be understood that the first sample image set and the second sample image set are different sample image sets. The first sample image set and the second sample image set include different sample images, so as to use different sample image sets and sample mask sets to achieve different stages of training of the target network model, so that the target network model has the function of background expansion.

[0087] Optionally, step S301 above may include steps 1 to 2:

[0088] Step 1: Determine a first sample mask image from the first sample mask set. The first sample mask image has a first region to be repaired, and the first region to be repaired is located within the first sample mask image.

[0089] Step 2: Use the first sample image set and the first sample mask image as the conditional information of the first model, and input the conditional information and the preset noise information into the encoding module of the initial network model for training. When the loss value of the decoding result output by the decoding module converges, the current initial network model is used as the first model.

[0090] Optionally, the first sample mask set can be any mask image with the same size as the first sample image set. The specific sizes of the first sample mask set and the first sample image set can be designed as needed, and this application does not impose any restrictions.

[0091] The specific location of the first region to be repaired within the first sample mask image is determined by those skilled in the art based on actual needs, and this application does not impose any restrictions. In some embodiments, the first sample image set also has a region to be repaired at the same location as the first region to be repaired.

[0092] For example, such as Figure 8 As shown, the initial network model includes an encoder E and a decoder D. A first sample image x0 is randomly selected from the first sample image set, and a first mask image M is randomly selected from the first sample mask set. The region to be repaired in the first mask image M is pre-created by those skilled in the art. For example, the region to be repaired in the first mask M is... Figure 8 The white area in the image is used to connect the first sample image x0 and the first mask image M as conditional information for the initial network model along the channel dimension, and is then combined with a preset noise function z. T Input encoder E together and establish the model objective function corresponding to the initial network model. The formula for the model objective function is:

[0093] L=||D(E(z t ,M,x0))-∈||,∈~N(0,I)

[0094] Wherein, the noise function z T The formula is:

[0095]

[0096] Where, α t =1-β t , t~U(1,T), where T is assumed to be 1000 by default, and N(0,I) represents the standard normal distribution.

[0097] Additionally, the initial network model can be configured with an optimizer to find the minimum loss value when the model's objective function converges as quickly as possible. Training stops when the loss value of the objective function converges, resulting in the first model. In this embodiment, the optimizer can be the Adam optimizer.

[0098] Preferably, the first sample image set can be a large-scale conventional image set, i.e., an image set that does not include defective images, in order to alleviate the problem of model overfitting.

[0099] Understandably, after the training process of the first model is completed, it is only necessary to input an original image and a binary mask image including the area to be repaired into the first model to obtain a completely new image that is different from the original image, so that the first model has the function of background repair.

[0100] Optionally, step S302 above may include steps a to b:

[0101] Step a: Determine a second sample mask image from the second sample mask set. The second sample mask image has a second region to be repaired, and the boundary line of the second region to be repaired coincides with the boundary line of the second sample mask image.

[0102] Step b: Use the second sample image set and the second sample mask image as the conditional information of the first model, and input the conditional information and the preset noise information into the encoding module of the first model for training. When the loss value of the decoding result output by the decoding module of the first model converges, the current first model is used as the target network model.

[0103] It should be understood that the overlap between the boundary line of the second region to be repaired and the boundary line of the second sample mask image means that the second region to be repaired is located in the boundary region of the second sample mask image, and the boundary line of the second region to be repaired coincides with the boundary line of the second sample mask image. For example, as... Figure 9 As shown, the second sample mask image A is a rectangle, and the second sample mask image A includes a left boundary line with the minimum x-axis coordinate of the second sample mask image A and a right boundary line with the maximum x-axis coordinate of the second sample mask image A.

[0104] The phrase "the boundary line of the second region to be repaired coincides with the boundary line of the second sample mask image" means that, when the second region to be repaired covers part or all of the image area of ​​the second sample mask image, at least one boundary line of the second region to be repaired coincides with the boundary line of the corresponding position in the second sample mask image. For example, as... Figure 10AAs shown, the upper boundary line of the second region to be repaired AA coincides with the upper boundary line of the second sample mask image B. Simultaneously, the lower boundary line of the second region to be repaired A coincides with the lower boundary line of the second sample mask image B, and the left boundary line of the second region to be repaired A coincides with the left boundary line of the second sample mask image B. Alternatively, the upper boundary line of the second region to be repaired A coincides with the upper boundary line of the second sample mask image B, and the lower boundary line of the second region to be repaired A coincides with the lower boundary line of the second sample mask image B, and the right boundary line of the second region to be repaired A coincides with the right boundary line of the second sample mask image B. When only one boundary line of the second region to be repaired A coincides with the corresponding boundary line of the second sample mask image B, such as... Figure 10B As shown, the second region to be repaired, A, has only one left boundary line that coincides with the left boundary line of the second sample mask image B, or the second region to be repaired, A, has only one right boundary line that coincides with the right boundary line of the second sample mask image B. The specific design can be customized as needed, and this application does not impose any restrictions.

[0105] For example, such as Figure 11 As shown, a second sample image x′0 is randomly selected from the second sample image set, and a second sample mask set M is used. S A second mask image M is randomly selected from the sample images. The second sample mask set includes only the mask images of the four halves of the region to be repaired: the upper half, the lower half, the left half, and the right half. S For example Figure 12 As shown, the region to be repaired in the second mask image M is pre-created by those skilled in the art. The second sample image x′0 and the second mask image M are connected in the channel dimension as conditional information of the first model, and then combined with a preset noise function z. R Input the encoder E of the first model together, and establish the model objective function corresponding to the first model. The formula for the model objective function is:

[0106] L=||D(E(z t ,M,x′0))-∈||,∈~N(0,I)

[0107] Wherein, the noise function z T The formula is:

[0108]

[0109] Where, α t =1-β t , t~U(1,T), where T is assumed to be 1000 by default, and N(0,I) represents the standard normal distribution.

[0110] Additionally, the first model can be equipped with an optimizer to find the minimum loss value when the model's objective function converges as quickly as possible. Training stops when the loss value of the objective function converges, resulting in the target network model. In this embodiment, the optimizer can be the Adam optimizer.

[0111] The second sample image set mentioned above is an image set that includes defective images, in order to complete the adaptation training of the first model to the defective image set.

[0112] Understandably, after training the target network model using a small set of defect images, it is only necessary to input an original defect image and a binary mask image including the area to be repaired into the target network model to obtain a brand new defect image with background augmentation based on the original defect image, so that the target network model has the function of background augmentation.

[0113] Optionally, the method for obtaining the second sample image set may include:

[0114] Obtain a pre-defined set of sample defect images;

[0115] Each sample defect image in the sample defect image set is cropped to obtain the second sample image set.

[0116] It should be understood that cropping each sample defect image can expand the data volume of the second sample image set, thereby further increasing the number of sample defect images compared to the original sample defect image set.

[0117] In the above-mentioned method for generating defective images, the initial network model is trained sequentially using different sample image sets and different sample mask sets. This allows for the training of the target network model at different stages, enabling the target network model to gradually acquire the function of background augmentation and reducing the problem of model overfitting.

[0118] The following describes an exemplary application of the embodiments of this application in a real-world application scenario.

[0119] Figure 13 This is a schematic flowchart illustrating a method for generating a defect image according to an embodiment of this application. Figure 13 As shown, the method includes the following steps:

[0120] Step S301: Construct an initial model, which includes encoder E and decoder D;

[0121] Step S302: Randomly select a first image x0 from a large-scale dataset, randomly generate a mask image, and manually create the region to be repaired in the mask image to form a first mask M including the region to be repaired. The large-scale dataset is, for example, a set of natural images.

[0122] Step S303: The first image x0 and the first mask M are concatenated as conditional information of the initial model in the channel dimension and combined with the preset noise function z. T Input encoder E together. Additionally, use the Adam optimizer (Adaptive Moment Estimation) as the optimizer for the training model to optimize the objective function. The learner rate of the optimizer is set to 0.0002 by default. Training stops when the loss value of the initial model's objective function converges, resulting in the first trained model D(E(z)). t ,M,x0));

[0123] Step S304: Randomly crop each image from the existing defect image set to expand the defect image set, obtain a defect image set, determine the second image x′0 from the defect image set, and select the mask set M. s A second mask M is randomly selected from the set of masks M. s Includes masks in the four directions (up, down, left, and right) to expand and generate the image in these four directions;

[0124] Step S305: Use the second image x′0 and the second mask M as the first model D(E(z) t The conditional information of (x, m, x0) is concatenated along the channel dimension and input into encoder E along with a preset noise function. Additionally, the Adam optimizer (Adaptive Moment Estimation) is used as the optimizer for training the model to optimize the objective function, with the learner rate set to 0.00002 by default. Training stops when the loss value of the objective function of the first model converges, resulting in the trained second model D(E(z)). t ,M,x′0)), and use the second model as the target model;

[0125] Step S306: Input the first defect image into the target model D(E(z) t In the formula M, x′0), the target model samples according to a recursive formula, where the initial term of the recursive formula is z. T ~N(0,I), and finally z0 can be obtained recursively. z0 is the final output second defect image. The second defect image is a new defect image obtained by expanding the first defect image in any one of the directions (up, down, left, right). The recursive formula is as follows:

[0126]

[0127]

[0128] Figure 14 This is a schematic diagram illustrating the generation of a second defect image from a target model, as provided in an embodiment of this application. Figure 14 As shown, after the first defect image is input into the target model, the target model obtains the corresponding cropped image x′0 and mask image M based on the first defect image. Then, it obtains the second defect image based on the cropped image x′0 and mask image M. The expansion direction of the second defect image is determined by the mask image M. For example, the mask image M expands with white areas. When the white area of ​​the mask image M is on the right side, the second defect image is obtained by expanding the first defect image to the right. Figure 14 The second defect image A is obtained by expanding the first defect image to the left when the white area of ​​the mask image M is on the left. For example... Figure 14 The second defect image B in the image.

[0129] It should be understood that, although Figure 1-13 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1-13 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0130] Based on the foregoing embodiments, this application provides an electronic device, which includes various modules and units included in each module, which can be implemented by a processor; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP) or field programmable gate array (FPGA), etc.

[0131] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 15 As shown, the electronic device includes a defect image acquisition module 501 and a defect image augmentation module 502, wherein:

[0132] Defect image acquisition module 501 is used to acquire a first defect image;

[0133] The defect image augmentation module 502 is used to input the first defect image into a preset target network model and output a second defect image through the target network model. The second defect image includes the first defect image and an augmented image, which is obtained by augmenting the background of the first defect image.

[0134] The preset target network model is obtained by training the initial network model using a training image set through at least two different training processes. The training image set includes a sample image set and a sample mask set, and the sample mask sets used in the at least two different training processes are different.

[0135] In some embodiments, the defect image augmentation module 502 described above may include:

[0136] The mask determination submodule is used to determine the target mask from a preset mask set through the target network model, and then output the second defect image based on the first defect image and the target mask.

[0137] In some embodiments, the target network model includes an encoding module, a decoding module, and a mask determination module. The mask determination module is used to determine a target mask from a preset mask set. The encoding module is used to obtain a target encoding sequence based on the target mask, a first defect image, and preset noise information. The decoding module is used to obtain a second defect image based on the target encoding sequence and the decoding mechanism corresponding to the target network model.

[0138] In some embodiments, the sample image set includes a first sample image set and a second sample image set, wherein the sample images included in the first sample image set and the second sample image set are different, and the sample mask set includes a first sample mask set and a second sample mask set. The above-mentioned electronic device may further include:

[0139] The first model training module is used to train the initial network model based on the first sample image set and the first sample mask set to obtain the first model;

[0140] The target network training module is used to train the first model based on the second sample image set and the second sample mask set to obtain the target network model.

[0141] In some embodiments, the initial network model includes an encoding module and a decoding module connected to the encoding module, and the first model training module mentioned above may include:

[0142] The first mask determination submodule is used to determine a first sample mask image from a first sample mask set. The first sample mask image has a first region to be repaired, and the first region to be repaired is located within the first sample mask image.

[0143] The first model training submodule is used to take the first sample image set and the first sample mask image as the conditional information of the first model, and input the conditional information and the preset noise information into the encoding module of the initial network model for training. When the loss value of the decoding result output by the decoding module converges, the current initial network model is taken as the first model.

[0144] In some embodiments, the target network training module described above may include:

[0145] The second mask determination submodule is used to determine a second sample mask image from a second sample mask set. The second sample mask image has a second region to be repaired, and at least one boundary line of the second region to be repaired coincides with the corresponding boundary line of the second sample mask image.

[0146] The target network training submodule is used to take the second sample image set and the second sample mask image as the conditional information of the first model, and input the conditional information and the preset noise information into the encoding module of the first model for training. When the loss value of the decoding result output by the decoding module of the first model converges, the current first model is used as the target network model.

[0147] In some embodiments, the target network training module may further include:

[0148] The initial sample acquisition submodule is used to obtain a preset sample defect image set before using the second sample image set and the second sample mask image as conditional information of the first model, and inputting the conditional information and preset noise information into the encoder of the first model for training.

[0149] The sample cropping submodule is used to crop each sample defect image in the sample defect image set to obtain a second sample image set.

[0150] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0151] It should be noted that, in the embodiments of this application... Figure 15 The module division of the illustrated electronic device is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into a single processing unit, exist as separate physical units, or be integrated into a single unit. The integrated units described above can be implemented in hardware, as software functional units, or in a combination of both.

[0152] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0153] This application provides a computer device, which may be a server, and its internal structure diagram may be as follows: Figure 16 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a method for generating defective images.

[0154] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method provided in the above embodiments.

[0155] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the method provided in the above-described method embodiments.

[0156] Those skilled in the art will understand that Figure 16 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0157] In one embodiment, the defect image generation method provided in this application can be implemented as a computer program, which can be implemented in various ways, such as... Figure 16The computer device shown operates on the computer. The computer device's memory can store various program modules that make up the electronic device. The computer program, composed of the various program modules, causes the processor to execute the steps in the defect image generation methods of the various embodiments of this application described in this specification.

[0158] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method for generating the defective image described above.

[0159] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0160] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0161] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0162] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0163] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or modules can be electrical, mechanical, or other forms.

[0164] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0165] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0166] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0167] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0168] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0169] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0170] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0171] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of generating a defect image, characterized by, The method comprises: obtaining a first defect image; inputting the first defect image into a preset target network model, and outputting a second defect image by the target network model, wherein the second defect image comprises the first defect image and an expanded image, the expanded image is obtained by expanding the background of the first defect image, and the expanded image is an image obtained by expanding in any direction around the background of the first defect image; wherein the preset target network model is obtained by training an initial network model at least twice using a training image set, the training image set comprises a sample image set and a sample mask set, the at least twice different training processes use different sample mask sets, and different sample mask sets have different repair areas.

2. The method of claim 1, wherein, The second defect image is output by the target network model, comprising: after determining a target mask from a preset mask set by the target network model, the second defect image is output according to the first defect image and the target mask.

3. The method of claim 2, wherein, The target network model comprises an encoding module and a decoding module, the encoding module is used to obtain a target encoding sequence according to the target mask, the first defect image and preset noise information, and the decoding module is used to obtain the second defect image according to the target encoding sequence and a decoding mechanism corresponding to the target network model.

4. The method of claim 1, wherein, The sample image set comprises a first sample image set and a second sample image set, the first sample image set and the second sample image set comprise different sample images, the sample mask set comprises a first sample mask set and a second sample mask set, and before the first defect image is input into the preset target network model, the method further comprises: training the initial network model according to the first sample image set and the first sample mask set to obtain a first model; training the first model according to the second sample image set and the second sample mask set to obtain the target network model.

5. The method of claim 4, wherein, The initial network model comprises an encoding module and a decoding module connected with the encoding module, and the initial network model is trained according to the first sample image set and the first sample mask set to obtain a first model, comprising: determining a first sample mask image from the first sample mask set, the first sample mask image has a first repair area, and the first repair area is located in the first sample mask image; the first sample image set and the first sample mask image are used as condition information of the first model, the condition information and preset noise information are input into the encoding module of the initial network model for training, and when the loss value of the decoding result output by the decoding module converges, the current initial network model is used as the first model.

6. The method of claim 5, wherein, The first model is trained according to the sample image set and the second sample mask set to obtain the target network model, comprising: determining a second sample mask image from the second sample mask set, the second sample mask image having a second to-be-repaired region, at least one boundary line of the second to-be-repaired region respectively coinciding with a corresponding boundary line of the second sample mask image; inputting the second sample image set and the second sample mask image as condition information of the first model, and inputting the condition information and preset noise information into an encoder of the first model for training, and taking the current first model as the target network model when a loss value of a decoding result output by a decoder of the first model converges.

7. The method of claim 6, wherein, Before the inputting the second sample image set and the second sample mask image as condition information of the first model, and inputting the condition information and preset noise information into an encoder of the first model for training, the method further comprises: obtaining a preset sample defect image set; cropping each sample defect image in the sample defect image set to obtain the second sample image set.

8. The method of any one of claims 1-7, wherein, The second defect image is obtained by expanding the background of the first defect image multiple times by the target network model, and the size of the second defect image is greater than or equal to a preset size.

9. An electronic device, comprising: The electronic device comprises: a defect image obtaining module configured to obtain a first defect image; a defect image expansion module configured to input the first defect image into a preset target network model, and output a second defect image by the target network model, the second defect image comprising the first defect image and an expansion image, the expansion image being obtained by expanding the background of the first defect image, the expansion image being an image obtained by expanding in any one direction around the background of the first defect image; wherein the preset target network model is obtained by training an initial network model at least twice using a training image set, the training image set comprising a sample image set and a sample mask set, the at least twice different training processes using different sample mask sets, the different sample mask sets having different to-be-repaired regions.

10. A computer device comprising a memory and a processor, the memory storing a computer program capable of running on the processor, characterized in that, The processor implements the steps of the method of any one of claims 1 to 8 when executing the program.

11. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the method of any one of claims 1 to 8. The computer program, when executed by the processor, implements the method of any one of claims 1 to 8.

Images