Defect image generation method, apparatus, computer equipment and storage medium

Abstract

This application relates to a method, apparatus, computer device, and storage medium for generating defect images. The method includes: acquiring a target image and target defect guidance information; the target defect guidance information includes target category defect guidance information and target image defect guidance information; preprocessing the target image to obtain first image information; inputting the first image information and the target defect guidance information into a trained defect image generation model to output a target defect image; the defect image generation model is used to obtain the first target defect image when the input target defect guidance information includes target category defect guidance information; and to obtain a second target defect image when the input target defect guidance information includes both target category defect guidance information and target image defect guidance information; the second target defect image has higher image precision than the first target defect image. Using this application, it is possible to obtain outputs with different image precision based on different inputs.

Description

Technical Field

[0001] This application relates to the field of sample generation technology, and in particular to a method, apparatus, computer device and storage medium for generating defect images. Background Technology

[0002] Defect image generation methods refer to techniques for generating sample images containing specific defects. In the field of industrial quality inspection, the more abundant the data for a specific defect category, the better the model's detection performance for that type of defect. Generating specific defect sample images through defect image generation methods can improve the model's detection performance.

[0003] Traditional image generation methods mainly fall into two categories: manual creation and deep learning model-based generation. Manual creation typically involves collecting and processing defect images in actual production environments. Deep learning model-based generation usually employs generative adversarial models to generate defect images. This method learns the distribution of real defect data to generate more realistic defect images.

[0004] Manually creating defect images is not only time-consuming, but also often results in significant differences between the created samples and real samples. In contrast, defect image generation methods based on generative adversarial models can reduce the time cost and narrow the gap between created and real samples. However, they require a large number of training samples. Given the scarcity of defect samples, most defects generated by generative adversarial models are common defects, resulting in fewer occasional and rare defect samples and a long-tail problem. Summary of the Invention

[0005] Therefore, it is necessary to provide a defect image generation method, apparatus, computer device, computer-readable storage medium, and computer program product to address the above-mentioned technical problems, which can control the position of generated defects in the feature distribution, generate scarce defect images, and avoid long-tail problems.

[0006] In a first aspect, this application provides a method for generating defective images, including:

[0007] Acquire target image and target defect guidance information; target defect guidance information includes target category defect guidance information and target image defect guidance information;

[0008] The target image is preprocessed to obtain the first image information;

[0009] The first image information and the target defect guidance information are input into the trained defect image generation model for processing, and the target defect image is output.

[0010] The trained defect image generation model is used to obtain a first target defect image when the input target defect guidance information includes target category defect guidance information; or, when the input target defect guidance information includes target category defect guidance information and target image defect guidance information, it is used to obtain a second target defect image; the second target defect image has higher image accuracy than the first target defect image.

[0011] Secondly, this application provides a defect image generation apparatus, comprising:

[0012] The defect acquisition module is used to acquire target images and target defect guidance information; the target defect guidance information includes target category defect guidance information and target image defect guidance information.

[0013] The preprocessing module is used to preprocess the target image to obtain the first image information;

[0014] The defect generation module is used to input the first image information and the target defect guidance information into the trained defect image generation model for processing and output the target defect image;

[0015] The trained defect image generation model is used to obtain a first target defect image when the input target defect guidance information includes target category defect guidance information; or, when the input target defect guidance information includes target category defect guidance information and target image defect guidance information, it obtains a second target defect image, wherein the second target defect image has higher image accuracy than the first target defect image.

[0016] Thirdly, this application provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the method described above.

[0017] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method.

[0018] Fifthly, this application provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the method described above.

[0019] The aforementioned defect image generation method, apparatus, computer equipment, computer-readable storage medium, and computer program product, by limiting the input target defect guidance information to include target category defect guidance information, outputs a first target defect image by a trained defect generation model; and by limiting the input target defect guidance information to include target category defect guidance information and target image defect guidance information, outputting a second target defect image, the second target defect image having higher image precision than the first target defect image. By limiting the input of different target defect guidance information to a single defect generation model, defect images with different image precisions can be obtained, eliminating the need for multiple models and reducing the cost and complexity of generating target defect images with different image precisions. Furthermore, this defect generation model can control the defect details of the generated target defect image by controlling the input target defect guidance information, that is, it can control the model to generate target defect images with multiple features by controlling the input information, thereby expanding the rare defect sample library and effectively solving the long-tail problem of data. Attached Figure Description

[0020] Figure 1 This application environment diagram illustrates a defect image generation method provided in an embodiment of this application.

[0021] Figure 2 A flowchart illustrating a defect image generation method provided in an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of the structure of a defect image generation model provided in an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of the structure of another defect image generation model provided in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the structure of another defect image generation model provided in an embodiment of this application;

[0025] Figure 6 A schematic flowchart illustrating another defect image generation method provided in an embodiment of this application;

[0026] Figure 7 A schematic diagram illustrating a process for obtaining a trained defect image generation model, provided in an embodiment of this application;

[0027] Figure 8 A structural block diagram of a defect image generation device provided in an embodiment of this application;

[0028] Figure 9 A structural block diagram of another defect image generation apparatus provided in an embodiment of this application;

[0029] Figure 10 An internal structural diagram of a computer device provided in an embodiment of this application;

[0030] Figure 11 An internal structural diagram of another computer device provided in an embodiment of this application;

[0031] Figure 12 This is an internal structure diagram of a storage medium provided in an embodiment of this application.

[0032] Figure label:

[0033] 102: Terminal; 104: Server; 800: Defect image generation device; 810: Defect acquisition module; 820: Preprocessing module; 830: Defect generation module; 840: Sample acquisition module. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0035] The defect image generation method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a communication network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0036] like Figure 2 As shown, this application provides a method for generating defect images, which is applied to... Figure 1 The method will be illustrated using terminal 102 or server 104 as examples. It is understood that the computer device may include at least one of terminal 102 and server 104. The method includes the following steps:

[0037] S202. Obtain the target image and target defect guidance information.

[0038] The target defect guidance information includes target category defect guidance information and target image defect guidance information. The target image refers to the image in which the defect is to be generated, and the target defect guidance information is used to guide the generation of the defect in the target image.

[0039] S204. Preprocess the target image to obtain the first image information.

[0040] Understandably, preprocessing may include cropping steps; therefore, the first image information may consist of multiple images. Furthermore, since the target image is used to generate defects, the first image information should include information about the region to be defected, so that the defect image generation model can generate the target defect in that region.

[0041] In cases where preprocessing includes cropping, if the size of the trained defect image generation model input in the first image information is smaller than the size of the target image, then the size of the final generated target defect image will be smaller than the size of the target image. In this case, the target defect image can be fused into the target image through Poisson fusion to obtain a defect image with the same size as the target image.

[0042] S206. Input the first image information and the target defect guidance information into the trained defect image generation model for processing, and output the target defect image.

[0043] The trained defect image generation model is used to obtain a first target defect image when the input target defect guidance information includes target category defect guidance information; or, when the input target defect guidance information includes target category defect guidance information and target image defect guidance information, it is used to obtain a second target defect image; the second target defect image has higher image accuracy than the first target defect image.

[0044] The aforementioned defect image generation method, by limiting the input target defect guidance information to include target category defect guidance information, outputs a first target defect image using a trained defect generation model; and by limiting the input target defect guidance information to include both target category defect guidance information and target image defect guidance information, it outputs a second target defect image, which has higher image precision than the first target defect image. By limiting the input of different target defect guidance information to a single defect generation model, defect images with different image precisions can be obtained. This eliminates the need for multiple models; a single model can generate defect images with varying image precisions, reducing the cost and complexity of generating target defect images with different precisions. Furthermore, this defect generation model can control the defect details of the generated target defect image by controlling the input target defect guidance information. This allows for the generation of target defect images with multiple features by controlling the input information, thereby expanding the rare defect sample library and effectively solving the long-tail problem of data.

[0045] In some embodiments, target category defect guidance information is used to guide the defect category of the target defect image; target image defect guidance information is used to further guide the defect details of the target defect image, given a defined defect category, where the defect details represent sub-category defects under the defect category.

[0046] For example, the target category defect guidance information can guide the target defect image to generate images of defect categories such as dirt and damage. The target image defect guidance information can further guide the target defect image to generate sub-category defects such as dirt covering and glue covering, or sub-category defects such as scratches, bumps, and missing parts, when the defect category is limited.

[0047] Inputting target category defect guidance information into a trained defect image generation model can output a first target defect image, which may be dirty or damaged. Inputting both target category defect guidance information and target image defect guidance information into the trained defect image generation model can output a second target defect image, which may be covered in dirt such as dust or glue, or a second target defect image, which may be damaged such as scratches, dents, or missing parts. Compared to the first target defect image, the second target defect image contains specific defect details and has higher image accuracy.

[0048] With the above settings, users can flexibly select the target defect guidance information in the trained defect image generation model according to their needs, thereby determining the image accuracy of the target defect image output by the trained defect image generation model. They can also control the target defect image to generate specific detailed defects under a specific category by controlling the target category defect guidance information, which can effectively expand scarce samples.

[0049] In some embodiments, the trained defect image generation model includes a backbone network, a first branch network, and a second branch network;

[0050] The first image information and the target defect guidance information are input into a trained defect image generation model for processing, and the target defect image is output, including:

[0051] See Figure 3 The first image information is input into the backbone network; the target category defect guidance information is input into the first branch network; the output features of the first branch network are input into the first attention module of the backbone network; the output features of the first attention module are input into the decoder of the backbone network; and the first target defect image is output.

[0052] In other embodiments, the trained defect image generation model includes a backbone network, a first branch network, and a second branch network.

[0053] The first image information and the target defect guidance information are input into a trained defect image generation model for processing, and the target defect image is output, including:

[0054] See Figure 3 The first image information is input into the backbone network; the target category defect guidance information is input into the first branch network; the target image defect guidance information is input into the second branch network; the output features of the first branch network are input into the first attention module; the output features of the second branch network are input into the second attention module of the backbone network; the output features of the first attention module and the second attention module are input into the fusion module of the backbone network; and the output features of the fusion module are input into the decoder to output the second target defect image.

[0055] The first attention module uses the output features of the first branch network as a query and the image feature information input to the first attention module from the backbone network as a key-value pair. The first attention module can accurately align and fuse the query with the position of the defect to be set in the image feature information, ensuring that the defect category input in the first branch network is generated at the position of the defect to be set.

[0056] The second attention module uses the output features of the second branch network as a query and the image feature information input to the second attention module from the backbone network as a key-value pair. The second attention module can accurately align and fuse the query with the position of the defect to be set in the image feature information, ensuring that defect details similar to those input to the image in the second branch network are generated at the position of the defect to be set.

[0057] The first attention module and the second attention module can both be cross-attention modules or multi-head cross-attention modules, and there are no restrictions here.

[0058] The fusion module can perform element-wise addition, feature concatenation, or attention fusion on the output features of the first attention module and the second attention module, without any restrictions.

[0059] When generating a first target defect image with relatively low image precision, the second attention module and fusion module in the backbone network do not participate in the generation of the first target defect image, reducing the computational complexity of generating low-precision target defect images and improving computational efficiency. Furthermore, by inputting the target defect guidance information into the attention module in the backbone network through either the first or second branch network for fusion, the target defect guidance information can accurately locate the region where the defect is to be set, preventing the defect image generation model from wasting attention on other regions.

[0060] In some embodiments, see Figure 3 The backbone network also includes a first encoder and a feature splicing module;

[0061] The first image information includes a first image block containing the defect area to be set, a second image block not containing the defect area to be set, and a binarized mask M1 of the first image block (hereinafter referred to as the first binarized mask of the first image block); the first image block and the second image block at least partially overlap.

[0062] The first and second image blocks can be obtained by cropping. The area in the target image to be defective is cropped into an image block of a preset size to obtain the first image block. The first image block is then masked to obtain a first binary mask image M1; the area to be defective can be marked as 1, and the background area other than the area to be defective can be marked as 0. After removing the area to be defective from the first image block, the second image block is obtained. The second image block can also be obtained by I1*(1-M1), where I1 represents the target image.

[0063] In some embodiments, before inputting the output features of the first attention module into the decoder of the backbone network and outputting the first target defect image, the method further includes:

[0064] The first image block and the second image block are input into the first encoder to obtain a first potential feature containing the defect region to be set and a second potential feature not containing the defect region to be set; the first encoder can be a variational autoencoder or other existing encoders.

[0065] The first binarized mask image, the first latent feature, and the second latent feature are input into the feature concatenation module, and features are concatenated along the channel dimension to obtain the first concatenated feature; the first concatenated feature is then input into the first attention module. The first concatenated feature serves as a key-value pair input into the first attention module.

[0066] In other embodiments, before inputting the output features of the first attention module and the second attention module into the fusion module of the backbone network, the method further includes:

[0067] The first image block and the second image block are input into the first encoder to obtain a first potential feature containing the defect region to be set and a second potential feature not containing the defect region to be set.

[0068] The binary mask image of the first image patch, the first latent feature, and the second latent feature are input into the feature concatenation module, and features are concatenated along the channel dimension to obtain the first concatenated feature. The first concatenated feature is then input into the first attention module and the second attention module, respectively. That is, the first concatenated feature serves as a key-value pair input into the first attention module and the second attention module.

[0069] The first and second latent features retain local features of different regions. The first binarized mask can accurately locate the region of interest, that is, it can accurately locate the region to be defected in this embodiment. This allows the defect image generation model to focus more on processing key regions and ignore background regions. By stitching together the first binarized mask, the first latent feature, and the second latent feature, multi-dimensional information fusion can be achieved, enhancing the expressive power of features.

[0070] In other embodiments, see Figure 4 The backbone network also includes a feature extraction module, located after the feature concatenation module and before the first and second attention modules. This module further processes the first concatenated features. This feature extraction module can include convolutional modules, pooling modules, etc., without specific limitations, and can be configured according to requirements. If a feature extraction module exists, the features input to the first and second attention modules become the output features of the feature extraction module. Simultaneously, the output features of the feature extraction module also serve as key-value pairs in the first and second attention modules, respectively.

[0071] In other embodiments, see Figure 5 The backbone network may also include a first alignment module and a second alignment module. The first alignment module can be used to align the output features of the first branch network with the output features of the feature splicing module or the feature extraction module, and the second alignment module can be used to align the output features of the second branch network with the output features of the feature splicing module or the feature extraction module.

[0072] The first alignment module and the second alignment module can simultaneously employ both linear layers and normalization layers.

[0073] Aligning the output features of the first branch network, the second branch network, and the feature concatenation or extraction module before inputting them into the attention module can reduce the computational complexity of the attention module, decrease resource consumption, and improve computational speed. Furthermore, with aligned feature dimensions, the attention module can more effectively fuse these features, thereby improving model performance.

[0074] In some embodiments, see Figures 3 to 5 The first branch network includes the second encoder;

[0075] Before inputting the output features of the first branch network into the first attention module of the backbone network, the method further includes:

[0076] The target category defect guidance information is input into the second encoder for processing, and the target category defect vector is output; the target category defect vector is used as input to the first attention module.

[0077] By encoding the target category defect guidance information into a target category defect vector, the first attention module can quickly capture the semantic information in the target category defect guidance information, which facilitates the target category defect guidance information to guide the defect category of the target image.

[0078] The target category defect guidance information includes at least one of text description information and category index information.

[0079] When the target category defect guidance information includes text description information, the second encoder includes CLIP text encoder (Contrastive Language-Image Pre-training Text Encoder), which can encode the text description information into a target category defect vector, or more specifically, a defect text description vector.

[0080] When the target category defect guidance information includes category index information, the second encoder includes a first embedding model. The first embedding model can employ an embedding layer, which can map discrete inputs into continuous, dense vector representations.

[0081] For example, the text description information could be "generating dirt" or "generating damage," and the category index information could be an index corresponding to a defect category, with category index 1 corresponding to generating dirt and category index 2 corresponding to generating damage.

[0082] The CLIP text encoder can understand complex language structures, and the embedding model can provide vector representations for discrete indices. Different encoders are used for different attributes of the target category defect guidance information, which can maximize the extraction of features from the target category defect guidance information.

[0083] In some embodiments, see Figures 3 to 5 The second branch network includes the third encoder.

[0084] Before inputting the output features of the second branch network into the second attention module of the backbone network, the method further includes:

[0085] The target image defect guidance information is input into the third encoder for processing, and the target image defect features are output. The target image defect features are then used as input to the second attention module.

[0086] Among them, the target image defect guidance information can be understood as the defect reference image.

[0087] The third encoder can employ an embedded architecture such as a Convolutional Neural Network (CNN) or a Transformer model. By using an embedded architecture to encode the defect guidance information of the target image, it can learn the feature representation of the defect.

[0088] In some embodiments, see Figure 6 Before inputting the first image information and the target defect guidance information into the trained defect image generation model for processing and outputting the target defect image, the method further includes:

[0089] S205. Train the preset defect image generation model to obtain the trained defect image generation model.

[0090] The step number does not restrict the order in which the steps are executed. For example, step S205 can be executed before step S204, and step S205 can also be executed before step S202. However, during the execution process, it is necessary to satisfy the condition that step S205 is executed before step S206 and step S202 is executed before step S204.

[0091] See Figure 7 Step S205 specifically includes:

[0092] S2051. Obtain the sample image set and the defect guidance information set.

[0093] The sample images in the sample image set are images containing defects. The defect guidance information in the defect guidance information set and the aforementioned target defect guidance information contain essentially the same defects, including category defect guidance information and image defect guidance information.

[0094] S2052. Construct a pre-defined defect image generation model.

[0095] The preset defect image generation model includes a backbone network, a first branch network, and a second branch network. The structure and function of the backbone network, the first branch network, and the second branch network have been described in detail above and will not be repeated here.

[0096] S2053. Preprocess the sample images in the sample image set to obtain the second image information.

[0097] The methods for preprocessing sample images and target images are the same, both including cropping and mask annotation.

[0098] Specifically, the defective regions in the sample image are cropped into an image block of a preset size to obtain a third image block. The third image block is then masked, with defective regions marked as 1 and non-defective regions marked as 0, resulting in a binary mask M2 (hereinafter referred to as the second binarized mask). Removing the defective regions from the third image block yields a fourth image block, which can also be obtained using I2*(1-M2), where I2 represents the sample image.

[0099] Cropping allows for focusing on key regions and removing irrelevant background information from sample images. This enables the defect image generation model to concentrate on the defect area, reduces interference from background noise during training, and standardizes the size of the input defect image generation model. Mask annotation allows for precise location of the defect area.

[0100] S2054. Input the defect guidance information from the second image information and the defect guidance information set into the preset defect image generation model for training until the training result reaches the preset target, and obtain the trained defect image generation model.

[0101] For example, achieving the preset target in training results could be achieved by reaching a preset number of training iterations, or by the loss function reaching a preset threshold, etc., without any restrictions here.

[0102] Once the trained defect image generation model is obtained, it can be put into use. During use, the parameters of the trained defect image generation model will not be updated again, which can prevent overfitting.

[0103] Obtaining a trained defect image generation model and not updating its parameters during use not only reduces computation but also ensures more stable output from the trained defect image generation model.

[0104] In some embodiments, after preprocessing the sample images in the sample image set to obtain the second image information, the method further includes:

[0105] The category defect guidance information in the second image information and the defect guidance information set are respectively input into the backbone network and the first branch network for the first stage of training until the training result reaches the first preset target, and the parameters of the modules in the backbone network that participated in the first stage of training are frozen.

[0106] Referring to the architecture of the aforementioned defective image generation model, the modules participating in the first stage of training in the frozen backbone network can include the first attention module. If a feature extraction module is present, the modules participating in the first stage of training can also include the feature extraction module. Therefore, the parameters of the frozen modules can include the parameters of the first attention module, and can also include the parameters of the feature extraction module.

[0107] For example, during the first stage of training, a variational autoencoder can be used to encode the second image. Specifically, a variational autoencoder is used to encode the third and fourth image blocks obtained from the second image, corresponding to the third and fourth latent features, respectively. Alternatively, the third latent feature containing defect region information can be forward-divided with T random noise superpositions to obtain noisy features. With noise injection into the third latent feature, the feature concatenation module concatenates the second binarized mask image, the noisy features, and the fourth latent feature along the channel dimension to obtain the second concatenated feature. If the defect image generation model does not have a feature extraction module, the second concatenated feature is input to the first attention module. If the defect image generation model has a feature extraction module, the second concatenated feature processed by the feature extraction module is output to the first attention module. The output feature of the first branch network is also input to the first attention module. The defect image generation model performs T iterations of diffusion-based backpropagation denoising, outputting the predicted denoised fifth latent feature. The mean squared error (MSE) between the predicted noise and the added noise is used as the loss function. The parameters of the modules participating in the first stage of training are optimized through backpropagation. The backbone network participating in the first stage of training is referred to as the third branch network. After the first stage of training, the defect image generation model can generate defect images of a specified category based on the target defect category guidance information.

[0108] The image defect guidance information from the defect guidance information set is input into the second branch network for the second stage of training. The parameters of the remaining modules in the backbone network that participate in the second stage of training, except for the frozen modules, are updated until the training result reaches the second preset target, and the trained defect image generation model is obtained.

[0109] For example, during the second stage of training, the second image information is input into the frozen third branch network, and the category defect information is input into the first branch network. The output features of the first attention module in the third branch network are input into the fusion module. The second stitched features output by the feature stitching module in the frozen third branch network or the second stitched features processed by the feature extraction module are also input into the second attention module. The features output by the second branch network are also input into the second attention module. The output features of the second attention module are input into the fusion module. The fusion module merges the output features of the first attention module and the second attention module and outputs the fused feature. The fused feature is used as the predicted sixth latent feature. The sixth latent feature is used to guide the reverse denoising of the defect image generation model. The second stage of training is the same as the first stage of training, but only the parameters of the modules in the backbone network that participate in the second stage of training and are not frozen are updated, such as the second attention module and the fusion module.

[0110] The statement that the training results reach the first or second preset target can be understood by referring to the statement that the training results reach the preset target, and will not be elaborated here.

[0111] During the first stage of training, the backbone network can focus on optimizing the parameters of the module that generates the first target defect image, ensuring that this module achieves optimal performance. After the first stage of training is complete, freezing the parameters of some modules prevents these parameters from being interfered with by the second stage of training, thus maintaining the optimal performance of the aforementioned modules. It is understandable that the backbone network is used not only to generate the fifth latent feature but also the sixth latent feature. If the defect image generation model is trained directly, parameter update conflicts may occur during the generation of the fifth and sixth latent features, making it difficult for the defect image generation model to converge. Staged training avoids these parameter update conflicts.

[0112] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0113] Based on the same inventive concept, this application also provides a defect image generation apparatus 800. The solution provided by this apparatus is similar to the solution described in the above method. Therefore, the specific limitations of one or more defect image generation apparatus 800 embodiments provided below can be found in the limitations of the defect image generation method above, and will not be repeated here.

[0114] See Figure 8 This application provides a defect image generation apparatus 800, comprising:

[0115] The defect acquisition module 810 is used to acquire the target image and target defect guidance information; the target defect guidance information includes target category defect guidance information and target image defect guidance information.

[0116] Preprocessing module 820 is used to preprocess the target image to obtain first image information;

[0117] The defect generation module 830 is used to input the first image information and the target defect guidance information into the trained defect image generation model for processing and output the target defect image.

[0118] The trained defect image generation model is used to obtain a first target defect image when the input target defect guidance information includes target category defect guidance information; or, when the input target defect guidance information includes target category defect guidance information and target image defect guidance information, it is used to obtain a second target defect image; the second target defect image has higher image accuracy than the first target defect image.

[0119] In some embodiments, see Figure 9 The defect image generation device 800 also includes a sample acquisition module 840;

[0120] The sample acquisition module 840 is used to acquire sample image sets and defect guidance information sets;

[0121] The preprocessing module 820 is also used to preprocess the sample images in the sample image set to obtain the second image information.

[0122] The defect generation module 830 is also used to construct a preset defect image generation model, and input the second image information and the defect guidance information in the defect guidance information set into the preset defect image generation model for training until the training result reaches the preset target, thus obtaining the trained defect image generation model.

[0123] The limitations on sample images, defect guidance information, preset defect image generation models, and the preset defect image generation model training process are the same as those for the limitations on defect image generation methods mentioned above, and will not be repeated here.

[0124] By training a pre-defined defect image generation model before putting it into use, the amount of computation can be reduced and the output can be stabilized.

[0125] Each module in the aforementioned defect image generation device 800 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0126] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to the defect image generation method. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the defect image generation method described above.

[0127] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 11As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it performs the steps in the aforementioned defect image generation method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen; the input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs or touchpads set on the casing of the computer device, or external keyboards, touchpads or mice, etc.

[0128] Those skilled in the art will understand that Figure 10 or Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0129] In some embodiments, a computer device is provided, the computer device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0130] In some embodiments, such as Figure 12 The diagram shows the internal structure of a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the above-described method embodiments.

[0131] In some embodiments, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0132] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0133] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0134] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for generating defect images, characterized in that, include: Acquire target image and target defect guidance information; The target defect guidance information includes target category defect guidance information and target image defect guidance information; The target category defect guidance information is used to guide the defect category of the target defect image; the target image defect guidance information is used to further guide the defect details of the target defect image within the defined defect category; the defect details represent sub-category defects under the defect category; the target category defect guidance information includes at least one of text description information and category index information; the target image defect guidance information is a defect reference image; The target image is preprocessed to obtain first image information; The first image information and the target defect guidance information are input into a trained defect image generation model for processing, and the target defect image is output. The trained defect image generation model includes a backbone network, a first branch network, and a second branch network. The step of inputting the first image information and the target defect guidance information into a trained defect image generation model for processing and outputting a target defect image includes: The first image information is input into the backbone network; the target category defect guidance information is input into the first branch network; the output features of the first branch network are input into the first attention module of the backbone network; the output features of the first attention module are input into the decoder of the backbone network; and the first target defect image is output. The first image information is input into the backbone network; the target category defect guidance information is input into the first branch network; the target image defect guidance information is input into the second branch network; the output features of the first branch network are input into the first attention module; the output features of the second branch network are input into the second attention module of the backbone network; the output features of the first attention module and the second attention module are input into the fusion module of the backbone network; the output features of the fusion module are input into the decoder, and a second target defect image is output; the second target defect image has higher image accuracy than the first target defect image.

2. The method according to claim 1, characterized in that, The backbone network also includes a first encoder and a feature splicing module; The first image information includes a first image block containing the defect region to be set, a second image block not containing the defect region to be set, and a binarized mask image of the first image block; the first image block and the second image block at least partially overlap. Before inputting the output features of the first attention module into the decoder of the backbone network and outputting the first target defect image, the method further includes: The first image block and the second image block are input into the first encoder to obtain a first potential feature containing the defect region to be set and a second potential feature not containing the defect region to be set. The binary mask image of the first image patch, the first latent feature, and the second latent feature are input into the feature stitching module, and feature stitching is performed along the channel dimension to obtain the first stitched feature; the first stitched feature is then input into the first attention module; or... Before inputting the output features of the first attention module and the second attention module into the fusion module of the backbone network, the method further includes: The first image block and the second image block are input into the first encoder to obtain a first potential feature containing the defect region to be set and a second potential feature not containing the defect region to be set. The binary mask image of the first image block, the first latent feature, and the second latent feature are input into the feature stitching module, and the features are stitched along the channel dimension to obtain the first stitched feature; the first stitched feature is then input into the first attention module and the second attention module respectively.

3. The method according to claim 1, characterized in that, The first branch network includes a second encoder; Before inputting the output features of the first branch network into the first attention module of the backbone network, the method further includes: The target category defect guidance information is input into the second encoder for processing, and a target category defect vector is output; the target category defect vector is used as input to the first attention module. When the target category defect guidance information includes the text description information, the second encoder includes a CLIP text encoder; or, When the target category defect guidance information includes the category index information, the second encoder includes a first embedding model.

4. The method according to claim 1, characterized in that, The second branch network includes a third encoder; Before inputting the output features of the second branch network into the second attention module of the backbone network, the method further includes: The target image defect guidance information is input to the third encoder for processing, and the target image defect features are output; the target image defect features are used to input the second attention module.

5. The method according to claim 1, characterized in that, Before inputting the first image information and the target defect guidance information into the trained defect image generation model for processing and outputting the target defect image, the method further includes: Acquire sample image sets and defect guidance information sets; Construct a pre-defined defect image generation model; The sample images in the sample image set are preprocessed to obtain the second image information; The second image information and the defect guidance information in the defect guidance information set are input into the preset defect image generation model for training until the training result reaches the preset target, thus obtaining the trained defect image generation model.

6. The method according to claim 5, characterized in that, After preprocessing the sample images in the sample image set to obtain the second image information, the method further includes: The second image information and the category defect guidance information in the defect guidance information set are respectively input into the backbone network and the first branch network for the first stage of training until the training result reaches the first preset target, and the parameters of the modules in the backbone network that participated in the first stage of training are frozen. The image defect guidance information in the defect guidance information set is input into the second branch network for the second stage of training. The parameters of the remaining modules in the backbone network that participate in the second stage of training, except for the frozen modules, are updated until the training result reaches the second preset target, and the trained defect image generation model is obtained.

7. A defect image generation apparatus, characterized in that, include: The defect acquisition module is used to acquire the target image and target defect guidance information; The target defect guidance information includes target category defect guidance information and target image defect guidance information; The target category defect guidance information is used to guide the defect category of the target defect image; the target image defect guidance information is used to further guide the defect details of the target defect image within the defined defect category; the defect details represent sub-category defects under the defect category; the target category defect guidance information includes at least one of text description information and category index information; the target image defect guidance information is a defect reference image; The preprocessing module is used to preprocess the target image to obtain first image information; The defect generation module is used to input the first image information and the target defect guidance information into the trained defect image generation model for processing and output the target defect image. The trained defect image generation model includes a backbone network, a first branch network, and a second branch network. In the process of inputting the first image information and the target defect guidance information into a trained defect image generation model for processing and outputting a target defect image, the defect generation module is specifically used for: The first image information is input into the backbone network; the target category defect guidance information is input into the first branch network; the output features of the first branch network are input into the first attention module of the backbone network; the output features of the first attention module are input into the decoder of the backbone network; and the first target defect image is output. The first image information is input into the backbone network; The target category defect guidance information is input into the first branch network, the target image defect guidance information is input into the second branch network, the output features of the first branch network are input into the first attention module, the output features of the second branch network are input into the second attention module of the backbone network, the output features of the first attention module and the second attention module are respectively input into the fusion module of the backbone network, and the output features of the fusion module are input into the decoder to output a second target defect image; the second target defect image has higher image accuracy than the first target defect image.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images