Defect image generation method, apparatus, computer equipment and storage medium

Abstract

The present application relates to a defect image generation method, device, computer equipment and storage medium. The method includes: extracting regions of the acquired industrial appearance defect image to be generated to obtain an appearance defect region image and an industrial background region image; inputting the appearance defect region image and its corresponding defect superimposed mask image into the coding network in the image generation model for processing, and outputting an appearance coding feature map and a mask coding feature map; determining a diffusion feature map according to the appearance coding feature map, the mask coding feature map and the appearance defect mask image corresponding to the appearance defect region image; inputting the diffusion feature map into the decoding network in the image generation model to output a target defect region image; performing image fusion of the target defect region image corresponding to the appearance defect region image and the industrial background region image to obtain a target industrial appearance defect image. The present application can improve the efficiency and accuracy of industrial appearance defect image generation.

Description

Technical Field

[0001] The present application relates to the field of image technology, and in particular to a defect image generation method, device, computer equipment and storage medium. Background Art

[0002] With the continuous development of detection technology, the demand for product defect detection in industrial scenarios is increasing day by day. As a typical data-driven algorithm, deep learning models are widely used in the process of industrial product defect detection. In the process of training deep learning models, the larger the number of image samples of a certain type of defect, the better the model convergence effect. However, some industrial appearance defect categories are difficult to collect in the production line.

[0003] To alleviate the data shortage problem of certain industrial appearance defect categories, the current solution is for production line workers to manually create defect images and pre-train multiple models to perform image diffusion on different types of defect images. However, artificially created defects often differ greatly from the actual defect morphology, resulting in a large deviation between the industrial appearance defect images predicted by the model and the actual appearance defects of the product. In addition, the manual production process and the training process of a large number of models are time-consuming, resulting in a large waste of human resources, which leads to low efficiency in defect image generation. Summary of the invention

[0004] Based on this, it is necessary to provide a defect image generation method, device, computer equipment and storage medium to address the above technical problems, which can improve the efficiency and accuracy of industrial appearance defect image generation.

[0005] In a first aspect, the present application provides a defect image generation method, comprising:

[0006] Extracting defect regions from the acquired industrial appearance defect image to be generated to obtain an appearance defect region image and an industrial background region image;

[0007] Respectively generate an appearance defect mask image and a defect superposition mask image corresponding to the appearance defect area image;

[0008] The appearance defect area image and the defect superimposed mask image are respectively input into the encoding network in the trained image generation model for processing, and the appearance encoding feature map corresponding to the appearance defect area image and the mask encoding feature map corresponding to the defect superimposed mask image are output;

[0009] Determine a diffusion feature map according to the appearance coding feature map, the mask coding feature map and the appearance defect mask image;

[0010] The diffusion feature map is input into the decoding network in the image generation model for processing, and the target defect area image is output;

[0011] The target defect area image corresponding to the appearance defect area image is fused with the industrial background area image to obtain the target industrial appearance defect image.

[0012] In a second aspect, the present application provides a defect image generating device, comprising:

[0013] An extraction module is used to extract defect areas from the acquired industrial appearance defect image to be generated, so as to obtain an appearance defect area image and an industrial background area image;

[0014] A generating module, used to respectively generate an appearance defect mask image and a defect superposition mask image corresponding to the appearance defect region image;

[0015] A processing module, used to input the appearance defect area image and the defect superimposed mask image into the encoding network in the trained image generation model for processing, and output an appearance coding feature map corresponding to the appearance defect area image and a mask coding feature map corresponding to the defect superimposed mask image;

[0016] A determination module, used to determine a diffusion feature map according to the appearance coding feature map, the mask coding feature map and the appearance defect mask image;

[0017] The generation module is also used to input the diffusion feature map into the decoding network in the image generation model for processing, and output the target defect area image;

[0018] The fusion module is used to fuse the target defect area image corresponding to the appearance defect area image with the industrial background area image to obtain the target industrial appearance defect image.

[0019] In a third aspect, the present application provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method when executing the computer program.

[0020] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above method when executed by a processor.

[0021] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and the computer program implements the steps in the above method when executed by a processor.

[0022] Compared with the prior art, the above-mentioned defect image generation method, device, computer equipment and storage medium achieve the automatic generation of appearance defect images of industrial products by adopting a trained stable diffusion model to generate industrial appearance defect images, and by stably diffusing the feature images output by the encoding network of the image generation model. In addition, different industrial appearance defect categories are effectively compatible during the image generation process, and a single model can be used to simultaneously generate high-quality industrial appearance defect images with reasonable defect scale differences, thereby improving the efficiency and accuracy of industrial appearance defect image generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A diagram of an application environment of a defect image generation method provided in an embodiment of the present application;

[0024] Figure 2 A schematic diagram of a defect image generation method provided in an embodiment of the present application;

[0025] Figure 3 A schematic diagram of a flow chart of another defect image generation method provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of a defect area extraction process of a defect sample image provided in an embodiment of the present application;

[0027] Figure 5 A schematic diagram of a model training process of a stable diffusion model provided in an embodiment of the present application;

[0028] Figure 6 A schematic diagram of a defect image generation process provided in an embodiment of the present application;

[0029] Figure 7 A structural block diagram of a defect image generating device provided in an embodiment of the present application;

[0030] Figure 8 An internal structure diagram of a computer device provided in an embodiment of the present application;

[0031] Fig. 9 An internal structure diagram of another computer device provided in an embodiment of the present application;

[0032] Fig.10 An internal structure diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0034] The defect image generation method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through a communication network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0035] like Figure 2 As shown, the embodiment of the present application provides a defect image generation method, which is applied to Figure 1 The terminal 102 or the server 104 in the example is used for explanation. It is understandable that the computer device may include at least one of the terminal and the server. The method comprises the following steps:

[0036] S202 , extracting defect regions from the acquired industrial appearance defect image to be generated, to obtain an appearance defect region image and an industrial background region image.

[0037] S204 , respectively generating an appearance defect mask image and a defect superposition mask image corresponding to the appearance defect region image.

[0038] S206, respectively inputting the appearance defect area image and the defect superimposed mask image into the encoding network in the trained image generation model for processing, and outputting the appearance coding feature map corresponding to the appearance defect area image and the mask coding feature map corresponding to the defect superimposed mask image.

[0039] S208: Determine a diffusion feature map according to the appearance coding feature map, the mask coding feature map, and the appearance defect mask image.

[0040] S210, inputting the diffusion feature map into the decoding network in the image generation model for processing, and outputting the target defect area image.

[0041] S212, fusing the target defect area image corresponding to the appearance defect area image with the industrial background area image to obtain a target industrial appearance defect image.

[0042] The industrial appearance defect image to be generated may be an image that needs to be diffused for appearance defects, which may be obtained from image detection equipment in an actual industrial product production line; or, it may be produced in small quantities and accurately by relevant technicians based on actual industrial appearance defects. It should be noted that the industrial appearance defect image to be generated is usually a defect image that is difficult or impossible to collect in large quantities in an actual industrial product production line.

[0043] Exemplarily, a pre-trained defect area localization model can be used to locate the appearance defect area image, and the industrial appearance defect image to be generated can be cropped according to the defect area position coordinates output by the defect area localization model to obtain the appearance defect area image and the industrial background area image.

[0044] Specifically, the appearance defect region image may be subjected to image binarization processing to obtain an appearance defect mask image; and the appearance defect mask image and the appearance defect region image may be subjected to image superposition processing to obtain a defect superposition mask image.

[0045] The image generation model may be a VAE (Variational Autoencoder), including an encoding network and a decoding network. The image generation model may be obtained by relevant technicians through pre-training based on a large number of appearance defect samples in industrial scenarios or appearance defect samples in natural scenarios.

[0046] The encoding network of the image generation model extracts image features of the appearance defect area image to obtain an appearance encoding feature map; and the encoding network of the image generation model extracts image features of the defect superimposed mask image to obtain a mask encoding feature map.

[0047] Exemplarily, the appearance coding feature map, the mask coding feature map and the appearance defect mask image can be subjected to image diffusion processing respectively, and specifically, they can be input into a pre-trained stable diffusion model to obtain an appearance feature diffusion map corresponding to the appearance coding feature map, a first mask feature diffusion map corresponding to the mask coding feature map and a second mask feature diffusion map corresponding to the appearance defect mask image respectively; the appearance feature diffusion map, the first mask feature diffusion map and the second mask feature diffusion map are fused to obtain a diffusion feature map. The diffusion feature map is input into the encoding network in the trained image generation model to reconstruct the feature image and obtain the target defect area image. Specifically, the target defect area image can be an image with defect features similar to those of the appearance defect area image.

[0048] The target defect area image corresponding to the appearance defect area image is fused with the industrial background area image to obtain a target industrial appearance defect image. The specific fusion method may be a distance weighted average replacement method of the surrounding pixel values ​​or a direct cropping replacement method. The target industrial appearance defect image is a defect image that is a fusion of the industrial scene area image and the industrial appearance defect image to be generated.

[0049] It can be seen that in the embodiment of the present application, by adopting the trained stable diffusion model to generate industrial appearance defect images, the automatic generation of industrial appearance defect images that are difficult to collect in industrial scenes is achieved. By stably diffusing the feature image output by the encoding network of the image generation model, the image quality of the generated appearance defect image is improved, and the defect scale difference is effectively controlled within a reasonable range; high-quality industrial appearance defect images can be achieved through a single model without the need to train different types of image generation models for different types of defects, which improves the efficiency of industrial appearance defect generation while ensuring the accuracy of industrial appearance defect generation.

[0050] In order to further improve the generation accuracy of the stable diffusion feature map, the text information of the defect text description and the category description can be combined in the feature diffusion process, and random noise factors can be introduced in the feature map fusion process to increase the diversity of image generation and accurately control the detail recovery of the generation process. Figure 3 A flowchart of another method for generating an industrial appearance defect image provided in an embodiment of the present application. This embodiment is further optimized on the basis of the above embodiment. Figure 3 As shown, the method includes:

[0051] S302 , extracting defect regions from the acquired industrial appearance defect image to be generated, to obtain an appearance defect region image and an industrial background region image.

[0052] S304 , respectively generating an appearance defect mask image and a defect superposition mask image corresponding to the appearance defect region image.

[0053] S306, respectively inputting the appearance defect area image and the defect superimposed mask image into the encoding network in the trained image generation model for processing, and outputting the appearance coding feature map corresponding to the appearance defect area image and the mask coding feature map corresponding to the defect superimposed mask image.

[0054] S308: Determine a fused coding feature map according to the appearance coding feature map, the mask coding feature map and the appearance defect mask image.

[0055] S310: Determine conditional coding features of the appearance defect area image.

[0056] S312: Input the fused coding feature map and the conditional coding feature into the trained stable diffusion model for processing, and output the diffusion feature map.

[0057] S314, input the diffusion feature map into the decoding network in the image generation model for processing, and output the target defect area image.

[0058] S316, fusing the target defect area image corresponding to the appearance defect area image with the industrial background area image to obtain a target industrial appearance defect image.

[0059] A feature extraction operation can be performed on the appearance defect mask image, and an image fusion operation can be performed on the extracted feature map, the appearance coding feature map and the mask coding feature map to obtain a fused coding feature map.

[0060] To further improve the generation accuracy of the fused coding feature map, in some embodiments, the fused coding feature map is determined according to the appearance coding feature map, the mask coding feature map and the appearance defect mask image, including:

[0061] Down-sampling the appearance defect mask image to obtain a sampling coding feature map;

[0062] Performing noise superposition processing on the appearance coding feature map to obtain a noise coding feature map;

[0063] The sampling coding feature map, the noise coding feature map and the mask coding feature map are fused to obtain a fused coding feature map.

[0064] The sampling multiple of the appearance defect mask image can be preset by relevant technical personnel according to actual needs. For example, the sampling multiple can be set to 8 times the width and height.

[0065] The noise superposition processing of the appearance coding feature map may specifically be a forward diffusion operation, that is, a preset number T of noise superposition operations are randomly performed, and the specific superimposed noise categories may include one or more categories, which is not limited in this embodiment.

[0066] The sampling coding feature map, the noise coding feature map and the mask coding feature map are subjected to a feature map fusion operation. Specifically, the feature maps can be spliced ​​according to image channels to obtain a fused coding feature map.

[0067] It can be seen that in this embodiment, by downsampling the appearance defect mask image and performing noise superposition processing on the appearance coding feature map, the diversity of the generated image is improved, and it is not limited to the potential features of the input. The image details of the input image are effectively reflected, and the image detail level is finely controlled, thereby improving the image quality of the subsequently generated diffusion feature map.

[0068] The conditional coding feature may be defect text description information of the appearance defect region image, and specifically, the defect text description may be performed according to the defect feature of the appearance defect region image.

[0069] In some embodiments, determining the conditional coding features of the appearance defect area image includes:

[0070] Determine defect text description information and defect category index information according to the defect category of the appearance defect area image;

[0071] Generate a defect text description vector corresponding to the defect text description information and a defect category index vector corresponding to the defect category index information respectively;

[0072] Generate conditional encoding features including defect text description vector and defect category index vector.

[0073] Among them, defect categories may include scratch defects, dent defects, and other types; defect text description information may be text information describing defect characteristics; defect category index information may be index identification information that can uniquely characterize defect categories; defect text description information and defect category index information for different defect categories are different. For example, the defect text description information for a scratch defect may be "a photo with scratch", and the corresponding defect category index information is "01"; the defect text description information for a dent defect may be "a photo with dent", and the corresponding defect category index information is "02".

[0074] Generate a defect text description vector corresponding to the defect text description information, specifically, input the defect text description information into CILPTextModel (Contrastive Language-Image Pre-training, a pre-training method or model based on contrastive text-image pairs) to perform text information encoding processing, and obtain a defect text description vector corresponding to the defect text description information. Among them, CILPTextModel can be a model that can encode text information in multimodal information and obtain text representation features.

[0075] Generate a defect category index vector corresponding to the defect category index information, specifically input the defect category index information into an Embedding model to obtain a defect category index vector output by the model. The Embedding model is specifically a model that converts discrete categorical data into continuous vector data, which can identify defect categories to a specific space to obtain category encoding, thereby performing defect category control in a more explicit manner. Generate conditional coding features including a defect text description vector and a defect category index vector.

[0076] It can be seen that in this embodiment, by introducing the defect text description vector and the defect category index vector, the richness of the generated conditional coding features is improved, thereby improving the generation quality of the subsequent diffusion feature map, and further improving the generation quality of the subsequent target defect area image.

[0077] The fusion coding feature map and the conditional coding feature are input into the trained stable diffusion model for processing to obtain the diffusion feature map output by the model. The stable diffusion model can adopt an existing model or can be pre-trained based on an industrial appearance defect application scenario.

[0078] This embodiment combines the industrial appearance application scenario and further provides a model training method for an image generation model. In some embodiments, before the appearance defect area image and the defect superimposed mask image are respectively input into the coding network in the trained image generation model for processing, and the appearance coding feature map corresponding to the appearance defect area image and the mask coding feature map corresponding to the defect superimposed mask image are output, the method further includes:

[0079] Extracting defective regions from the acquired industrial appearance defect sample images to obtain defective region sample images;

[0080] The sample image of the defect area is input into the encoding network in the preset image generation model for processing, and the defect sample feature map is output; the weight parameter of the encoding network is the initial weight parameter;

[0081] Input the defect sample feature map into the decoding network in the image generation model for processing, and output the predicted defect area image;

[0082] The decoding network of the image generation model is trained according to the predicted defect area image and the defect area sample image until the preset model training end condition is met to obtain a trained image generation model.

[0083] The industrial appearance defect sample images can be specifically collected from industrial production lines, and the number of industrial appearance defect sample images can be determined according to actual needs, for example, it can be 10 images.

[0084] It should be noted that since there may be multiple defects in the industrial appearance defect sample image and the sizes of each defect in the image are different, in order to ensure that the foreground features of smaller defects are not submerged in the background features while retaining some background information and making the generated foreground and background boundaries of large and small defects more reasonable, the defect area can be extracted by setting the cropping ratio.

[0085] In some embodiments, extracting defect areas from the acquired industrial appearance defect sample images to obtain defect area sample images includes:

[0086] Performing image segmentation on the defect area contour in the acquired industrial appearance defect sample image to obtain a defect contour image;

[0087] Determine the defect contour area of ​​the defect contour image; determine the target image size based on the defect contour area and a preset foreground-background ratio range interval;

[0088] Based on the target image size, the defect area in the industrial appearance defect sample image is cropped to obtain a defect area sample image.

[0089] The foreground-background ratio range interval may be pre-set by relevant technical personnel according to actual needs. For example, the foreground-background ratio range interval may be [0.125-0.5].

[0090] The defect area contour in the industrial appearance defect sample image is located, and the located defect area contour is segmented to obtain a defect contour image. The defect contour area of ​​the defect contour image is determined, specifically by counting the number of pixels. According to the defect contour area of ​​the defect contour image and the image area of ​​the industrial appearance defect sample image, the target image size is determined based on a preset foreground-background ratio range. According to the target image size, the defect area in the industrial appearance defect sample image is cropped to obtain a defect area sample image.

[0091] like Figure 4 A schematic diagram of a defect area extraction process of an industrial appearance defect sample image is shown. The industrial appearance defect sample image includes defect A and defect B. Based on a preset foreground-background ratio range, the defect area contour of defect A is cropped to obtain a defect area sample image A; and based on a preset foreground-background ratio range, the defect area contour of defect B is cropped to obtain a defect area sample image B.

[0092] It can be seen that in this embodiment, by cropping the industrial appearance defect image based on the preset foreground-background ratio, the foreground features of small defects will not be submerged in the background features during the subsequent model training process. At the same time, some background information is retained, making the generated foreground-background boundary of large defects more reasonable.

[0093] In order to more efficiently utilize the training computing power and reduce the mutual interference of images of different sizes during the training process, the defect area sample images of different sizes are trained in batches according to different sizes. For example, the image sizes of the defect area sample images obtained include 512*512, 256*256, 128*128, 64*64 and 32*32, and the training batch sizes are set to 4, 8, 16, 32 and 64 respectively according to the image size.

[0094] The preset image generation model VAE is trained using sample images of defective areas. Under normal circumstances, the image generation model is usually trained directly using an open source sample library constructed by a natural language processing task to obtain an iterative image generation model, and the weight parameters of the model are the initial weight parameters. However, due to the large differences between industrial scenes and natural scenes, and the fact that this application needs to be compatible with many industrial scenes, such as metal casings, plastic casings, PCB (Printed Circuit Board) panels, and glass surfaces, the direct and complete use of the initial weight parameters cannot meet the industrial scene requirements of this application. Therefore, in the process of model training of the image generation model, this embodiment uses the initial weight parameters as the weight parameters of the encoding network, and fine-tunes the weight parameters of the decoding network to meet the needs of appearance defect detection in industrial scenes.

[0095] Input the sample image of the defective area into the encoding network in the preset image generation model for processing, and obtain the defective sample feature map output by the encoding network; input the defective sample feature map into the decoding network in the image generation model for processing, and obtain the predicted defective area image output by the decoding network; determine the loss value of the encoding network based on the predicted defective area image and the sample image of the defective area and the preset loss function; train the decoding network of the image generation model according to the loss value of the encoding network until the preset model training end condition is met, and obtain the trained image generation model. Among them, the loss function may include the MSE (Mean Squared Error) loss function and the LPIPS (Learned Perceptual Image Patch Similarity) loss function, etc. The end condition of the model training may be that the loss value of the encoding network reaches the preset loss threshold, or the loss value of the encoding network tends to be stable, etc.

[0096] In the above embodiment, during the model training process of the image generation model, the initial weight parameters of the encoding network in the natural scene are used, and the weight parameters of the decoding network are iteratively trained based on the image features in the industrial appearance scene, and the differences between the industrial scene and the natural scene are taken into account during the training process. By fine-tuning the encoding weight parameters, the image generation model can be compatible with multiple industrial appearance defect scenes, thereby improving the quality of defect image generation in industrial scenes.

[0097] This embodiment combines the application scenario of industrial appearance defects and further provides a method for training a stable diffusion model. In some embodiments, before inputting the fused coding feature map and the conditional coding feature into the trained stable diffusion model for processing and outputting the diffusion feature map, the method further includes:

[0098] Extracting defective regions from the acquired industrial appearance defect sample images to obtain defective region sample images;

[0099] Respectively generate a defect mask sample image and a superimposed mask sample image corresponding to the defect area sample image, and determine a sample coding feature of the defect area sample image;

[0100] The defect area sample image and the superimposed mask sample image are respectively input into the encoding network in the trained image generation model for processing, and the appearance sample feature map corresponding to the defect area sample image and the mask sample feature map corresponding to the superimposed mask sample image are output;

[0101] Determine a fused sample feature map according to the appearance sample feature map, the mask sample feature map and the defect mask sample image;

[0102] The fused sample feature map and the sample coding feature are input into the preset stable diffusion model for processing, and the predicted sample feature map is output;

[0103] The stable diffusion model is trained according to the fused sample feature map and the predicted sample feature map until a preset model training end condition is met, thereby obtaining a trained stable diffusion model.

[0104] The defect area sample image is binarized to obtain a defect mask sample image corresponding to the defect area sample image; the defect area sample image and the defect mask sample image are superimposed to obtain a superimposed mask sample image.

[0105] Among them, the sample coding features of the defect area image may include a sample defect text description vector and a sample defect category index vector; the sample defect text description vector is obtained by vector encoding the sample defect text description information by the CILPTextModel model; the sample defect category index vector is obtained by vector encoding the sample defect category index information by the Embedding model. Both the CILPTextModel model and the Embedding model can be used as text coding models. Specifically, 3 to 4 character special identifiers can be randomly generated for each type of defect area image, and the character special identifier may not have actual meaning, and the defect English name is used to describe the defect.

[0106] It should be noted that since the text encoding model is introduced while training the stable diffusion model, uncommon prefixes can be introduced in the text-guided description to avoid the features corresponding to the text description from deviating further from the original pre-trained model, which affects the generation quality and increases the training time. In addition, using the English names corresponding to the defects can make full use of the original pre-trained weights and reduce the model iteration time. For example, the naming rules for the sample defect text description information of the defect area image can be as follows:

[0107] Bruise type naming: {a photo with "sys dent"};

[0108] Scratch type naming: {a photo with "dixg scratch"};

[0109] Among them, "sys" and "dixg" are the preset uncommon prefixes.

[0110] It should be noted that since it is expected to achieve multi-category defect generation, relying solely on text guidance with weak control may not be able to accurately distinguish and generate defects of a specified category. Therefore, the introduced trainable embedding model can identify the defect category to a specific space to obtain category guidance encoding, and perform category control in a more explicit way.

[0111] The defect area sample image and the superimposed mask sample image are respectively input into the encoding network in the trained image generation model for processing, and the appearance sample feature map corresponding to the defect area sample image and the mask sample feature map corresponding to the superimposed mask sample image output by the encoding network are obtained. The appearance sample feature map is processed by noise superposition to obtain a sample noise coding feature map; the defect mask sample image is processed by downsampling to obtain a sample sampling coding feature map; the sample noise coding feature map, the sample sampling coding feature map and the mask sample feature map are processed by feature map splicing to obtain a fused sample coding feature map.

[0112] The fused sample feature map and the sample encoding feature are input into the multi-head attention network U-Net in the preset stable diffusion model for processing to obtain the predicted sample feature map output by the model. According to the fused sample feature map and the predicted sample feature map, the stable diffusion loss value is determined based on the preset loss function; the stable diffusion model is trained based on the stable diffusion loss value until the preset model training end condition is met to obtain a trained stable diffusion model. Among them, the loss function may include a conditional noise prediction loss function, a variational inference objective loss function or a self-supervised loss, etc.; the model training end condition may be that the stable diffusion loss value reaches a preset loss threshold or the stable diffusion loss value tends to be stable, etc.

[0113] It can be seen that in this embodiment, by combining the trained image generation model to fuse the features of the sample images of the defect area during the training of the stable diffusion model, and combining the sample encoding features to feature encode the defect category and defect text description of the sample images of the defect area, the training accuracy of the U-Net multi-head attention network of the stable diffusion model is improved, thereby improving the training accuracy of the stable diffusion model.

[0114] In a specific embodiment describing the stable diffusion model training process, as shown in FIG. Figure 5Schematic diagram of the model training process of a stable diffusion model shown in FIG. It should be noted that the image generation model, as a part of the stable diffusion model, i.e., the self-encoding and decoding network, participates in the training process of the multi-head attention U-Net of the stable diffusion model, and trains its own decoding network at the same time as the U-Net training. Specifically, the defect area sample image is input into the encoding network of the image generation model to obtain the appearance sample feature map output by the encoding network; and the superimposed mask sample image is input into the encoding network of the image generation model to obtain the mask sample feature map output by the encoding network. The appearance sample feature map is subjected to noise superposition operation to obtain the sample encoding feature map. The appearance sample feature map is input into the decoding network of the image generation model to obtain the predicted defect area image output by the decoding network; according to the predicted defect area image and the defect area sample image, the encoding network of the image generation network is trained to obtain the trained image generation model. The defect mask sample image is downsampled, and the downsampled result is subjected to feature map fusion or feature map splicing with the mask sample feature map and the sample noise encoding feature map to obtain the fused sample encoding feature map. The sample defect text description information is input into CILPTextModel for feature encoding, and the sample defect category index information is input into the Embedding model for feature encoding. The encoded result and the fused sample encoding feature map are input into the multi-head attention U-Net of the stable diffusion model to obtain the predicted sample feature map output by U-Net. Based on the predicted sample feature map and the fused sample feature map, the U-Net model is trained to obtain a stable diffusion model that has completed training.

[0115] In a specific embodiment describing the process of generating an industrial appearance defect image, as shown in FIG. Figure 6The schematic diagram of the generation process of an industrial appearance defect image is shown. The appearance defect area image is input into the encoding network of the trained image generation model to obtain the appearance encoding feature map output by the encoding network; and the defect superimposed mask image is input into the encoding network of the trained image generation model to obtain the mask encoding feature map output by the encoding network; the appearance encoding feature map is superimposed with noise to obtain the noise encoding feature map. The appearance defect mask image is downsampled, and the result obtained after downsampling is fused or spliced ​​with the mask encoding feature map and the noise encoding feature map to obtain a fused encoding feature map. The defect text description information of the appearance defect area image is feature encoded by CILPTextModel to obtain a defect text description vector; and the defect category index information is feature encoded by the Embedding model to obtain a defect category index vector. The defect text description vector, the defect category index vector and the fused encoding feature map are input into the multi-head attention U-Net of the stable diffusion model to obtain the diffusion feature map output by the model. The diffusion feature map is input into the decoding network of the trained image generation model for processing to obtain the target defect area image output by the decoding network. The target defect area image is fused with the industrial background area image to obtain the target industrial appearance defect image.

[0116] It should be understood that, although the steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0117] Based on the same inventive concept, the embodiment of the present application also provides a defect image generation device. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more defect image generation device embodiments provided below can refer to the limitations of the defect image generation method above, and will not be repeated here.

[0118] like Figure 7 As shown, the embodiment of the present application provides a defect image generating device 700, comprising:

[0119] An extraction module 702 is used to extract defect areas from the acquired industrial appearance defect image to be generated, and obtain an appearance defect area image and an industrial background area image;

[0120] A generating module 704, used to generate an appearance defect mask image and a defect superposition mask image corresponding to the appearance defect region image respectively;

[0121] The processing module 706 is used to input the appearance defect area image and the defect superimposed mask image into the encoding network in the trained image generation model for processing, and output the appearance encoding feature map corresponding to the appearance defect area image and the mask encoding feature map corresponding to the defect superimposed mask image;

[0122] A determination module 708, configured to determine a diffusion feature map according to the appearance coding feature map, the mask coding feature map and the appearance defect mask image;

[0123] The generation module 704 is also used to input the diffusion feature map into the decoding network in the image generation model for processing, and output the target defect area image;

[0124] The fusion module 710 is used to fuse the target defect area image corresponding to the appearance defect area image with the industrial background area image to obtain a target industrial appearance defect image.

[0125] In some embodiments, in determining the diffusion feature map according to the appearance coding feature map, the mask coding feature map and the appearance defect mask image, the determination module 708 is specifically used to:

[0126] Determine a fused coding feature map according to the appearance coding feature map, the mask coding feature map and the appearance defect mask image;

[0127] Determining conditional coding features of the appearance defect region image;

[0128] The fused coding feature map and the conditional coding features are input into the trained stable diffusion model for processing, and the diffusion feature map is output.

[0129] In some embodiments, in determining the fused coding feature map according to the appearance coding feature map, the mask coding feature map and the appearance defect mask image, the determination module 708 is specifically used to:

[0130] Down-sampling the appearance defect mask image to obtain a sampling coding feature map;

[0131] Performing noise superposition processing on the appearance coding feature map to obtain a noise coding feature map;

[0132] The sampling coding feature map, the noise coding feature map and the mask coding feature map are fused to obtain a fused coding feature map.

[0133] In some embodiments, in determining the conditional coding features of the appearance defect area image, the determination module 708 is specifically used to:

[0134] Determine defect text description information and defect category index information according to the defect category of the appearance defect area image;

[0135] Generate a defect text description vector corresponding to the defect text description information and a defect category index vector corresponding to the defect category index information respectively;

[0136] Generate conditional encoding features including defect text description vector and defect category index vector.

[0137] In some embodiments, the defect image generating device 700 further includes:

[0138] A sample image acquisition module is used to extract defect areas from the acquired industrial appearance defect sample images to obtain defect area sample images;

[0139] The sample feature extraction module is used to input the sample image of the defect area into the encoding network in the preset image generation model for processing, and output the defect sample feature map; the weight parameter of the encoding network is the initial weight parameter;

[0140] The sample feature encoding module is used to input the defect sample feature map into the decoding network in the image generation model for processing and output the predicted defect area image;

[0141] The generation model training module is used to train the decoding network of the image generation model according to the predicted defect area image and the defect area sample image until the preset model training end condition is met to obtain the trained image generation model.

[0142] In some embodiments, in extracting defect areas from acquired industrial appearance defect sample images to obtain defect area sample images, the sample image acquisition module is specifically used to:

[0143] Performing image segmentation on the defect area contour in the acquired industrial appearance defect sample image to obtain a defect contour image;

[0144] Determine the defect contour area of ​​the defect contour image; determine the target image size based on the defect contour area and a preset foreground-background ratio range interval;

[0145] Based on the target image size, the defect area in the industrial appearance defect sample image is cropped to obtain a defect area sample image.

[0146] In some embodiments, the defect image generating device 700 further includes:

[0147] A sample defect extraction module is used to extract defect areas from the acquired industrial appearance defect sample image to obtain a defect area sample image;

[0148] A sample image generation module, used to generate a defect mask sample image and a superimposed mask sample image corresponding to the defect area sample image, and determine the sample coding features of the defect area sample image;

[0149] A sample encoding processing module is used to input the defect area sample image and the superimposed mask sample image into the encoding network in the trained image generation model for processing, and output the appearance sample feature map corresponding to the defect area sample image and the mask sample feature map corresponding to the superimposed mask sample image;

[0150] A sample feature fusion module is used to determine a fused sample feature map according to the appearance sample feature map, the mask sample feature map and the defect mask sample image;

[0151] The feature stable diffusion module is used to input the fused sample feature map and the sample encoding feature into a preset stable diffusion model for processing, and output a predicted sample feature map;

[0152] The diffusion model generation module is used to train the stable diffusion model according to the fusion sample feature map and the prediction sample feature map until the preset model training end condition is met to obtain the trained stable diffusion model.

[0153] Each module in the defect image generating device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0154] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to defect image generation. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in the above-mentioned defect image generation method are implemented.

[0155] In some embodiments, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, the steps in the above-mentioned defect image generation method are implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen; the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0156] Those skilled in the art will understand that Figure 8 or Fig. 9The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0157] In some embodiments, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0158] In some embodiments, Fig.10 The figure shows an internal structure diagram of a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0159] In some embodiments, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0160] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0161] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0162] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A defect image generation method, characterized in that: include: Extracting defect regions from the acquired industrial appearance defect image to be generated to obtain an appearance defect region image and an industrial background region image; Respectively generating an appearance defect mask image and a defect superposition mask image corresponding to the appearance defect region image; Inputting the appearance defect area image and the defect superimposed mask image into the encoding network in the trained image generation model for processing respectively, and outputting the appearance encoding feature map corresponding to the appearance defect area image and the mask encoding feature map corresponding to the defect superimposed mask image; Determine a diffusion feature map according to the appearance coding feature map, the mask coding feature map and the appearance defect mask image; Inputting the diffusion feature map into the decoding network in the image generation model for processing, and outputting the target defect area image; The target defect area image corresponding to the appearance defect area image is fused with the industrial background area image to obtain a target industrial appearance defect image.

2. The method according to claim 1, characterized in that The step of determining a diffusion feature map according to the appearance coding feature map, the mask coding feature map and the appearance defect mask image comprises: Determining a fused coding feature map according to the appearance coding feature map, the mask coding feature map and the appearance defect mask image; Determining conditional coding features of the appearance defect area image; The fused coding feature map and the conditional coding feature are input into a trained stable diffusion model for processing, and a diffusion feature map is output.

3. The method according to claim 2, characterized in that The step of determining a fused coding feature map according to the appearance coding feature map, the mask coding feature map and the appearance defect mask image comprises: Downsampling the appearance defect mask image to obtain a sampling coding feature map; Performing noise superposition processing on the appearance coding feature map to obtain a noise coding feature map; The sampling coding feature map, the noise coding feature map and the mask coding feature map are fused to obtain a fused coding feature map.

4. The method according to claim 2, characterized in that: The determining of the conditional coding features of the appearance defect area image comprises: Determining defect text description information and defect category index information according to the defect category of the appearance defect area image; respectively generating a defect text description vector corresponding to the defect text description information and a defect category index vector corresponding to the defect category index information; A conditional coding feature is generated including the defect text description vector and the defect category index vector.

5. The method according to claim 1, characterized in that Before respectively inputting the appearance defect region image and the defect superimposed mask image into the encoding network in the trained image generation model for processing and outputting the appearance coding feature map corresponding to the appearance defect region image and the mask coding feature map corresponding to the defect superimposed mask image, the method further includes: Extracting defect areas from the acquired industrial appearance defect sample images to obtain defect area sample images; Input the defect area sample image into the encoding network in the preset image generation model for processing, and output the defect sample feature map; the weight parameter of the encoding network is the initial weight parameter; Inputting the defect sample feature map into the decoding network in the image generation model for processing, and outputting a predicted defect area image; The decoding network of the image generation model is trained according to the predicted defect area image and the defect area sample image until a preset model training end condition is met, thereby obtaining the trained image generation model.

6. The method according to claim 5, characterized in that The step of extracting defect areas from the acquired industrial appearance defect sample images to obtain defect area sample images includes: Performing image segmentation on the defect area contour in the acquired industrial appearance defect sample image to obtain a defect contour image; Determine the defect contour area of ​​the defect contour image; determine the target image size according to the defect contour area and based on a preset foreground-background ratio range interval; Based on the target image size, the defect area in the industrial appearance defect sample image is cropped to obtain a defect area sample image.

7. The method according to claim 2, characterized in that Before inputting the fused coding feature map and the conditional coding feature into the trained stable diffusion model for processing and outputting the diffusion feature map, the method further includes: Extracting defect areas from the acquired industrial appearance defect sample images to obtain defect area sample images; Respectively generating a defect mask sample image and a superimposed mask sample image corresponding to the defect area sample image, and determining a sample coding feature of the defect area sample image; Inputting the defect area sample image and the superimposed mask sample image into the encoding network in the trained image generation model for processing respectively, and outputting the appearance sample feature map corresponding to the defect area sample image and the mask sample feature map corresponding to the superimposed mask sample image; Determine a fused sample feature map according to the appearance sample feature map, the mask sample feature map and the defect mask sample image; Inputting the fused sample feature map and the sample coding feature into a preset stable diffusion model for processing, and outputting a predicted sample feature map; The stable diffusion model is trained according to the fused sample feature map and the predicted sample feature map until a preset model training end condition is met, thereby obtaining the trained stable diffusion model.

8. A defect image generating device, characterized in that: include: An extraction module is used to extract defect areas from the acquired industrial appearance defect image to be generated, so as to obtain an appearance defect area image and an industrial background area image; A generating module, used to respectively generate an appearance defect mask image and a defect superposition mask image corresponding to the appearance defect region image; A processing module, used to input the appearance defect area image and the defect superimposed mask image into the encoding network in the trained image generation model for processing, and output an appearance coding feature map corresponding to the appearance defect area image and a mask coding feature map corresponding to the defect superimposed mask image; A determination module, configured to determine a diffusion feature map according to the appearance coding feature map, the mask coding feature map and the appearance defect mask image; The generation module is further used to input the diffusion feature map into the decoding network in the image generation model for processing, and output the target defect area image; The fusion module is used to fuse the target defect area image corresponding to the appearance defect area image with the industrial background area image to obtain a target industrial appearance defect image.

9. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images