Image sample generation method, device, equipment and storage medium

Abstract

The present invention discloses a method, apparatus, device, and storage medium for generating image samples. These methods overcome the difficulty of collecting machine vision image samples. Using this method, a large number of positive sample images can be generated using only a very small number of positive samples, and usable negative samples can be generated based on the various defect types learned from the dataset. Furthermore, the method overcomes the poor stability of existing image sample generation methods. The network and training method used in this method can accurately retain the characteristics of product samples, generating near-realistic sample images.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for generating machine vision image samples based on a controlled diffusion model. Background Art

[0002] Machine vision refers to technology used in industrial applications by capturing images of equipment and products for measurement, inspection, and identification. Deep learning, on the other hand, is an algorithmic technology that builds deep neural networks to learn the characteristics of samples in a dataset and use these characteristics to complete various downstream tasks. Due to its wide applicability, deep learning is widely used in machine vision inspection and identification tasks.

[0003] Machine vision technology using deep learning methods is widely used in industrial product identification and inspection. In the specialized field of machine vision-based anomaly detection, simply collecting a sufficient number of positive samples (normal samples) and negative samples (abnormal samples, defect samples), selecting an appropriate deep learning network, and training it can produce a well-performing detection model.

[0004] Because deep learning methods rely on collecting large amounts of data, their effectiveness is significantly reduced when the data volume is small or of poor quality. In industrial scenarios like machine vision anomaly detection, the small batch size of equipment and products often prevents the collection of a large number of normal samples. Furthermore, collecting a large number of defective samples is impossible due to the potential for damaging the product or the difficulty of manual interference and destruction of the equipment. This makes deep learning-based machine vision methods difficult to apply in these scenarios.

[0005] There are two main approaches to solving these problems. One involves using image editing techniques to rotate, mirror, adjust color, and add noise to existing samples to generate new positive samples. Alternatively, new negative samples can be generated by artificially modifying the texture of samples or embedding them with other defective images. The other approach involves using neural network-based image generation techniques, such as generative adversarial networks (GANs), to transfer models trained on general image datasets to industrial scenarios. By inputting existing sample images or text descriptions, similar image data can be generated.

[0006] Of these two methods, image editing technology has a limited operating space and cannot generate a large number of samples; the generated samples are highly similar to existing samples, which affects the accuracy of subsequent deep learning training; in addition, it relies on manual operation, which is inefficient and time-consuming.

[0007] However, image generation technology is difficult to migrate. Images generated using the migrated model are likely to have the characteristics of the original dataset. The migrated model requires a lot of parameter adjustment practice, and the generated images are uncontrollable. It is easy to have images with mixed features, making it difficult to obtain high-quality, close-to-real usable images. Summary of the Invention

[0008] In view of the above problems, the present invention provides an image sample generation method, apparatus, device, and storage medium for overcoming or at least partially resolving the above problems. By collecting a large amount of multi-category industrial image data to form a universal industrial image dataset, constructing a stable diffusion model for generating positive samples, and building an improved control network for generating negative samples, a universal industrial image data generation network is obtained, enabling the generation of a usable industrial machine vision anomaly detection dataset from very few positive samples.

[0009] The present invention provides the following solutions:

[0010] A method for generating an image sample, comprising:

[0011] Acquire a plurality of industrial images and create a pre-training data set; the pre-training data set includes an original image sample of each industrial image, an image description text of each industrial image, and a sample defect mask image corresponding to each industrial image;

[0012] Constructing an image generation network based on a stable diffusion model; the stable diffusion model includes an image encoder and an image decoder;

[0013] Combined with the original image samples and image description text in the pre-training dataset, the image encoder and the image decoder are trained using a low-rank adaptation method to obtain a positive sample generation model;

[0014] Constructing a controlled image generation network, the controlled image generation network comprising the image generation network and a control network combined with the image generation network; the control network is used to introduce a defect mask image and a conditional similarity parameter to control the decoding process of the image decoder;

[0015] The original image samples in the pre-training dataset are used as picture input, the image description text is used as text input, and the defect mask image is used as conditional input, and the image encoder and the image decoder are trained using the low-rank adaptation method to obtain a negative sample generation model;

[0016] Acquire a sample image and determine a sample descriptor for the sample image, wherein the sample image is a normal product image; input the sample image and the sample descriptor into the positive sample generation model to obtain a positive sample image;

[0017] Determine a defect description word and a defect mask image of the positive sample image; input the positive sample image, the defect description word and the defect mask image into the negative sample generation model to obtain a negative sample image.

[0018] Preferably, the industrial images are constructed by collecting various types of industrial product images and / or using the Mvtec-AD dataset.

[0019] Preferably, the image encoder comprises an encoding diffusion block for encoding an input image into an 8×8 intermediate diffusion block; the image decoder comprises a decoding diffusion block for re-encoding the intermediate diffusion block into an image.

[0020] Preferably: the stable diffusion model also includes a text encoder and an image information creator; the text encoder is used to encode the text input of the image generation network using the text encoder part in the language image contrast model; the image information creator is composed of several Unet encoder-decoder structures connected together.

[0021] Preferably, the value of the conditional similarity parameter is determined according to the degree of control required by the conditional input.

[0022] Preferably: the sample image and the sample description word are input into the positive sample generation model, and the network parameters are set to obtain the positive sample image;

[0023] Inputting the positive sample image, the defect description word, and the defect mask image into the negative sample generation model, and setting the network parameters to obtain a negative sample image;

[0024] The network parameters include the number of iterations, random seeds, sampling methods, Figure 1 Consistency.

[0025] Preferably, the positive sample defect mask image is determined by manual drawing or random generation.

[0026] An image sample generating device, comprising:

[0027] A pre-training data set acquisition unit is used to acquire a plurality of industrial images and generate a pre-training data set; the pre-training data set includes an original image sample of each industrial image, an image description text of each industrial image, and a sample defect mask image corresponding to each industrial image;

[0028] An image generation network construction unit, configured to construct an image generation network based on a stable diffusion model; the stable diffusion model includes an image encoder and an image decoder;

[0029] A positive sample generation model acquisition unit is used to train the image encoder and the image decoder using a low-rank adaptation method in combination with the original image samples and image description text in the pre-training data set to obtain a positive sample generation model;

[0030] a controlled image generation network construction unit, configured to construct a controlled image generation network, the controlled image generation network comprising the image generation network and a control network coupled to the image generation network; the control network being configured to introduce a defect mask and conditional similarity parameters to control the decoding process of the image decoder;

[0031] a negative sample generation model acquisition unit, configured to use the original image samples in the pre-training dataset as picture input, the image description text as text input, and the defect mask image as conditional input, and train the image encoder and the image decoder using the low-rank adaptation method to obtain a negative sample generation model;

[0032] A positive sample image generation unit is configured to obtain a sample image and determine a sample descriptor for the sample image, wherein the sample image is a normal product image; and input the sample image and the sample descriptor into the positive sample generation model to obtain a positive sample image;

[0033] The negative sample image generation unit is used to determine the defect descriptor and defect mask image of the positive sample image; input the positive sample image, the defect descriptor and the defect mask image into the negative sample generation model to obtain a negative sample image.

[0034] An image sample generating device, comprising a processor and a memory:

[0035] The memory is used to store program code and transmit the program code to the processor;

[0036] The processor is configured to execute the above-mentioned image sample generation method according to the instructions in the program code.

[0037] A computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store program codes, and the program codes are used to execute the above-mentioned image sample generation method.

[0038] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0039] The embodiments of the present application provide an image sample generation method, apparatus, device, and storage medium that overcome the difficulty of collecting machine vision image samples. Using this method, a large number of positive sample images can be generated by collecting only a very small number of positive samples, and usable negative samples can be generated based on the various defect types learned in the data set.

[0040] At the same time, the problem of poor stability of existing image sample generation methods is overcome. The network and training method used in this method can accurately retain the characteristics of product samples and generate nearly real sample images.

[0041] In addition, the uncontrollable problem of existing image sample generation methods is overcome. The improved controlled image generation network constructed by this method can well receive and process mask inputs, so that the generated negative samples have user-controlled morphology and features.

[0042] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort.

[0044] Figure 1 is a flow chart of a method for generating an image sample provided by an embodiment of the present invention;

[0045] Figure 2 This is a step framework diagram of a method for generating an image sample provided by an embodiment of the present invention;

[0046] Figure 3 is a diagram of the structure of a training data set provided by an embodiment of the present invention;

[0047] Figure 4 Schematic diagram of a stable diffusion network structure provided by an embodiment of the present invention;

[0048] Figure 5 is a schematic diagram of a low-rank adaptive training method provided by an embodiment of the present invention;

[0049] Figure 6 Schematic diagram of the steps for generating positive samples provided by an embodiment of the present invention;

[0050] Figure 7 is a schematic diagram of a controlled image generation network provided by an embodiment of the present invention;

[0051] Figure 8 This is a schematic diagram of the coupling between the control decoder and the generated network image decoder provided by an embodiment of the present invention;

[0052] Figure 9 Schematic diagram of the steps for generating negative samples provided by an embodiment of the present invention;

[0053] Figure 10 is a schematic diagram of an image sample generating device provided by an embodiment of the present invention;

[0054] Figure 11 Schematic diagram of an image sample generating device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of the present invention.

[0056] See also Figure 1 , is a method for generating an image sample provided by an embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0057] S101: Acquire several industrial images and prepare a pre-training dataset; the pre-training dataset includes an original image sample of each industrial image, an image description text of each industrial image, and a sample defect mask image corresponding to each industrial image; wherein, the industrial image can include as many industrial images of various types as possible, for example, the industrial image is constructed using collected images of various types of industrial products and / or uses the Mvtec-AD dataset. The Mvtec-AD dataset, a general industrial image dataset, typically contains 5354 high-resolution color images of different targets and texture types. It contains normal (i.e., does not contain defects) images for training, and abnormal images for testing. There are 70 different types of abnormal defects, such as scratches, dents, contamination, and different structural changes.

[0058] S102: Construct an image generation network based on a stable diffusion model; the stable diffusion model includes an image encoder and an image decoder; specifically, the image encoder includes a coding diffusion block, which is used to encode the input image into an 8×8 intermediate diffusion block; the image decoder includes a decoding diffusion block, which is used to re-encode the intermediate diffusion block into an image.

[0059] Furthermore, the stable diffusion model also includes a text encoder and an image information creator; the text encoder is used to encode the text input of the image generation network using the text encoder part in the language image contrast model; the image information creator is composed of several Unet encoder-decoder structures connected together.

[0060] S103: combining the original image samples and image description text in the pre-training dataset with a low-rank adaptation method to train the image encoder and the image decoder to obtain a positive sample generation model;

[0061] S104: Constructing a controlled image generation network, comprising the image generation network and a control network coupled thereto. The control network is configured to introduce a defect mask image and a conditional similarity parameter to control the decoding process of the image decoder. Specifically, the value of the conditional similarity parameter is determined based on the degree of control required by the conditional input. The control network primarily introduces the defect mask image, and the addition of the similarity parameter allows the generated image to controllably maintain certain original image features.

[0062] S105: using the original image sample in the pre-training dataset as the picture input, the image description text as the text input, and the defect mask image as the conditional input, and using the low-rank adaptation method to train the image encoder and the image decoder to obtain a negative sample generation model;

[0063] S106: Acquire a sample image and determine a sample descriptor for the sample image, wherein the sample image is a normal product image; input the sample image and the sample descriptor into the positive sample generation model to obtain a positive sample image;

[0064] S107: Determine a defect description word and a defect mask image of the positive sample image; input the positive sample image, the defect description word and the defect mask image into the negative sample generation model to obtain a negative sample image.

[0065] In a specific implementation, the sample image and the sample description word are input into the positive sample generation model, and the network parameters are set to obtain the positive sample image;

[0066] Inputting the positive sample image, the defect description word, and the defect mask image into the negative sample generation model, and setting the network parameters to obtain a negative sample image;

[0067] The network parameters include the number of iterations, random seeds, sampling methods, Figure 1 Consistency.

[0068] The image sample generation method provided in the embodiments of the present application collects a large amount of multi-category industrial image data to form a general industrial image dataset, constructs a stable diffusion model for generating positive samples, and constructs an improved control network for generating negative samples, thereby obtaining a general industrial image data generation network, and realizing the generation of a usable industrial machine vision anomaly detection dataset from very few positive samples.

[0069] A method that uses a low-rank adaptation method to migrate the image generation network model to an industrial image dataset for training greatly improves the stability and controllability of the image generation network, enabling the image generation network to generate positive sample images consistent with the target product characteristics.

[0070] The improved control network is combined with the stable diffusion model. This method adds a conditional similarity parameter to the control network, allowing the control network to obtain negative sample images with different defect levels, greatly expanding the use of mask images.

[0071] This method uses an improved controlled image generation network to generate negative product images. This method is highly stable, controllable, and flexible, ensuring accurate generation of desired negative product images of various types while preserving sample characteristics.

[0072] The following is a detailed introduction to the image sample generation method provided in the embodiments of the present application.

[0073] This method creates an industrial product image dataset, builds an image generation network based on the Stable Diffusion model, and uses the Low-Rank Adaption method to train a positive sample generation model from the constructed industrial image dataset. This model can be used to generate positive samples; it also builds a generation control network (negative sample generation model) based on the improved control network (ControlNet), and retrains the positive sample generation model into a controlled generation model, which can be used to generate negative samples. The specific steps are as follows: Figure 2 shown.

[0074] The specific implementation steps are as follows:

[0075] (1) Collect and select industrial image data and make it into a pre-training dataset. The image data should contain as many industrial images of various types as possible, which can be defective or normal samples. The dataset can be constructed using a large number of product images collected by users. If it is not easy for users to collect product images, or if there are fewer product images collected, the Mvtec-AD dataset can also be used. The Mvtec-AD dataset can be used alone or in combination with the collected images. After that, for each image, retain an original image sample, describe and save a piece of image description text, and make and save a sample defect mask image. The final dataset has the following characteristics: Figure 3 The structure shown.

[0076] (2) Construct an image generation network based on the stable diffusion model. The stable diffusion model consists of a text encoder, an image information creator, an image encoder, and an image decoder. Its structure is as follows: Figure 4 shown.

[0077] like Figure 4 As shown in the figure, the text encoder uses the text transformer component from the Contrastive Language-Image Pretraining (CLIP) framework. The image information generator group consists of N connected Unet encoder-decoder structures, with N set to a maximum of 50 and adjustable in subsequent use. The image encoder consists of an encoding diffusion block that encodes the input image into an 8×8 intermediate diffusion block. The image decoding block, consisting of a decoding diffusion block, re-encodes the intermediate diffusion block into an image.

[0078] (3) Use the low-rank adaptation method to train the image generation network. The dataset is the original sample and description text part of the dataset obtained in step (1). The low-rank adaptation method is as follows Figure 5 shown.

[0079] The low-rank adaptation method acts on the dense layer, keeping the pre-trained weights frozen, and instead trains the rank decomposition matrix parameters shown in Figures A and B. This allows the training model to retain the frozen weights while obtaining an efficient training matrix for fine-tuning the model to adapt to the new dataset.

[0080] In this method, the image encoder and image decoder parts of the stable diffusion model shown in step (2) are trained using a low-rank adaptation method. The pre-trained model of the stable diffusion model is transferred to the industrial image pre-training dataset obtained in (1) to train a positive sample generation model adapted to industrial images.

[0081] (4) Use the positive sample generation model to generate positive samples of the target product. Figure 6 As shown in the figure, firstly, an image of a normal product is collected as a sample, and the sample image is input into the positive sample generation model, and a specific description word, such as "brass color, gear", is input; then the network parameters are set, such as the number of iterations, random seed, sampling method, original Figure 1 After setting the parameters, run the model to generate positive sample images.

[0082] (5) Construct a controlled image generation network based on the improved control network. Add the control network to the image generation network, and the structure of the combined controlled image generation network is as follows: Figure 7 As shown in Figure 2, new conditional inputs are introduced to control the image encoding and decoding process in the image generation network.

[0083] Among them, such as Figure 8The figure shows a schematic diagram of the conditional encoder structure of the controlled network, a schematic diagram of the structure of the basic block of the control network, and a schematic diagram of the modified image decoder structure. The parameter M is added to the improved image decoder as a conditional similarity parameter. The control network decoder structure is used before the Mth step in the network, and the generation network image decoder structure is used after the Mth step. The value of M can be modified when the model is used to control the degree to which the generated image is controlled by the conditional input.

[0084] (6) Use the improved controlled image generation network to train the negative sample generation model. Use the dataset obtained in step (1), take the original sample in the dataset as the image input, the description text as the text input, and the mask image as the conditional input, and use the same low-rank adaptation method as step (3) to train the improved controlled image generation network to obtain the negative sample generation model.

[0085] (7) Use the negative sample generation model to generate negative samples of the target product. Figure 9 As shown, similar to step (4), use the positive sample image in step (4) as the input image, add descriptive words, and manually draw or randomly generate a defect mask image as the conditional input. After setting the parameters, you can run it to get the output negative sample image.

[0086] In summary, the image sample generation method provided in this application overcomes the problem of difficulty in collecting machine vision image samples. Using this method, only a very small number of positive samples can be collected to generate a large number of positive sample images, and usable negative samples can be generated based on the various defect types learned in the data set.

[0087] At the same time, the problem of poor stability of existing image sample generation methods is overcome. The network and training method used in this method can accurately retain the characteristics of product samples and generate nearly real sample images.

[0088] In addition, the uncontrollable problem of existing image sample generation methods is overcome. The improved controlled image generation network constructed by this method can well receive and process mask inputs, so that the generated negative samples have user-controlled morphology and features.

[0089] See also Figure 10 , the embodiment of the present application may also provide an image sample generating device, such as Figure 10 As shown, the device may include:

[0090] The pre-training data set acquisition unit 1001 is used to acquire a plurality of industrial images and generate a pre-training data set; the pre-training data set includes an original image sample of each industrial image, an image description text of each industrial image, and a sample defect mask image corresponding to each industrial image;

[0091] An image generation network construction unit 1002 is configured to construct an image generation network based on a stable diffusion model; the stable diffusion model includes an image encoder and an image decoder;

[0092] A positive sample generation model acquisition unit 1003 is configured to train the image encoder and the image decoder using a low-rank adaptation method in combination with the original image samples and image description text in the pre-training dataset to obtain a positive sample generation model;

[0093] A controlled image generation network construction unit 1004 is configured to construct a controlled image generation network, wherein the controlled image generation network includes the image generation network and a control network coupled with the image generation network; the control network is configured to introduce a defect mask and conditional similarity parameters to control the decoding process of the image decoder;

[0094] A negative sample generation model acquisition unit 1005 is configured to use the original image samples in the pre-training dataset as picture input, the image description text as text input, and the defect mask image as conditional input, and train the image encoder and the image decoder using the low-rank adaptation method to obtain a negative sample generation model;

[0095] The positive sample image generation unit 1006 is configured to obtain a sample image and determine a sample descriptor for the sample image, wherein the sample image is a normal product image; and input the sample image and the sample descriptor into the positive sample generation model to obtain a positive sample image.

[0096] The negative sample image generation unit 1007 is used to determine the defect description word and the defect mask image of the positive sample image; input the positive sample image, the defect description word and the defect mask image into the negative sample generation model to obtain a negative sample image.

[0097] like Figure 11 As shown, an image sample generating device provided by an embodiment of the present application may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, the memory 11, and the communication interface 12 communicate with each other via the communication bus 13.

[0098] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field programmable gate array, or other programmable logic devices.

[0099] The processor 10 may call a program stored in the memory 11 . Specifically, the processor 10 may execute the operations in the embodiment of the image sample generating method.

[0100] The memory 11 is used to store one or more programs. The program may include program code, and the program code includes computer operating instructions. In the embodiment of the present application, the memory 11 stores at least a program for implementing the following functions:

[0101] Acquire a plurality of industrial images and create a pre-training data set; the pre-training data set includes an original image sample of each industrial image, an image description text of each industrial image, and a sample defect mask image corresponding to each industrial image;

[0102] Constructing an image generation network based on a stable diffusion model; the stable diffusion model includes an image encoder and an image decoder;

[0103] Combined with the original image samples and image description text in the pre-training dataset, the image encoder and the image decoder are trained using a low-rank adaptation method to obtain a positive sample generation model;

[0104] Constructing a controlled image generation network, the controlled image generation network comprising the image generation network and a control network combined with the image generation network; the control network is used to introduce a defect mask and a conditional similarity parameter to control the decoding process of the image decoder;

[0105] The original image samples in the pre-training dataset are used as picture input, the image description text is used as text input, and the defect mask image is used as conditional input, and the image encoder and the image decoder are trained using the low-rank adaptation method to obtain a negative sample generation model;

[0106] Acquire a sample image and determine a sample descriptor for the sample image, wherein the sample image is a normal product image; input the sample image and the sample descriptor into the positive sample generation model to obtain a positive sample image;

[0107] Determine a defect description word and a defect mask image of the positive sample image; input the positive sample image, the defect description word and the defect mask image into the negative sample generation model to obtain a negative sample image.

[0108] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required for at least one function (such as a file creation function, a data reading and writing function), etc.; the data storage area can store data created during use, such as initialization data, etc.

[0109] In addition, the memory 11 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0110] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0111] Of course, it needs to be explained that Figure 11 The structure shown does not constitute a limitation on the image sample generating device in the embodiment of the present application. In actual applications, the image sample generating device may include Figure 11 More or fewer components than shown, or combinations of certain components.

[0112] An embodiment of the present application may further provide a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the steps of the above-mentioned image sample generation method.

[0113] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0114] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present application.

[0115] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0116] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A method for generating an image sample, characterized in that: include: Acquire a plurality of industrial images and create a pre-training data set; the pre-training data set includes an original image sample of each industrial image, an image description text of each industrial image, and a sample defect mask image corresponding to each industrial image; Construct an image generation network based on a stable diffusion model; The stable diffusion model includes an image encoder and an image decoder; Combined with the original image samples and image description text in the pre-training dataset, the image encoder and the image decoder are trained using a low-rank adaptation method to obtain a positive sample generation model; Constructing a controlled image generation network, the controlled image generation network comprising the image generation network and a control network combined with the image generation network; the control network is used to introduce a defect mask image and a conditional similarity parameter to control the decoding process of the image decoder; The original image samples in the pre-training dataset are used as picture input, the image description text is used as text input, and the defect mask image is used as conditional input, and the image encoder and the image decoder are trained using the low-rank adaptation method to obtain a negative sample generation model; Acquire a sample image and determine a sample descriptor for the sample image, wherein the sample image is a normal product image; input the sample image and the sample descriptor into the positive sample generation model to obtain a positive sample image; Determining a defect description word and a defect mask image of the positive sample image; The positive sample image, the defect description word, and the defect mask image are input into the negative sample generation model to obtain a negative sample image.

2. The image sample generation method according to claim 1, characterized in that: The industrial images are constructed by collecting various types of industrial product images and / or using the Mvtec-AD dataset.

3. The image sample generation method according to claim 1, characterized in that: The image encoder includes an encoding diffusion block for encoding an input image into an 8×8 intermediate diffusion block; the image decoder includes a decoding diffusion block for re-encoding the intermediate diffusion block into an image.

4. The image sample generation method according to claim 1, wherein: The stable diffusion model also includes a text encoder and an image information creator; the text encoder is used to encode the text input of the image generation network using the text encoder part in the language image contrast model; the image information creator is composed of several Unet encoder-decoder structures connected together.

5. The image sample generation method according to claim 1, characterized in that: The value of the conditional similarity parameter is determined according to the degree of control required by the conditional input.

6. The image sample generation method according to claim 1, characterized in that: Inputting the sample image and the sample description word into the positive sample generation model, and setting the network parameters to obtain the positive sample image; Inputting the positive sample image, the defect description word, and the defect mask image into the negative sample generation model, and setting the network parameters to obtain a negative sample image; The network parameters include the number of iteration steps, random seeds, sampling methods, and original image consistency.

7. The image sample generation method according to claim 1, characterized in that: The positive sample defect mask image is determined by manually drawing or randomly generating.

8. An image sample generating device, characterized in that: include: A pre-training data set acquisition unit is used to acquire a plurality of industrial images and generate a pre-training data set; the pre-training data set includes an original image sample of each industrial image, an image description text of each industrial image, and a sample defect mask image corresponding to each industrial image; Image generation network construction unit, used to build an image generation network based on a stable diffusion model; The stable diffusion model includes an image encoder and an image decoder; A positive sample generation model acquisition unit is used to train the image encoder and the image decoder using a low-rank adaptation method in combination with the original image samples and image description text in the pre-training data set to obtain a positive sample generation model; a controlled image generation network construction unit, configured to construct a controlled image generation network, the controlled image generation network comprising the image generation network and a control network coupled to the image generation network; the control network being configured to introduce a defect mask and conditional similarity parameters to control the decoding process of the image decoder; a negative sample generation model acquisition unit, configured to use the original image samples in the pre-training dataset as picture input, the image description text as text input, and the defect mask image as conditional input, and train the image encoder and the image decoder using the low-rank adaptation method to obtain a negative sample generation model; A positive sample image generation unit is configured to obtain a sample image and determine a sample descriptor for the sample image, wherein the sample image is a normal product image; and input the sample image and the sample descriptor into the positive sample generation model to obtain a positive sample image; a negative sample image generating unit, configured to determine a defect description word and a defect mask image of the positive sample image; The positive sample image, the defect description word, and the defect mask image are input into the negative sample generation model to obtain a negative sample image.

9. An image sample generating device, characterized in that: The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the image sample generating method according to any one of claims 1 to 7 according to instructions in the program code.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store program code, and the program code is used to execute the image sample generation method according to any one of claims 1 to 7.

Images