Cloth defect image generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a cloth defect image generation method and device, electronic equipment and a storage medium. The method comprises the steps that defect text information input by a user is determined; and inputting the defect text information into a preset integrated model to obtain a cloth defect image. Wherein the integrated model is generated by integrating a plurality of second models obtained after a plurality of first models obtained after unsupervised hybrid pre-training based on the multi-class defect data are finely adjusted. Through the embodiment of the invention, the data for training can be automatically generated, so that the data acquisition period can be shortened, and the data volume and the data surface can be increased. Besides, a de-noising diffusion probability model is utilized, multi-class defect data unsupervised hybrid pre-training is introduced, the model is guided to learn and understand data distribution of images in the field of cloth defects, and huge difference between data distribution of natural images and data distribution of cloth defect images is made up.

Description

Technical Field

[0001] The present invention relates to the technical field of cloth detection, and in particular to a method for generating a cloth defect image, a device for generating a cloth defect image, an electronic device and a storage medium. Background Art

[0002] With the rapid development of the textile industry, in order to help cloth production sites quickly identify production errors and increase output, downstream task algorithms based on deep learning neural networks are widely used in the quality inspection of cloth products. The training of these methods usually requires a large amount of data.

[0003] However, for mature production lines, there are problems such as long defect data collection cycle and insufficient data volume. The defect morphology of different cloth models varies greatly. Many downstream task algorithms have difficulty in compatibility with data of different cloth models. When changing the production line, it takes a long time to deploy the new algorithm.

[0004] In response to the above problems, if more similar data can be generated in a short period of time using a small amount of available samples, it can greatly alleviate the current business pain points in the field of cloth defect detection. Summary of the invention

[0005] In view of the above problems, a method for generating a cloth defect image, a device for generating a cloth defect image, an electronic device and a storage medium are proposed to overcome the above problems or at least partially solve the above problems, including:

[0006] A method for generating a cloth defect image, the method comprising:

[0007] Determine the defect text information entered by the user;

[0008] Inputting the defect text information into a preset integrated model to obtain the fabric defect image;

[0009] The integrated model is generated by integrating multiple second models obtained after fine-tuning multiple first models obtained after unsupervised mixed pre-training based on multi-category defect data.

[0010] Optionally, the method further comprises:

[0011] Displaying an interactive interface, wherein the interactive interface displays a text input box;

[0012] The step of determining the defect text information input by the user includes:

[0013] The text information input by the user in the text input box is received, and the defect text information is determined according to the text information input in the text input box.

[0014] Optionally, the method further comprises:

[0015] Generate a first data set and a second data set; wherein the first data set includes good product images and defective images; the second data set includes defective images, labels corresponding to the defective images, and category texts corresponding to the defective images;

[0016] According to the first data set, unsupervised mixed pre-training of multi-category defect data is performed to obtain multiple first models;

[0017] The multiple first models are trained according to the second data set to obtain the second model.

[0018] Optionally, the unsupervised mixed pre-training of multi-category defect data is performed according to the first data set to obtain multiple first models, including:

[0019] According to the first data set, unsupervised mixed pre-training of multi-category defect data is performed on the preset denoising diffusion probability model based on the latent space to obtain multiple first models.

[0020] Optionally, training the plurality of first models according to the second data set to obtain the second model includes:

[0021] Adding a text encoder and a cross self-attention layer to the plurality of first models;

[0022] According to the second data set, the multiple first models after adding text encoders and cross self-attention layers are trained to obtain the second model.

[0023] Optionally, the method further comprises:

[0024] A weight is set for each category text; the weight is inversely proportional to the frequency corresponding to the category text.

[0025] Optionally, the method further comprises:

[0026] The multiple second models are integrated to obtain the integrated model.

[0027] An embodiment of the present invention further provides a device for generating a cloth defect image, the device comprising:

[0028] A determination module, used to determine the defect text information input by the user;

[0029] A generation module is used to input the defect text information into a preset integrated model to obtain the fabric defect image; wherein the integrated model is generated by integrating multiple second models obtained after fine-tuning multiple first models obtained after unsupervised mixed pre-training based on multi-category defect data.

[0030] Optionally, the device further comprises:

[0031] A display module, used to display an interactive interface, wherein the interactive interface displays a text input box;

[0032] The determination module is used to receive the text information input by the user in the text input box, and determine the defect text information according to the text information input in the text input box.

[0033] Optionally, the device further comprises:

[0034] A training module is used to generate a first data set and a second data set; wherein the first data set includes images of good products and images of defects; the second data set includes defect images, labels corresponding to the defect images, and category texts corresponding to the defect images; based on the first data set, unsupervised mixed pre-training of multi-category defect data is performed to obtain multiple first models; based on the second data set, the multiple first models are trained to obtain the second model.

[0035] Optionally, the training module is used to perform unsupervised mixed pre-training of multi-category defect data on a preset latent space-based denoising diffusion probability model according to the first data set to obtain a plurality of first models.

[0036] Optionally, the training module is used to add a text encoder and a cross-self-attention layer to the multiple first models; according to the second data set, the multiple first models after adding the text encoder and the cross-self-attention layer are trained to obtain the second model.

[0037] Optionally, the training module is further used to set a weight for each category text; the weight is inversely proportional to the frequency corresponding to the category text.

[0038] Optionally, the training module is also used to integrate the multiple second models to obtain the integrated model.

[0039] An embodiment of the present invention further provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the above method for generating a cloth defect image when executed by the processor.

[0040] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for generating a cloth defect image as described above is implemented.

[0041] The embodiments of the present invention have the following advantages:

[0042] In an embodiment of the present invention, the defect text information input by the user is determined; the defect text information is input into a preset integrated model to obtain a fabric defect image. The integrated model is generated by integrating multiple second models obtained by fine-tuning multiple first models obtained after unsupervised hybrid pre-training based on multi-category defect data.

[0043] Through the embodiments of the present invention, data for training can be automatically generated, thereby reducing the data collection cycle and enriching the data volume and data surface.

[0044] In addition, the denoising diffusion probability model is used and unsupervised mixed pre-training of multi-category defect data is introduced to guide the model to learn and understand the data distribution of images in the field of cloth defects, thereby making up for the huge difference between the data distribution of natural images and cloth defect images. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0046] Figure 1 is a flowchart of a method for generating a cloth defect image according to an embodiment of the present invention;

[0047] Figure 2 is a flowchart of another method for generating a cloth defect image according to an embodiment of the present invention;

[0048] Figure 3 is a schematic diagram of a user interface according to an embodiment of the present invention;

[0049] Figure 4 is a flowchart of pre-training steps in an embodiment of the present invention;

[0050] Figure 5 is a schematic diagram of a latent space basis model D according to an embodiment of the present invention;

[0051] Figure 6 is a flow chart of the steps of fine-tuning of an embodiment of the present invention;

[0052] Figure 7 is a partial step flow chart of a fine-tuning of an embodiment of the present invention;

[0053] Figure 8 is a schematic diagram of another latent space basis model D according to an embodiment of the present invention;

[0054] Fig. 9is a schematic diagram of an integrated model of an embodiment of the present invention;

[0055] Fig.10 is a schematic diagram of a defect image and a corresponding label according to an embodiment of the present invention;

[0056] Fig.11 is an overall step flow chart of an embodiment of the present invention;

[0057] Fig.12 It is a structural schematic diagram of a device for generating a cloth defect image according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] In the field of industrial vision, the latest popular methods for generating image data are mainly based on adversarial generative networks and denoising diffusion probability models.

[0060] Methods based on adversarial generative networks usually consist of two networks: a generator and a discriminator. Such models can usually generate more realistic data, but the adversarial training process is often not stable enough and is prone to problems such as gradient disappearance or gradient explosion, as well as mode collapse, that is, the generator always generates the same or similar image samples.

[0061] The method based on the denoising diffusion probability model is based on the molecular diffusion properties in physics, statistical theory, and data distribution probability, and includes a forward denoising process and a reverse denoising process. This method has high model interpretability and is relatively stable during training. However, a large number of studies on this method are based on the field of natural images. Since the data distribution of natural images is very different from that of cloth defect images, directly applying the existing denoising diffusion probability model to the task of generating cloth defect images will cause the model to generate a large number of images with low industrial scene authenticity, poor light and shadow texture effects, and wrong defect types, which does not meet the needs of actual on-site conditions. In addition, some types of cloth defects have rare data, low frequency of occurrence, or the defect features are difficult to learn and have low distinction from background textures, making it difficult to obtain better image generation results through unified training.

[0062] Therefore, this application designs a multimodal image generation method and system for cloth defects. Using a denoising diffusion probability model, unsupervised hybrid pre-training of multi-category defect data is introduced to guide the model to learn and understand the data distribution of images in the field of cloth defects, thus making up for the huge difference between the data distribution of natural images and cloth defect images.

[0063] For details, please refer to Figure 1 , shows a flowchart of the steps of a method for generating a cloth defect image according to an embodiment of the present invention.

[0064] like Figure 1 As shown, the method for generating the cloth defect image may include the following steps:

[0065] Step 101: Determine the defect text information input by the user.

[0066] In actual applications, users can first input defect text information into the fabric defect image generation system.

[0067] Step 102: Input the defect text information into a preset integrated model to obtain a fabric defect image. The integrated model is generated by integrating multiple second models obtained by fine-tuning multiple first models obtained after unsupervised hybrid pre-training based on multi-category defect data.

[0068] Then, the fabric defect image generation system can input the defect text information into a preset integrated model; the preset integrated model can automatically reconstruct the image based on the defect text information and the random fabric image, thereby generating a fabric defect image. Compared with collecting training data, the embodiment of the present invention automatically generates training data, thereby reducing the data collection cycle and enriching the data volume and data surface.

[0069] In addition, the denoising diffusion probability model is used and unsupervised mixed pre-training of multi-category defect data is introduced to guide the model to learn and understand the data distribution of images in the field of cloth defects, thereby making up for the huge difference between the data distribution of natural images and cloth defect images.

[0070] Exemplarily, the defect text information can be any one of 0 encoder abnormality, 1 rip, 2 edge fall, 3 yarn breakage, 4 flying flowers, 5 horizontal stripes, 6 missed stitches, 7 holes, 8 pilling, 9 weak dirt, 10 yarn knots, 11 handwriting, 12 vertical bars, 13 thread ends, 14 dirt, 15 discrete dirt, 16 creases, and 17 wrinkles. It can also be the category text of other set defects, which is not limited to the embodiments of the present invention.

[0071] In an embodiment of the present invention, the defect text information input by the user is determined; the defect text information is input into a preset integrated model to obtain a fabric defect image. The integrated model is generated by integrating multiple second models obtained by fine-tuning multiple first models obtained after unsupervised hybrid pre-training based on multi-category defect data.

[0072] Through the embodiments of the present invention, data for training can be automatically generated, thereby reducing the data collection cycle and enriching the data volume and data surface.

[0073] In addition, the denoising diffusion probability model is used and unsupervised mixed pre-training of multi-category defect data is introduced to guide the model to learn and understand the data distribution of images in the field of cloth defects, thereby making up for the huge difference between the data distribution of natural images and cloth defect images.

[0074] Reference Figure 2 , shows a flowchart of another method for generating a cloth defect image according to an embodiment of the present invention, which may include the following steps:

[0075] Step 201: display an interactive interface, wherein the interactive interface displays a text input box.

[0076] In some feasible embodiments, an interactive interface may be displayed in advance, and a text input box may be displayed in the interactive interface; the user may input corresponding defect text information into the text input box.

[0077] Exemplarily, in order to facilitate users to use and observe the prediction results, the algorithm of the embodiment of the present invention can be integrated into a system (i.e., a cloth defect image generation system); the system can create an API interface to implement the call of the algorithm; in addition, a user interaction interface can also be designed.

[0078] For example, Figure 3 As shown, the user can enter the defect text information in the text input box of "Input category text" in the cloth defect image generation user interface, and click the "Confirm generation" button to display the cloth defect image generated based on the defect text information in the interactive interface.

[0079] Step 202: Receive text information input by the user in the text input box, and determine defect text information according to the text information input in the text input box.

[0080] After receiving the text information input by the user in the text input box, the fabric defect image generation system can analyze and identify it to determine the corresponding defect text information.

[0081] Step 203: Input the defect text information into the preset integrated model to obtain a cloth defect image.

[0082] Then, the fabric defect image generation system can input the defect text information into the preset integrated model.

[0083] The preset integrated model can automatically reconstruct the image based on the defect text information and the random cloth image, thereby generating a cloth defect image. Compared with collecting training data, the embodiment of the present invention automatically generates training data, thereby reducing the data collection cycle and enriching the data volume and data surface.

[0084] In addition, the denoising diffusion probability model is used and unsupervised mixed pre-training of multi-category defect data is introduced to guide the model to learn and understand the data distribution of images in the field of cloth defects, thereby making up for the huge difference between the data distribution of natural images and cloth defect images.

[0085] In an embodiment of the present invention, the defect text information input by the user is determined; the defect text information is input into a preset integrated model to obtain a fabric defect image. The integrated model is generated by integrating multiple second models obtained by fine-tuning multiple first models obtained after unsupervised hybrid pre-training based on multi-category defect data.

[0086] Through the embodiments of the present invention, data for training can be automatically generated, thereby reducing the data collection cycle and enriching the data volume and data surface.

[0087] In addition, the denoising diffusion probability model is used and unsupervised mixed pre-training of multi-category defect data is introduced to guide the model to learn and understand the data distribution of images in the field of cloth defects, thereby making up for the huge difference between the data distribution of natural images and cloth defect images.

[0088] In an embodiment of the present invention, the above method may further include the following steps:

[0089] Generate a first data set and a second data set; wherein the first data set includes images of good products and images of defects; the second data set includes defect images, labels corresponding to the defect images, and category texts corresponding to the defect images; based on the first data set, perform unsupervised mixed pre-training of multi-category defect data to obtain multiple first models; based on the second data set, train the multiple first models to obtain a second model.

[0090] In some feasible embodiments, a first data set and a second data set may be generated first; wherein the first data set may include good product images and defective images, and the second data set may include defective images, labels corresponding to the defective images, and category texts corresponding to the defective images.

[0091] Specifically, a good product image may refer to image data of a cloth without defects, and a defective image may refer to image data of a cloth with defects.

[0092] The label is an image, called a label image or a mask image. The label image has the same size as the defect image, but the pixel values ​​of the label image represent the category to which each pixel belongs. In binary segmentation tasks, the pixel values ​​of the label image are usually 0 or 1 (or 255), representing the background and foreground, respectively.

[0093] The category text corresponding to the defect image can be used by professional technical personnel of the organization to classify and annotate the defect image.

[0094] In practical applications, the original images can be first collected on the cloth production line through a high-resolution camera to ensure that the image clarity meets the training requirements and include as many production sites, multiple production lines, multiple cloth models, good products and multiple defect types as possible. The defect types include but are not limited to holes, yarn knots, wrinkles, broken wefts, etc.

[0095] Then, professional technicians can be organized to classify and annotate the collected images to ensure that there are enough samples for each category text. Good product images do not need to be annotated and are directly saved in the "good product" directory. For defective images, binary segmentation labels are annotated and single images and their labels are saved one by one in the directory with the corresponding category text name.

[0096] Preprocess defective images and their labels as well as good product images to ensure the consistency of the quality of the preprocessed images and labels.

[0097] Pre-processing includes but is not limited to:

[0098] a. Image denoising: Use filtering algorithm to remove image noise.

[0099] b. Unified size: resize all images to a uniform resolution of 512x512 pixels.

[0100] c. Data enhancement: Increase data diversity through random rotation, flipping, scaling, and center cropping operations.

[0101] Construct the first data set suitable for the unsupervised hybrid pre-training strategy of multi-category defect data: shuffle the preprocessed good product images and defect images and divide them into training set and validation set.

[0102] Construct the second data set suitable for the fine-tuning strategy of the multi-category cloth defect pre-training model: the pre-processed defect images and their labels are used for fine-tuning, and one defect image corresponds to one label and one category text as a triple data sample. Such triple data samples are shuffled and divided into training set, validation set, and test set to ensure that each sub-dataset contains data of all defect types.

[0103] After obtaining the first data set and the second data set, unsupervised mixed pre-training of multi-category defect data is performed based on the first data set to obtain multiple first models; then, fine-tuning training can be performed on the multiple first models based on the second data set to obtain the second model.

[0104] In one embodiment of the present invention, the above-mentioned step of performing unsupervised mixed pre-training of multi-category defect data according to the first data set to obtain multiple first models can be implemented by the following sub-steps:

[0105] According to the first data set, unsupervised mixed pre-training of multi-category defect data is performed on the preset denoising diffusion probability model based on the latent space to obtain multiple first models.

[0106] In some feasible embodiments, a preset denoising diffusion probability model based on a latent space may be pre-trained with unsupervised mixing of multi-category defect data according to the first data set to obtain a plurality of first models.

[0107] Exemplarily, a pre-training model can be designed first.

[0108] The denoising diffusion probability model G based on latent space is used for pre-training, including encoder, decoder, and latent space base model D, such as Figure 4 shown.

[0109] The encoder is a deep convolutional neural network that maps the first dataset to a latent space representation. The encoder is used to learn a high-dimensional representation of image data while capturing the statistical properties of the input image distribution. In fact, the encoder outputs the mean μ and variance σ of the latent space. 2 , these parameters are used to define the distribution in the latent space, namely

[0110] μ,σ=encoder(I)

[0111] The latent space base model D is another convolutional neural network based on UNet, such as Figure 5 As shown in the figure, it includes three downsampling and three upsampling, namely the first downsampling stage, the second downsampling stage, the third downsampling stage, the first upsampling stage, the second upsampling stage, and the third upsampling stage. Each stage contains convolution, normalization, activation function, and pooling layer.

[0112] The latent space basis model D can be used to further process and transform the data and learn how to restore the original image representation in a given noisy latent space representation, that is, to predict the latent space representation before noise based on the latent space representation after noise.

[0113] The process of obtaining the latent space representation z uses the reparameterization technique. Specifically, a vector ∈ is sampled from a standard normal distribution, and the mean and variance of the encoder output are used to calculate the latent space representation z = μ + σ·∈.

[0114] At the same time, noise is added to each latent space representation z to generate a noisy latent space representation, and the latent space base model D is used to predict the latent space representation z before noise addition based on the latent space representation after noise addition.

[0115] The decoder is also a deep convolutional neural network that maps the latent space representation back to the original data space, that is, generates and reconstructs the image from the latent space representation z

[0116]

[0117] Design unsupervised pre-training loss function:

[0118] a. Image reconstruction loss L1: Calculate the output of G The mean square error between (reconstructed image) and the original image I (the image in the training set of the first data set) before adding noise is calculated. When calculating the mean square error of the image, each color channel is calculated separately and then the average is taken. The formula of L1 is as follows:

[0119]

[0120] Where n is the total number of color channels.

[0121] b. KL divergence regularization loss L2: Introducing the KL divergence regularization term for the encoder output, that is, the distribution of the latent space, ensures that the probability distribution of the latent space is close to the standard normal distribution p(z), which helps to ensure the continuity and smoothness of the latent space. In fact, it is necessary to assume that the encoder output, that is, the distribution of the latent space, conforms to the standard normal distribution q(z|x), that is where μ and σ 2 is the mean and variance of the latent variable output by the encoder. The formula for L2 is as follows, where d is the dimension of the latent space, and are the mean and variance of the ith latent variable respectively. d is the dimension of the latent space

[0122]

[0123] c. The total unsupervised pre-training loss function L = a·L1+b·L2, where a and b are adjustable constant hyperparameters with values ​​between 0 and 1.

[0124] Execute unsupervised hybrid pre-training strategy:

[0125] During the training process, all parameters of G participate in parameter updating. The specific process is as follows:

[0126] a. Divide the image data of the training set in the first data set into different batches, randomly initialize the model parameters of G, and set the threshold of the unsupervised pre-training loss function L and the number of training iterations.

[0127] b. For each batch of image data input into G, the encoder obtains the latent space representation. At the same time, different levels of noise are added to the latent space representation to obtain the noisy latent space representation, and D is used to predict the latent space representation before noise addition, and the image is reconstructed through the decoder.

[0128] c. Calculate the unsupervised pre-training loss function L, and calculate the gradient of L with respect to the parameters of G through back propagation, and perform parameter updates.

[0129] d. Repeat the above training processes b and c until L of the model on the validation set in the first data set reaches a preset threshold or the number of repetitions reaches the set number of training iterations.

[0130] e. Save the parameter weights of model G as P, thereby obtaining multiple first models.

[0131] In one embodiment of the present invention, the above-mentioned step of training multiple first models according to the second data set to obtain the second model can be implemented by the following sub-steps:

[0132] Adding text encoders and cross-self-attention layers to multiple first models;

[0133] According to the second data set, multiple first models with text encoders and cross self-attention layers added are trained to obtain a second model.

[0134] In some feasible embodiments, text encoders and cross self-attention layers may be added to multiple first models; then, the multiple first models to which the text encoders and cross self-attention layers are added may be fine-tuned and trained according to the second data set to obtain the second model.

[0135] In an embodiment of the present invention, the above method may further include the following steps:

[0136] The multiple second models are integrated to obtain an integrated model.

[0137] In some feasible embodiments, integrated learning can be used to integrate multi-model prediction results; specifically, multiple second models can be integrated to obtain an integrated model.

[0138] In an embodiment of the present invention, the above method may further include the following steps:

[0139] Set a weight for each category text; the weight is inversely proportional to the frequency corresponding to the category text.

[0140] In practical applications, a weight may be set for the category text of each defect image; wherein, in order to improve the quality of generating rare category defect images, the weight may be inversely proportional to the frequency corresponding to the category text.

[0141] For example, Figure 6 - Figure 7 As shown, in order to achieve controllable generation of categories, Figure 5 Based on the model G in , a text encoder TE is introduced; the text encoder TE is a Transformer-based model that maps the input text to an embedding vector m with the same dimension as the latent space representation z.

[0142] When adding noise, you can sample from the latent space distribution and then add noise to the sampled latent space representation.

[0143] The embedding vector m contains the semantic features of the text.

[0144] right Figure 6 and Figure 7 A cross-attention layer AL (Cross-Attention Layer) is added between each stage of the UNet of the latent space base model D in , such as Figure 8 As shown. Based on the intermediate layer features f and the embedding vector m of UNet, the content of the generated image can be guided during the image generation process to ensure that the generated image matches the text description.

[0145] AL consists of fully connected layers FL1, FL2, FL3, a convolutional layer CONV, and a cross self-attention calculation operation SA. The fully connected layer is used to map the feature map dimension to the serialized vector dimension, and CONV is used to map the serialized vector dimension back to the feature map dimension. The calculation formula of SA is as follows, where Q and K are the vectors obtained by mapping f through FL1 and FL2, V is the vector obtained by mapping m through FL3, and d k is the vector dimension of QKV.

[0146] Q=FL1(f),K=FL2(f),V=FL3(m)

[0147]

[0148] The calculation formula of the cross self-attention layer AL is as follows.

[0149] AL=CONV(SA)

[0150] Rare category enhancement loss function L3: Due to the imbalanced distribution of different defect categories in the field data, in the second data set, the number of samples of each defect category is counted, and then the frequency of each defect category c is calculated

[0151] Then assign a weight to each c This can give rare categories a higher weight.

[0152] Therefore, the calculation of the rare category enhancement loss function L3 is as follows, where is the calculated value of the image reconstruction loss L1 on the data of defect category c, and C is the total number of defect categories.

[0153]

[0154] The total loss function in the fine-tuning stage is L ′ =q·L3+p·L2, where q and p are adjustable constant hyperparameters with values ​​between 0 and 1. By introducing the rare category enhancement loss function L3, the modeling ability of the fine-tuning model G' for rare category defect features is improved.

[0155] Train the optimal hyperparameter model for a single defect type. During the training process, only the parameters of the text encoder TE and the cross self-attention layer AL of G are updated, and the remaining parameters are locked. The specific process is as follows:

[0156] a. Divide the triplet data of the training set in the second dataset into different batches, and use the parameter weights P obtained in the second step to initialize the model parameters of G, use the open source CLIP (Contrastive Language-Image Pre-training) pre-training weights to initialize the text encoder TE, and use random initialization for the cross self-attention layer. At the same time, set the threshold of the rare category enhancement loss function L' and the number of training iterations.

[0157] b. For each batch of data input into G, the encoder obtains the latent space representation. At the same time, different levels of noise are added to the latent space representation to obtain the latent space representation after noise addition, and the embedding vector of the defect category is obtained through the text encoder. The latent space representation before noise addition is predicted using D with AL, and the image is reconstructed through the decoder.

[0158] c. Calculate the loss function L', calculate the gradient of L' with respect to the parameters of G' through back propagation, and update the parameters of TE and AL.

[0159] d. Repeat the above training process b and c until the L' of the model on a certain defect type c in the validation set of the second data set reaches a preset threshold or the number of repetitions reaches the set number of training iterations.

[0160] e. Save the model G′ that performs best in the generation results of a certain type of defect c c , the parameter weight is P c , thus obtaining the second model.

[0161] Train the optimal hyperparameter model for multiple defect types:

[0162] For each defect type c, the optimal parameter weight P of the fine-tuning model G' is obtained c , the optimal hyperparameter model of the corresponding defect category c is recorded as G′ c .

[0163] Using the ensemble learning method based on the fusion of prediction results, an ensemble model G is proposed. ALL , and perform inference on the test set of the second dataset. The structures used in the inference stage in G' are the decoder, the latent space base model D with the cross self-attention mechanism, and the text encoder. The encoder is not used directly, but directly from the learned latent space distribution (mean μ and variance σ 2 ) in the resampled latent space representation z.

[0164] like Fig. 9 As shown in Figure 2, due to the characteristics of the image generation task, weighted averaging is used to merge the optimal hyperparameter model G′ of all defect categories c. c For each triple sample (I, Mask, Text) in the test set of the second dataset, through G′0, G′1, G′2, ..., G′ c After testing, we can get C (the total number of defect categories) of reconstructed images, each of which is recorded as

[0165]

[0166] The final model output is:

[0167]

[0168] Among them, G ALL It is an integrated model.

[0169] In this way, the generative models under the optimal hyperparameters of different defect categories are integrated to improve the generalization ability of the model for all defect categories.

[0170] like Fig.10, shows a defect image 10 corresponding to a thread head, and a label 11 corresponding to the defect image 10.

[0171] like Fig.11 As shown, various images of the scene can be generated, and after preprocessing and annotation, they can be saved as the first data set and the second data set respectively; the first data set is used for unsupervised mixed pre-training of multi-category defect data, and the second data set is used for fine-tuning strategies for pre-training models of multi-category cloth defects. Through ensemble learning, the ensemble model G can be obtained. ALL .

[0172] The user can call the integrated model through the interactive interface to generate new fabric defect images for training the model for detecting fabric defects.

[0173] In the embodiment of the present invention, an interactive interface is displayed, and the interactive interface displays a text input box; the text information input by the user in the text input box is received, and the defect text information is determined according to the text information input in the text input box; the defect text information is input into a preset integrated model to obtain a fabric defect image. Through the embodiment of the present invention, data for training can be automatically generated, thereby reducing the data collection cycle and enriching the data volume and data surface.

[0174] In addition, the denoising diffusion probability model is used and unsupervised mixed pre-training of multi-category defect data is introduced to guide the model to learn and understand the data distribution of images in the field of cloth defects, thereby making up for the huge difference between the data distribution of natural images and cloth defect images.

[0175] In addition, rare category enhancement loss functions and integrated learning techniques are introduced in the embodiments of the present invention to improve the quality and authenticity of various types of cloth defect images generated by the model and improve the generalization of the model. The method of the present application can model different types of cloth defects, including but not limited to holes, stains, color differences, and uneven fabric structures. This helps to accelerate the development of downstream task algorithms and project implementation.

[0176] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0177] Reference Fig.12 , shows a schematic structural diagram of a device for generating a cloth defect image according to an embodiment of the present invention, which may include the following modules:

[0178] Determining module 1201, used to determine the defect text information input by the user;

[0179] The generation module 1202 is used to input the defect text information into a preset integrated model to obtain a fabric defect image; wherein the integrated model is generated by integrating multiple second models obtained after fine-tuning multiple first models obtained after unsupervised mixed pre-training based on multi-category defect data.

[0180] In an optional embodiment of the present invention, the device further comprises:

[0181] A display module, used to display an interactive interface, wherein the interactive interface displays a text input box;

[0182] The determination module 1201 is used to receive text information input by a user in a text input box, and determine defect text information according to the text information input in the text input box.

[0183] In an optional embodiment of the present invention, the device further comprises:

[0184] A training module is used to generate a first data set and a second data set; wherein the first data set includes images of good products and images of defects; the second data set includes defect images, labels corresponding to the defect images, and category texts corresponding to the defect images; based on the first data set, unsupervised mixed pre-training of multi-category defect data is performed to obtain multiple first models; based on the second data set, multiple first models are trained to obtain a second model.

[0185] In an optional embodiment of the present invention, the training module is used to perform unsupervised mixed pre-training of multi-category defect data on a preset denoising diffusion probability model based on a latent space according to the first data set to obtain multiple first models.

[0186] In an optional embodiment of the present invention, a training module is used to add a text encoder and a cross-self-attention layer to multiple first models; according to a second data set, the multiple first models after adding the text encoder and the cross-self-attention layer are trained to obtain a second model.

[0187] In an optional embodiment of the present invention, the training module is further used to set a weight for each category text; the weight is inversely proportional to the frequency corresponding to the category text.

[0188] In an optional embodiment of the present invention, the training module is further used to integrate multiple second models to obtain an integrated model

[0189] In an embodiment of the present invention, the defect text information input by the user is determined; the defect text information is input into a preset integrated model to obtain a fabric defect image. The integrated model is generated by integrating multiple second models obtained by fine-tuning multiple first models obtained after unsupervised hybrid pre-training based on multi-category defect data.

[0190] By introducing an unsupervised hybrid pre-training strategy for multi-category defect data, the model is guided to learn and understand the data distribution of images in the field of cloth defects, while reducing the annotation cost.

[0191] The rare category enhancement loss function is introduced to give higher weights to rare defect categories, guiding the model optimization process to pay more attention to these categories.

[0192] Multiple models are trained using different hyperparameter settings, and the prediction results of multiple models are integrated through ensemble learning to improve the model's generalization ability for all defect categories.

[0193] An embodiment of the present invention further provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the above-mentioned method for generating a fabric defect image when executed by the processor.

[0194] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for generating a cloth defect image as described above is implemented.

[0195] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0196] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0197] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0198] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0199] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0200] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0201] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0202] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0203] The above provides a detailed introduction to a method for generating a fabric defect image, a device for generating a fabric defect image, an electronic device and a storage medium. In this article, specific examples are used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for generating a cloth defect image, characterized in that: The method comprises: Determine the defect text information entered by the user; Inputting the defect text information into a preset integrated model to obtain the fabric defect image; The integrated model is generated by integrating multiple second models obtained after fine-tuning multiple first models obtained after unsupervised mixed pre-training based on multi-category defect data.

2. The method according to claim 1, characterized in that The method further comprises: Displaying an interactive interface, wherein the interactive interface displays a text input box; The step of determining the defect text information input by the user includes: The text information input by the user in the text input box is received, and the defect text information is determined according to the text information input in the text input box.

3. The method according to claim 1, characterized in that The method further comprises: Generate a first data set and a second data set; wherein the first data set includes good product images and defective images; the second data set includes defective images, labels corresponding to the defective images, and category texts corresponding to the defective images; According to the first data set, unsupervised mixed pre-training of multi-category defect data is performed to obtain multiple first models; The multiple first models are trained according to the second data set to obtain the second model.

4. The method according to claim 3, characterized in that The method of performing unsupervised mixed pre-training of multi-category defect data according to the first data set to obtain multiple first models includes: According to the first data set, unsupervised mixed pre-training of multi-category defect data is performed on the preset denoising diffusion probability model based on the latent space to obtain multiple first models.

5. The method according to claim 4, characterized in that The step of training the plurality of first models according to the second data set to obtain the second model comprises: Adding a text encoder and a cross self-attention layer to the plurality of first models; According to the second data set, the multiple first models after adding text encoders and cross self-attention layers are trained to obtain the second model.

6. The method according to claim 3, characterized in that The method further comprises: A weight is set for each category text; the weight is inversely proportional to the frequency corresponding to the category text.

7. The method according to claim 3, characterized in that The method further comprises: The multiple second models are integrated to obtain the integrated model.

8. A device for generating a cloth defect image, characterized in that: The device comprises: A determination module, used to determine the defect text information input by the user; A generation module is used to input the defect text information into a preset integrated model to obtain the fabric defect image; wherein the integrated model is generated by integrating multiple second models obtained after fine-tuning multiple first models obtained after unsupervised mixed pre-training based on multi-category defect data.

9. An electronic device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the method for generating a cloth defect image according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for generating a cloth defect image according to any one of claims 1 to 7 is implemented.

Images