Synthetic leather defect image generation and semantic annotation method

Abstract

The invention discloses a synthetic leather defect image generation and semantic annotation method, relates to the technical field of image generation, and solves the problems that an existing image generation model is limited in controllability and editability and not flexible enough, and it is difficult to generate an image through image-text combination. The method comprises the following steps: segmenting an input large-size synthetic leather defect image into a plurality of local sub-images with the same size; editing the text description about the image through an improved language model to generate an editing instruction and an edited semantic annotation; the segmented local sub-images, the editing instruction and the edited semantic annotation are input into an image generation model together to generate an image with defects, and meanwhile, the generated image is provided with the semantic annotation, so that the data enhancement effect of the synthetic leather defect image can be achieved; meanwhile, defect images with different effects can be generated according to user semantic definition, and the training effect and training efficiency of the synthetic leather defect detection model can be greatly improved.

Description

Technical Field

[0001] This application relates to the field of image generation technology, and in particular to a method for generating images of defects in synthetic leather and its semantic annotation. Background Technology

[0002] Synthetic leather products are widely used in clothing, bags, shoes and even furniture, and have gradually become an indispensable part of people's lives. As people's living standards continue to improve, their requirements for the quality and appearance of synthetic leather products are also becoming higher and higher. At present, the appearance quality inspection of synthetic leather mainly relies on manual visual observation. With the development of computer technology, deep learning-based machine vision technology has gradually begun to be applied in the detection of defects in synthetic leather. Compared with traditional manual defect detection methods, this method can improve the quality and efficiency of detection, while also reducing the labor cost of detection (Hong Jingshan, Zhu Yingdan, Song Kangkang, et al. Research progress of industrial defect detection based on deep learning [J]. Computer Science, 2024, 51(10):261-275.). However, deep learning defect detection models require a large amount of data on normal synthetic leather images or defective synthetic leather images during training. In actual manufacturing scenarios, normal synthetic leather images are predefined, but defective synthetic leather images are randomly acquired, often with a long collection period, especially for uncommon defects that are prone to customer complaints. This results in insufficient samples and significant differences in the proportion of different types of samples in the synthetic leather defect image data. Consequently, the defect detection model's adaptability is insufficient, and the false negative rate for uncommon defects in synthetic leather is high. Furthermore, the lack of unified and standardized semantic annotations for the collected synthetic leather defect images prevents the defect detection model from automatically recognizing image defect information. Therefore, given the limited amount of synthetic leather defect image data available, we need to increase the number of defect samples through data augmentation techniques and establish a semantic annotation format containing rich defect information to meet the training requirements of deep learning models for defect detection.

[0003] In this context, image generation models have emerged, such as GANs (Goodfellow IJ, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets[J]. Advances in neural information processing systems, 2014, 27). However, GAN models have limited controllability and editability, are not flexible enough, and are difficult to combine with text to generate images. Therefore, there is a need for a controllable model that can accurately generate images that match the text description. Summary of the Invention

[0004] The purpose of this application is to overcome the problems of existing synthetic leather defect images being few in number, having long acquisition cycles, high semantic annotation costs, and limited controllability and editability of image generation models, as well as their inflexibility and difficulty in combining text and images to generate images, and to provide a method for generating synthetic leather defect images and their semantic annotation.

[0005] Firstly, a method for generating and semantically annotating images of defects in synthetic leather is provided, including:

[0006] Obtain synthetic leather defect images and their semantic annotation ranges, as well as user requirements, wherein the user requirements include one or more of the defect type, defect size, and defect location in the synthetic leather defect images to be generated, and one or more of the color, size, and number of synthetic leather defect images;

[0007] The synthetic leather defect image is segmented into at least one local sub-image, and semantic annotation is performed on each local sub-image within the scope of user requirements;

[0008] The semantic annotations of the user requirements and the synthetic leather defect images are processed using an improved language model to generate editing instructions and semantic annotations for the edited images;

[0009] An image generation model is constructed and trained, wherein the image generation model includes an encoder, a U-Net, and a decoder. The encoder is used to reduce the dimension of the input image into a low-dimensional image while adding noise to obtain a low-dimensional latent representation. The U-Net is used to combine the input image with the edited text description to generate a denoised latent representation. The decoder is used to convert the denoised latent representation output by the U-Net back into the image space to generate a synthetic leather defect image that conforms to the description of the input text description.

[0010] The local sub-image, editing instructions, and semantic annotations of the edited image are input into the trained image generation model to generate a synthetic leather defect image that meets the semantic definition required by the user.

[0011] In some possible implementations, image segmentation techniques are used to segment the synthetic leather defect image into at least one local sub-image, wherein the local sub-images are of the same size and conform to the size information of the synthetic leather defect image required by the user.

[0012] In some possible implementations, the U-Net includes downsampling, upsampling, and skip connections, with a prompt-to-prompt method (i.e., a prompt-to-prompt method) introduced between the downsampling and upsampling. The prompt-to-prompt method includes a cross-attention weight borrowing mechanism and CLIP loss.

[0013] In some possible implementations, the downsampling is used to analyze the image and extract image features in conjunction with the input text description, the upsampling is used to accurately locate and generate details using the image features extracted by downsampling, and the skip connection is used to pass the details lost during downsampling to upsampling and concatenate them with the feature maps of the corresponding layers in the upsampling path to recover accurate details when reconstructing the image.

[0014] In some possible implementations, the prompt-to-prompt method is used to allow two similar text descriptions to share cross-attention weights during the denoising process. The cross-attention weight borrowing mechanism is used to encourage the image generation model to process text information in a more similar way during the generation process when dealing with similar text descriptions, so as to make the generated images more consistent in semantics and visuals. The CLIP loss is used to determine whether the changes in the generated image relative to the input image of the image generation model conform to the input text description and whether they are consistent in semantics.

[0015] In some possible implementations, the image generation model employs an improved stable diffusion model, which includes a spatial text compiler, U-Net, and a decoder. For an image x, the diffusion process outputs a latent representation to the spatial text compiler. Add noise to produce a noisy latent representation. The noise level will change with the time step. And increase, by learning a network Given image conditions and text instruction conditions In the case of predicting the noise added to the noisy latent representation zt, where the objective function of the diffusion process is: .

[0016] In some possible implementations, during the process of generating synthetic leather defect images by the image generation model, the encoder converts the input image of the image generation model into a noisy latent representation. The noisy latent representation and the text description are input into U-Net. Downsampling is used to analyze the image based on the text description and extract features. The extracted features, along with the details lost in the process, are passed to upsampling for precise localization and detail generation, and a denoised latent representation is generated and passed to the decoder. The decoder converts the denoised latent representation back into the image space and generates a synthetic leather defect image that conforms to the text description. The text description includes editing instructions and semantic annotations of the edited image.

[0017] Among some possible implementations, methods for improving language model acquisition include:

[0018] After filtering and collecting meaningful and clear text descriptions from the LAION-Aesthetics dataset, we paired the manually edited instructions with the output descriptions. The paired manually edited instructions and output descriptions were then input into the language model for one round of training to obtain an improved language model.

[0019] Secondly, an apparatus for generating and semantically annotating images of defects in synthetic leather is provided, comprising:

[0020] The basic image acquisition module is used to acquire synthetic leather defect images and their semantic annotation ranges, as well as user requirements. The user requirements include one or more of the defect type, defect size, and defect location in the synthetic leather defect images to be generated, as well as one or more of the color, size, and quantity of the synthetic leather defect images.

[0021] The image segmentation module is used to segment the synthetic leather defect image into at least one local sub-image and perform semantic annotation on each local sub-image within the scope of user requirements;

[0022] The text re-editing module is used to process the user requirements and the semantic annotations of the synthetic leather defect images through an improved language model to generate editing instructions and semantic annotations of the edited images;

[0023] The model building and training module is used to build and train an image generation model. The image generation model includes an encoder, a U-Net, and a decoder. The encoder is used to reduce the dimension of the input image into a low-dimensional image while adding noise to obtain a low-dimensional latent representation. The U-Net is used to combine the input image with the edited text description to generate a denoised latent representation. The decoder is used to convert the denoised latent representation output by the U-Net back into the image space to generate a synthetic leather defect image that conforms to the description of the input text description.

[0024] The image generation module is used to input the local sub-image, editing instructions, and semantic annotations of the edited image into the trained image generation model to generate a synthetic leather defect image that meets the semantic definition required by the user.

[0025] Thirdly, a computer-readable storage medium is provided that stores program code for execution by a device, the program code including steps for performing a method as described in any of the implementations of the first aspect above.

[0026] This application has the following beneficial effects: By constructing and training an image generation model, the model can generate new synthetic leather defect images that meet user needs based on existing synthetic leather defect images. It also possesses strong controllability and editability. Furthermore, by improving the language model to uniformly process user input, it generates higher-quality instructions and corresponding text descriptions with a relatively uniform format (i.e., editing commands and semantic annotations of the edited images). This allows the image generation model to better "understand" the text descriptions, thereby enabling it to generate higher-quality synthetic leather defect images that meet user needs. In addition, this application can obtain thousands or even more synthetic leather images containing defects from a single image after image segmentation and image generation processing. This not only greatly expands the training data for the synthetic leather defect detection model and significantly shortens the construction cycle of the synthetic leather defect image dataset, but also the generated new synthetic leather images come with built-in semantic annotations, greatly saving the manual and time costs of manually annotating new synthetic leather images. Attached Figure Description

[0027] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0028] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart of the method for generating and semantically annotating defect images of synthetic leather according to Embodiment 1 of this application;

[0030] Figure 2 This is a flowchart of image segmentation in the method for generating and semantically annotating images of defects in synthetic leather according to Embodiment 1 of this application;

[0031] Figure 3 This is a flowchart of the image generation model in the method for generating and semantically annotating images of synthetic leather defects according to Embodiment 1 of this application;

[0032] Figure 4 This is a structural diagram of the U-Net in the image generation model of the method for generating and semantically annotating images of synthetic leather defects in Embodiment 1 of this application;

[0033] Figure 5This is a flowchart of CLIP loss calculation in the method for generating and semantically annotating synthetic leather defect images according to Embodiment 1 of this application;

[0034] Figure 6(a) is an example of the input image, text description, and corresponding generated image in Embodiment 1 of this application. Figure 1 ;

[0035] Figure 6(b) is an example of the input image, text description, and corresponding generated image in Embodiment 1 of this application. Figure 2 ;

[0036] Figure 6(c) is an example of the input image, text description, and corresponding generated image in Embodiment 1 of this application. Figure 3 ;

[0037] Figure 7 This is a structural block diagram of the apparatus for generating and semantically annotating images of defects in synthetic leather according to Embodiment 2 of this application.

[0038] Figure label:

[0039] 100. Basic Image Acquisition Module; 200. Image Segmentation Module; 300. Text Re-editing Module; 400. Model Building and Training Module; 500. Image Generation Module. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] Example 1

[0042] like Figure 1 As shown, the method for generating and semantically annotating images of defects in synthetic leather according to Embodiment 1 of this application includes:

[0043] S100. Obtain the synthetic leather defect image and its semantic annotation range, as well as the user requirements. The user requirements include one or more of the defect type, defect size, and defect location in the synthetic leather defect image to be generated, and one or more of the color, size, and quantity of the synthetic leather defect image. The synthetic leather defect image can be provided by the user, selected from a preset image library, randomly extracted, or downloaded from the Internet. The user requirements include the semantic annotation of the synthetic leather defect image required by the user, which may include information such as the defect type, defect size, and defect location of the synthetic leather defect image required by the user, as well as information such as the color, size, and quantity of the synthetic leather defect image required by the user. For example: white leather surface with knife marks, or white leather surface with knife marks, or 300px*300px white leather surface with knife marks, or 300px*300px white leather surface with a 1mm long knife mark on the upper right corner of the surface, or one thousand 300px*300px white leather surfaces with knife marks. The knife mark location and knife mark size are randomly set, etc.

[0044] S200. The synthetic leather defect image is segmented into at least one local sub-image and semantic annotation is performed on each local sub-image within the scope of user requirements. The semantic annotation is done manually, that is, it is to indicate whether there is a defect in the local sub-image, as well as the type, size, location and other information of the defect.

[0045] Specifically, image segmentation technology is used to segment the synthetic leather defect image into at least one local sub-image. The local sub-images are of the same size and conform to the size information of the synthetic leather defect image required by the user. The image segmentation technology can read the size information of the input image and, according to the user's needs, the user can customize the size of the image to be segmented. All images after segmentation are of the same size.

[0046] like Figure 2 The diagram shows the image segmentation process. For a large input image of synthetic leather defects, the image size (H x W) is first obtained. Simultaneously, the desired size (Y x Y) of the local sub-images (i.e., the size information of the synthetic leather defect image required by the user) is determined. If Y is divisible by both H and W, then (H / Y) x (W / Y) local sub-images of the same size are output. For example, an input image of size 2048 x 2048 can be segmented into 16 images of size 512 x 512, or into 64 images of size 256 x 256. This design has the following characteristics:

[0047] (1) Users can set the desired size of the local synthetic leather defect image according to their own needs. If the segmentation requirements cannot be met, that is, the input image of size HxW cannot be completely segmented into the image of size YxY that the user wants, some images will not be segmented into images of size YxY. When the user wants the input image to be segmented into multiple local sub-images without any remaining parts, it is desirable to extract the defects in the defect image to the greatest extent possible, so that the defects in the subsequently generated defective images are similar to the existing defects.

[0048] (2) Segmenting the input large-size synthetic leather defect image into multiple local synthetic leather defect images of the same size can increase the number of samples in the existing synthetic leather dataset, thereby achieving a preliminary image data enhancement effect.

[0049] S300: The semantic annotations of the user requirements and the synthetic leather defect images are processed by an improved language model to generate editing instructions and semantic annotations of the edited images.

[0050] Specifically, the improved language model operates entirely in the text domain. The input to the improved language model is the semantic annotation of the synthetic leather defect image and the content information of the synthetic leather defect image required by the user (i.e., the semantic annotation of the initial synthetic leather defect image and the semantic annotation of the synthetic leather defect image to be generated). The output is the corresponding editing instruction and the edited text description. In order for the improved language model to better complete the text description matching process, for example, if the input text description is "picture of white leather" and the user wants to change the image to "white leather with knife marks on the surface", the improved language model needs to be able to generate both the editing instruction "make the leather surface have knife marks" and an output text description that has been appropriately modified, "image of white leather with knife marks on the surface".

[0051] To make the existing language model (i.e., the GPT model) more suitable for this task, it was modified on a small triplet dataset. Approximately 600 meaningful and clear text descriptions were filtered and collected from the LAION-Aesthetics dataset (i.e., the LAION aesthetics dataset, a large-scale open network aesthetics dataset for artificial intelligence). The instructions and output descriptions were then manually edited and paired. After pairing, the results were fed into the language model for one round of training to improve the language model.

[0052] After training, the improved language model includes more generated instructions and edited text descriptions, and can generate higher quality instructions and corresponding text descriptions for input text descriptions.

[0053] The improved language model operates entirely in the text domain, which allows for the generation of a large and diverse set of edits while ensuring a correspondence between image changes and text instructions. This enables the model to generate creative and reasonable instructions and descriptions, facilitating a better "understanding" of text descriptions during subsequent image generation.

[0054] In this embodiment, the improved language model uses a Transformer-based language model fine-tuned on a leather dataset. This model can generate new editing instructions and a revised text description (i.e., semantic annotations for the synthetic leather defect image required by the user) after editing the input text description. The overall idea is to map the "description of the image" to a combination of "editing instructions" and "new description." This design has the following characteristics:

[0055] (1) The weight file used during training consists of approximately 500 pairs of text-image pairs related to leather selected from the LAION-Aesthetic dataset, including the leather making process, leather products, and leather itself. By rewriting the text describing the images from a photography-related perspective in the dataset into text descriptions that focus on describing the content of the images, and manually writing editing instructions and semantic annotations for the edited images, for example, an image about "part of a brown leather jacket" has a text description in the dataset that is "photograph of a brown leather jacket, close-up, rich in detail". By rewriting it, it is transformed into "image of a brown leather jacket". At the same time, in order to turn the above image into an image about "part of a brown leather jacket with wrinkles", it is necessary to write the editing instruction "add wrinkle texture" and the semantic annotation of the edited image "brown leather jacket with wrinkle texture".

[0056] (2) During the training of the weighted file, 400 data pairs were selected as the training set, and the remaining data pairs were used as the test set. During training, the model predicted relevant editing instructions for the input text description, and then obtained the semantic annotation of the edited image based on the predicted editing instructions. In the process of predicting editing instructions, the model calculated a probability distribution representing the possible editing instructions. For example, for the image of a brown leather jacket, the model might predict editing instructions such as "add wrinkles", "add a hole", "make it curl", and other results that are not related to leather defects. Since the only correct instruction is "add wrinkles", the model calculates the cross-entropy loss based on the difference between its predicted result and the actual result. Then, through the backpropagation algorithm, the obtained loss value is backpropagated from the output layer to the input layer to calculate the gradient of the loss function with respect to each parameter in the model, so that the model has a higher probability of predicting the editing instruction "add wrinkles" when it encounters the text "image of a brown leather jacket" again. Since the amount of data used for training is not large, multiple rounds of training may lead to overfitting. Therefore, only one round of training with fixed parameters is performed.

[0057] (3) The operation is carried out entirely in the text domain. The trained weight file is combined with a large language model to receive text descriptions about the image. The operation in the text domain can ensure that there is a correspondence between image changes and text instructions, while generating a large number of diverse edits. This makes the model creative while generating reasonable instructions and descriptions.

[0058] (4) The “text-first” strategy of mapping “description of the image” to a combination of “editing instructions” and “new description” can make the subsequent images generated by the image generation model have high-quality text supervision signals, so that the generated images conform to the input text description.

[0059] S400. Construct and train an image generation model, wherein the image generation model includes an encoder, a U-Net (i.e., a U-Net network, a U-shaped symmetric semantic segmentation network), and a decoder. The encoder is used to reduce the dimension of the input image into a low-dimensional image while adding noise to obtain a low-dimensional latent representation. The U-Net is used to combine the input image with the edited text description to generate a denoised latent representation. The decoder is used to convert the denoised latent representation output by the U-Net back into the image space to generate a synthetic leather defect image that conforms to the description of the input text description.

[0060] In this embodiment, the image generation model uses an improved stable diffusion model, which includes a spatial text compiler, U-Net, and a decoder. That is, the encoder of the stable diffusion model is replaced with the spatial text compiler.

[0061] like Figure 3 As shown, for an image The diffusion process will contribute to the potential representation output by the spatial text compiler. Add noise to produce a noisy latent representation. The noise level will change with the time step. And increase, by learning a network Given image conditions In the case of text instruction conditional CT, predictions are added to the latent representation with noise. The noise above, where the objective function of the diffusion process is:

[0062] ;

[0063] In the above equation, L represents the potential diffusion objective function that needs to be minimized; This represents the expected value of the noise that needs to be calculated; The noise is added to the potential space, which is the target predicted by the model view. The noise follows a standard normal distribution with a mean of 0 and a variance of 1. t represents the step size of the noise level added during the diffusion process. Its value ranges from t to T, which is a specific moment in all time steps T of the entire diffusion process. This represents the noisy latent representation obtained by adding noise to the latent vector encoded by the spatial text compiler. This indicates the network that needs to be trained to predict the addition of noisy latent representations. Noise in; This indicates the image conditions, i.e., the input image to be edited; This represents textual instructions, specifically textual descriptions that tell the model how to edit the image. The latent representation obtained after encoding the image conditions; This indicates an encoder; in this case, a spatial text compiler is used.

[0064] For operators: The square of the 2-norm, geometrically representing the square of the Euclidean distance, is used in this formula to measure the difference between the "actual noise" and the "predicted noise," i.e., the mean squared error (MSE). In general, the objective function aims to make the network... Able to represent the state based on the current noise potential Time step t, latent representation and text instruction conditions This allows for accurate prediction of the actual noise to be added in the subsequent denoising process.

[0065] It should be noted that the same dataset used in the training dataset of the image generation model is the LAION-Aesthetic dataset, which is the same dataset used in the training of the improved language model. This is because the improved language model was trained first, and the LAION-Aesthetic dataset was already processed during the training of that model (selection of text-image pairs and rewriting of text descriptions). Therefore, no additional processing of this dataset is needed when training the image generation model. (The main model used in the text image generation model, stable diffusion, is a text image generation model. The training requires a text-image pair dataset. Stable diffusion is a generative artificial intelligence model based on deep learning.)

[0066] During the training of the image generation model, the encoder first converts the leather image of the input text-image pair into a "latent leather representation" and simultaneously uses CLIP (i.e., contrastive language-image pre-trained model) to convert the corresponding leather text description into a "text vector". Then, the model generates random noise at a randomly selected time step t. This noise is then added to the "latent leather representation" to obtain the "noisy latent leather representation". The "noisy latent leather representation", the "text vector", and the selected time step t are input into the U-Net. This structure is mainly used to combine the "text vector" and the "noisy latent leather representation" to predict the amount of noise added to the "leather latent representation". The predicted noise is then... Will add real noise Both are input into the loss function to calculate the difference between them. This is the mean squared error. The parameters of U-Net are then adjusted based on this error (if the prediction is accurate, the parameters are maintained; if the prediction is inaccurate, the parameters are adjusted so that the next prediction can be more accurate). Because stable diffusion is a process of adding noise to the input image and then combining it with text descriptions to denoise the latent representation of the noise to obtain a new image, during training, the main focus is on predicting the noise rather than predicting the original image. At the same time, predicting the noise has the same effect as generating the image, but it performs more stably during training.

[0067] In the image generation model, the input image is first reduced in dimensionality to a low-dimensional image by a spatial text compiler. Simultaneously, noise is added to further transform it into a low-dimensional noisy latent representation, generating a new text description that fuses spatial and text embedding codes. Then, the two outputs of the spatial text compiler are fed into U-Net. U-Net analyzes the image based on the text description and extracts image features to transform the image into a latent representation that conforms to the text description. This representation is then passed to the decoder for re-conversion into image space, thereby generating an image that conforms to the text description. This design has the following characteristics:

[0068] (1) The ordinary encoder in the original stable diffusion was replaced with a spatial text compiler. This compiler uses Fourier position coding to encode the spatial and textual descriptions, including frequency sequence generation, angle calculation, and sine and cosine transforms. In the frequency sequence generation part, the frequency freq of the geometric sequence is... Where B represents the number of frequency bands; in the angle calculation section, for angles, there are angles. x，b =x*2 b *2π, where x represents the normalized coordinates ranging from 0 to 1, and b represents the frequency component ranging from 0 to B-1; in the sine and cosine transform part, sine and cosine codes are calculated for each angle. For the encoded code, encoded = [sin(angles), cos(angles)], and finally the captured description is converted into a vector of length 2*B.

[0069] (2) The spatial text compiler can perform multi-scale position awareness, where the low-frequency component b is used to capture coarse position information and the high-frequency classification B is used to capture fine position information. This allows the text compiler to focus on and extract position features at different scales. At the same time, the continuity of the sine and cosine functions makes adjacent coordinate values ​​generate smoothly changing codes, which helps the text compiler learn continuous spatial relationships, thereby ensuring continuity.

[0070] (3) Compared to the original encoder, the spatial text compiler can handle spatially related text descriptions better. For example, for a white leather image without defects, if the user wants to create a hole on the left side of the image, there can be a text description "create a hole on the left side of the white leather image". For this text description, the original compiler mainly focuses on "white leather" and "create a hole", while the spatial text compiler will prioritize the spatial location information of "left side of the image" and then process the two descriptions "white leather" and "create a hole".

[0071] (4) In order to support image conditions The input is first addressed by adding an extra input channel to the first convolutional layer of the model. Specifically, this involves adding a noisy latent representation. and image conditions The results E(cI) encoded by encoder E are concatenated so that the model can simultaneously consider the potential representation of the image and related image condition information at the beginning of data processing, thereby better editing the image based on this information.

[0072] (5) The input of the original stable diffusion text conditional mechanism has been changed to text editing commands. This allows the model to apply the mechanism originally used to generate images by combining text descriptions to processing text editing instructions, so that the model can perform corresponding editing operations on the images according to these instructions.

[0073] (6) The decoder and encoder pairing in the image generation model is a convolutional neural network that reconstructs the latent representation back into a pixel image. It is also the last part of the image generation model. It mainly learns how to reconstruct the semantically simplified and compressed low-dimensional latent representation generated by U-Net back into a visually reasonable and coherent image space. As the final output stage of the image generation model, the decoder is used to reconstruct the low-dimensional latent representation generated by U-Net after denoising back into a high-resolution pixel image.

[0074] (7) There is also a scheduler in the image generation model that performs noise detection on the output image of U-Net. If the output latent representation after a single denoising by U-Net still has noise, it will be returned to U-Net for a second denoising.

[0075] In this embodiment, U-Net includes downsampling, upsampling, and skip connections. A cue-to-cue-word method is introduced between downsampling and upsampling, comprising a cross-attention weight borrowing mechanism and CLIP loss. Downsampling is used to analyze the image and extract image features in conjunction with the input text description. Upsampling is used to accurately locate and generate details using the image features extracted by downsampling. Skip connections are used to pass the details lost during downsampling to upsampling, concatenating them with the feature maps of the corresponding layers in the upsampling path to recover accurate details during image reconstruction. The cue-to-cue-word method allows two similar text descriptions to share cross-attention weights during denoising. The cross-attention weight borrowing mechanism encourages the image generation model to process text information in a more similar manner when dealing with similar text descriptions, resulting in more semantically and visually consistent generated images. The CLIP loss is used to determine whether the changes in the generated image relative to the input image of the image generation model conform to the input text description and whether they maintain semantic consistency.

[0076] One of the segmented sub-images and the edited text description are input into the generative model. The spatial text compiler extracts the spatial location information from the text description and converts it into spatial embedding encoding. At the same time, it converts the remaining text descriptions into text embedding encoding. By fusing the spatial embedding encoding and text embedding encoding, a new text description is generated. Meanwhile, the input image is transformed into a positional image and noise is added to further transform it into a low-dimensional noisy latent representation. The two outputs of the encoding are then fed into U-Net. U-Net analyzes the image based on the newly generated text description and extracts image features to transform the image into a latent representation that conforms to the text description. Its output is passed to the encoder to be converted back into image space, thereby generating an image that conforms to the text description.

[0077] like Figure 4 As shown, the area within the red box represents downsampling, the area within the blue box represents upsampling, the orange arrows indicate skip connections, and the black arrows indicate the method of introducing cue words to cue words for each sampling block. After the input image and input text description are passed to U-Net, they first enter the downsampling layer. This module analyzes the image and extracts image features in conjunction with the text description. The image features obtained from the downsampling layer are then output to the upsampling layer, where precise localization of image features, generation of details, and restoration of spatial resolution are performed. During this process, skip connections also pass some details lost during downsampling to the upsampling layer, concatenating them with the feature maps of the corresponding sampling layers in the upsampling path to ensure accurate detail recovery during image reconstruction. This design has the following characteristics:

[0078] (1) The downsampling block in the U-Net downsampling layer consists of a series of convolutional layers with a kernel size of 3, max pooling layers with a kernel size of 2, and a ReLU activation function used to correct the convolutional image. This is used to gradually reduce the spatial resolution of the input image and increase the number of channels, while simultaneously extracting image features from local details to global semantics by combining text descriptions. Figure 4 As can be seen from the process within the red box, the resolution of the input image gradually decreases and the number of channels gradually increases as the downsampling layer moves down. Furthermore, during the downsampling process, the area of ​​the original input image that each feature point can "see" becomes larger and larger. After reaching the bottleneck layer, the network is able to capture the global structure and semantic information of the image, which is crucial for understanding the text description and combining it to generate a reasonable image.

[0079] (2) The upsampling block in the U-Net upsampling layer consists of a series of convolutional layers with a kernel size of 3, an upward convolutional layer with a kernel size of 2, and a ReLU activation function used to correct the image after convolution. This is used to gradually increase the resolution of the image processed by the downsampling layer and reduce the number of channels. At the same time, the image is analyzed and image features are extracted in conjunction with the text description to ensure that the final generated image content is both clear and consistent with the text description. Figure 4 As can be seen from the process within the blue box, the output image after downsampling gradually increases in resolution and decreases in the number of channels as the upsampling layer moves up. During the upsampling process, the upsampling block combines text descriptions to further extract and analyze features from the image, ensuring that the generated image conforms to the instructions.

[0080] (3) High-resolution images input into U-Net or after a small number of downsampling blocks contain rich details and texture information, while low-resolution images that have undergone feature extraction through multiple downsampling blocks and are close to the bottleneck layer contain strong semantic and structural information. Because high and low resolution images contain different information, images lose details such as edges and textures during downsampling. The skip connections in U-Net will reconstruct the details lost during downsampling by splicing them with the feature maps of the corresponding sampling layers in the upsampling path, thereby ensuring that the details of the original image can be accurately recovered during the reconstruction process, and ensuring that the parts of the image that do not want to be changed can be better preserved.

[0081] (4) In the process of denoising and extracting features from images by downsampling and upsampling, a prompt word to prompt word method is also introduced in each sampling block to allow two similar text descriptions before and after editing to share cross attention weights in the denoising process of generating corresponding images.

[0082] In this embodiment, the cue-to-cue method introduced during downsampling and upsampling includes a cross-attention weight borrowing mechanism and CLIP loss calculation. During denoising, the model progressively predicts and removes noise based on the noisy latent representation and the input text description. The cue-to-cue method ensures continuity between image pairs by borrowing cross-attention weights. Specifically, for two similar text descriptions before and after editing, in the denoising process of generating corresponding images, the two generation processes share cross-attention weights in a certain proportion of the denoising steps. This is introduced to ensure that the image pair only changes the parts required by the instruction, maintaining a high degree of structural consistency. This design has the following characteristics:

[0083] (1) The cross-attention mechanism is used to interact text information with latent representations. After introducing the prompt-to-prompt method, when processing similar text prompts, in the selected denoising steps, the cross-attention weights in the process of generating an image from one text description will be borrowed by the process of generating an image from another text description. The effect of this is to make the two generation processes process the text information more similar in these steps, thereby making the generated images more consistent in semantics and visually. There is a parameter p used to determine what proportion of steps will borrow cross-attention weights in the whole denoising process: when the value of p is large, more denoising steps will share cross-attention weights, and the similarity of the generated images will be very high. When p equals 1, it means that the weights are shared throughout the process. At this time, the two images will be very similar, and the editing effect will be weakened. When the value of p is small, fewer denoising steps share cross-attention weights, and the differences in the generated images will be greater. When p equals 0, it means that the weights are not shared at all, which is equivalent to independent generation. At this time, the consistency of the images cannot be guaranteed.

[0084] (2) CLIP loss is used to determine whether the changes in the generated image relative to the input image conform to the input text description and whether they are semantically consistent. Its flowchart is as follows: Figure 5 As shown, the red box represents the calculation of text direction vector change, and the blue box represents the calculation of image direction vector change. The calculation formulas are as follows: and ,in This indicates the direction of semantic change from text A to text B. This represents the direction of visual content change from image A to image B. The cosine similarity can be calculated based on the changes in these two directional vectors, using the following formula: .

[0085] in Let || denote the dot product, || denote the L2 norm of the vectors, and cosine similarity ranges from -1 to 1. and When the directions are very close, the cosine similarity is close to 1, indicating that the changes in the image perfectly correspond to the changes required by the text instruction; if and When the directions are perpendicular, the cosine similarity approaches 0, indicating orthogonality and suggesting that the changes in the image are unrelated to the changes required by the text command; however, when... and When the direction deviates or is completely opposite, the cosine similarity becomes negative, and the changes in the image completely contradict the changes required by the text instruction.

[0086] (3) Since it is difficult to pre-determine the optimal parameter p for a given text description, multiple sets of image pairs are generated by randomly sampling different p values ​​(between 0.1 and 0.9). Then, the calculated CLIP index is used to filter these image pairs, and finally, the image pairs whose image changes are most consistent with the text description are retained as the result.

[0087] S500. Input the local sub-image, editing instructions, and semantic annotations of the edited image into the trained image generation model to generate synthetic leather defect images that meet the semantic definition required by the user. The generated partial images are shown in Figures 6(a), 6(b), and 6(c). In this way, one or a few sample images can be used to generate thousands or even more synthetic leather images containing defects through the method of this embodiment. This not only greatly expands the training data of the synthetic leather defect detection model, but also the generated new synthetic leather images come with semantic annotations. There is no need for manual semantic annotation of the generated synthetic leather images, which greatly saves the manual and time costs of semantic annotation of new synthetic leather images.

[0088] In this embodiment, by constructing and training an image generation model, the model can generate new synthetic leather defect images that meet user needs based on existing synthetic leather defect images. This model offers strong controllability and editability. Furthermore, by improving the language model to uniformly process user input, it generates higher-quality instructions and corresponding text descriptions with a more consistent format (i.e., editing commands and semantic annotations of the edited image). This allows the image generation model to better "understand" the text descriptions, thereby enabling it to generate higher-quality synthetic leather defect images that meet user needs. For example, users can specify features such as holes, knife marks, and wrinkles on the leather image, as well as the location of these features within the image. This significantly improves the controllability and editability of synthetic leather defect image generation and its semantic annotation. Simultaneously, the improved language model's unified optimization of user input greatly lowers the user's barrier to entry and improves the quality of synthetic leather defect image generation and its semantic annotation.

[0089] Example 2

[0090] like Figure 7 As shown, the apparatus for generating and semantically annotating images of defects in synthetic leather according to Embodiment 2 of this application includes:

[0091] The basic image acquisition module 100 is used to acquire synthetic leather defect images and their semantic annotation ranges, as well as user requirements. The user requirements include one or more of the defect type, defect size, and defect location in the synthetic leather defect images to be generated, as well as one or more of the color, size, and quantity of the synthetic leather defect images.

[0092] The image segmentation module 200 is used to segment the synthetic leather defect image into at least one local sub-image and perform semantic annotation on each local sub-image within the scope of user requirements;

[0093] The text re-editing module 300 is used to process the user requirements and the semantic annotations of the synthetic leather defect image through an improved language model to generate editing instructions and semantic annotations of the edited image;

[0094] The model building and training module 400 is used to build and train an image generation model. The image generation model includes an encoder, a U-Net, and a decoder. The encoder is used to reduce the dimension of the input image to a low-dimensional image while adding noise to obtain a low-dimensional latent representation. The U-Net is used to combine the input image with the edited text description to generate a denoised latent representation. The decoder is used to convert the denoised latent representation output by the U-Net back into the image space to generate a synthetic leather defect image that conforms to the description of the input text description.

[0095] The image generation module 500 is used to input the local sub-image, editing instructions, and semantic annotations of the edited image into the trained image generation model to generate a synthetic leather defect image that meets the semantic definition required by the user.

[0096] It should be noted that other specific implementations of the apparatus for generating and semantically annotating synthetic leather defect images in this embodiment can be found in the specific implementations of the method for generating and semantically annotating synthetic leather defect images described above. To avoid redundancy, these will not be repeated here.

[0097] Example 3

[0098] This application relates to a computer-readable storage medium in embodiment 3, which stores program code for execution by a device, the program code including steps for performing the method as described in any implementation of embodiment 1 of this application;

[0099] The computer-readable storage medium may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM); the computer-readable storage medium may store program code, and when the program stored in the computer-readable storage medium is executed by a processor, the processor is used to perform the steps of the method in any of the implementations of Embodiment 1 of this application.

[0100] The above are merely preferred embodiments of this application; however, the scope of protection of this application is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in this application, based on the technical solution and its improved concept, should be covered within the scope of protection of this application.

Claims

1. A method for generating and semantically annotating images of defects in synthetic leather, characterized in that, include: Obtain synthetic leather defect images and their semantic annotation ranges, as well as user requirements, wherein the user requirements include one or more of the defect type, defect size, and defect location in the synthetic leather defect images to be generated, and one or more of the color, size, and number of synthetic leather defect images; The synthetic leather defect image is segmented into at least one local sub-image, and semantic annotation is performed on each local sub-image within the scope of user requirements; The semantic annotations of the user requirements and the synthetic leather defect images are processed using an improved language model to generate editing instructions and semantic annotations for the edited images; An image generation model is constructed and trained, wherein the image generation model includes an encoder, a U-Net, and a decoder. The encoder is used to reduce the dimension of the input image into a low-dimensional image while adding noise to obtain a low-dimensional latent representation. The U-Net is used to combine the input image with the edited text description to generate a denoised latent representation. The decoder is used to convert the denoised latent representation output by the U-Net back into the image space to generate a synthetic leather defect image that conforms to the description of the input text description. The local sub-image, editing instructions, and semantic annotations of the edited image are input into the trained image generation model to generate a synthetic leather defect image that meets the semantic definition required by the user.

2. The method for generating and semantically annotating synthetic leather defect images according to claim 1, characterized in that, The synthetic leather defect image is segmented into at least one local sub-image using image segmentation technology, wherein the local sub-images are of the same size and conform to the size information of the synthetic leather defect image required by the user.

3. The method for generating and semantically annotating images of defects in synthetic leather according to claim 1, characterized in that, The U-Net includes downsampling, upsampling, and skip connections. A cue word to cue word method is introduced between the downsampling and upsampling methods. The cue word to cue word method includes a cross-attention weight borrowing mechanism and CLIP loss.

4. The method for generating and semantically annotating synthetic leather defect images according to claim 3, characterized in that, The downsampling is used to analyze the image and extract image features in conjunction with the input text description. The upsampling is used to accurately locate and generate details using the image features extracted by downsampling. The skip connection is used to pass the details lost during downsampling to upsampling and concatenate them with the feature maps of the corresponding layers in the upsampling path to recover accurate details when reconstructing the image.

5. The method for generating and semantically annotating images of defects in synthetic leather according to claim 3 or 4, characterized in that, The prompt-to-prompt method allows two similar text descriptions to share cross-attention weights during the denoising process. The cross-attention weight borrowing mechanism encourages the image generation model to process text information in a more similar way during the generation process when dealing with similar text descriptions, so as to make the generated images more consistent in semantics and visuals. The CLIP loss is used to determine whether the changes in the generated image relative to the input image of the image generation model conform to the input text description and whether they are consistent in semantics.

6. The method for generating and semantically annotating images of defects in synthetic leather according to claim 1, characterized in that, The image generation model employs an improved stable diffusion model, which includes a spatial text compiler, U-Net, and a decoder. For an image x, the diffusion process adds noise to the latent representation z=ε(x) output by the spatial text compiler to produce a noisy latent representation z. t The noise level increases with time step t∈T, by learning a network. Given image conditions cI and text instruction conditions cT, predict the noise to be added to the noisy latent representation zt, where the objective function of the diffusion process is: ； Where L represents the potential diffusion objective function that needs to be minimized; This represents the expected value of the noise that needs to be calculated; represents the actual noise added to the potential space; t represents the step size of the noise level added during diffusion. This represents the noisy latent representation obtained by adding noise to the latent vector encoded by the spatial text compiler. This indicates the network that needs to be trained; Indicates image conditions; Indicates text instruction conditions; The latent representation obtained after encoding the image conditions; Representational text compiler.

7. The method for generating and semantically annotating images of defects in synthetic leather according to claim 1, characterized in that, In the process of generating synthetic leather defect images by the image generation model, the encoder converts the input image of the image generation model into a noisy latent representation. The noisy latent representation and the text description are input into U-Net. Downsampling is used to analyze the image based on the text description and extract features. The extracted features and the details lost in the process are passed to upsampling for precise localization and detail generation, and a denoised latent representation is generated and passed to the decoder. The decoder converts the denoised latent representation back into the image space and generates a synthetic leather defect image that conforms to the text description. The text description includes editing instructions and semantic annotations of the edited image.

8. The method for generating and semantically annotating images of defects in synthetic leather according to claim 1, characterized in that, Improved methods for obtaining language models include: After filtering and collecting meaningful and clear text descriptions from the LAION-Aesthetics dataset, we paired the manually edited instructions with the output descriptions. The paired manually edited instructions and output descriptions were then input into the language model for one round of training to obtain an improved language model.

9. An apparatus for generating and semantically annotating images of defects in synthetic leather, characterized in that, include: The basic image acquisition module is used to acquire synthetic leather defect images and their semantic annotation ranges, as well as user requirements. The user requirements include one or more of the defect type, defect size, and defect location in the synthetic leather defect images to be generated, as well as one or more of the color, size, and quantity of the synthetic leather defect images. The image segmentation module is used to segment the synthetic leather defect image into at least one local sub-image and perform semantic annotation on each local sub-image within the scope of user requirements; The text re-editing module is used to process the user requirements and the semantic annotations of the synthetic leather defect images through an improved language model to generate editing instructions and semantic annotations of the edited images; The model building and training module is used to build and train an image generation model. The image generation model includes an encoder, a U-Net, and a decoder. The encoder is used to reduce the dimension of the input image into a low-dimensional image while adding noise to obtain a low-dimensional latent representation. The U-Net is used to combine the input image with the edited text description to generate a denoised latent representation. The decoder is used to convert the denoised latent representation output by the U-Net back into the image space to generate a synthetic leather defect image that conforms to the description of the input text description. The image generation module is used to input the local sub-image, editing instructions, and semantic annotations of the edited image into the trained image generation model to generate a synthetic leather defect image that meets the semantic definition required by the user.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code for execution by the device, the program code including steps for performing the method as described in any one of claims 1-8.

Images