Training method for compliance degree of interlaced diffusion optimization generated content to cue word

Abstract

The invention discloses a training method for the compliance degree of content generated through interlaced diffusion optimization on cue words. The method comprises the following steps: acquiring picture data # imgabs0 #, picture data # imgabs1 #, text data # imgabs2 # and text data # imgabs3 # from a data set, wherein the text data # imgabs2 # and the text data # imgabs3 # correspond to the picture data; fusing the picture data # imgabs4 # with first noise # imgabs5 # to obtain picture data # imgabs6 #, and inputting the fused picture data # imgabs7 # and text data # imgabs8 # into a diffusion generation network model to obtain first prediction noise; based on the first prediction noise, recovering picture data # imgabs9 #, and obtaining picture data # imgabs10 #; the image data # imgabs11 # and the image data # imgabs12 # are analyzed, and target noise is obtained; fusing the picture data # imgabs13 # with the first noise # imgabs14 # to obtain picture data # imgabs15 #, and inputting the picture data # imgabs16 # and the text data # imgabs17 # into the diffusion generation network model again to obtain second prediction noise; and training a neural network model based on the target noise and the second prediction noise. According to the method, the training process of Stable Diffusion can be enhanced, so that the generated image result is closer to the prompt.

Description

Technical Field

[0001] The present invention relates to the technical field of optimizing a Stable-Diffusion training process, and in particular to a training method for optimizing the compliance of generated content with prompt words through staggered diffusion. Background Art

[0002] Stable Diffusion is an advanced text-to-image generation technique that uses deep neural networks to convert input text prompts into detailed images. Although this technique has made significant progress in the field of image generation, it still has room for improvement in following text prompts. The generated images sometimes deviate from the input description, resulting in a certain degree of ambiguity and uncertainty in the results. Summary of the invention In view of this, the purpose of the present invention is to propose a training method for the compliance of generated content with prompt words by staggered diffusion optimization, which can strengthen the training process of Stable Diffusion, so that the generated image results are closer to the prompt words. By staggering different diffusion steps and prompt word optimization during the training process, the prompt word compliance of the model is improved, the ambiguity and uncertainty of the generated image are reduced, thereby improving the practicality and reliability in the application of accurate image generation.

[0003] According to one aspect of the present invention, a method for training the compliance of generated content with prompt words by staggered diffusion optimization is provided, and the method comprises the following steps: Get image data from a dataset and image data And the text data corresponding to the image data and text data ; The image data Fusion First Noise Get image data , and the fused image data With text data Input the diffusion generation network model to obtain the first predicted noise; restore the image data based on the first predicted noise , get the image data ; Analyze the image data With picture data , obtain the target noise; The image data Fusion First Noise Get image data , and the image data With text data Input the diffusion generation network model again to obtain the second prediction noise; The neural network model is trained based on the target noise and the second predicted noise.

[0004] In the above technical solution, the interlaced diffusion training method of this case aims to enhance the training efficiency of the Stable Diffusion model to generate images that better meet the prompt requirements. This method significantly improves the model's compliance with the prompt by alternating different diffusion steps and prompt optimization strategies during the training phase, effectively reduces the ambiguity and uncertainty of the generated image, and thus enhances the practicality and reliability of the model in the field of accurate image generation. Specifically, this method injects noise into the image data and trains the model to recover the original image from these noises, thereby enhancing the model's robustness to various types of noise and improving its generalization ability in a variety of practical application scenarios. In addition, the image data and the corresponding text data are input into the model together, so that the model can learn and master the intrinsic connection between the image content and the text description, thereby achieving better performance in image generation or text-to-image conversion tasks. Introducing noise during the training process not only enables the model to make accurate predictions under interference conditions, but also has important significance for dealing with various interferences and non-ideal conditions that may be encountered in practical applications. By comparing the predicted noise with the target noise, the performance of the model can be evaluated more accurately, thereby achieving a more targeted optimization of the model training process. This method is highly adaptable and can be applied to different data sets and task types. It does not rely on a specific data distribution, but instead learns how to extract and recover information from noise, thereby comprehensively improving the performance of the model. This flexibility and efficiency make the interleaved diffusion training method proposed in this patent have broad application prospects in the field of image generation.

[0005] In some embodiments, the noise adding method adopts the following steps: In time When the noise-fused image data is multiplied by an attenuation factor, the original image data and the first noise Multiply by another factor Add together to get; in, represents the attenuation coefficient, as time goes by As time goes by, it gradually decreases.

[0006] In the above technical solution, compared with the traditional noise addition method, this method gradually incorporates noise into the data by introducing a time-dependent attenuation factor. This process simulates the natural accumulation of noise in the physical world, which greatly promotes the model's in-depth learning of the underlying structure and noise pattern of the data. Through the attenuation coefficient, the noise level can be adjusted at different time points to provide the model with all-round adaptive training from mild to severe noise environments. Compared with those methods that only rely on simple forward and reverse processes, the gradual denoising strategy of this method can generate higher quality images, ensuring the clarity and realism of the generated results. In addition, through this progressive denoising method, the model can handle data distribution more smoothly, effectively reducing the risk of overfitting of the discriminator, thereby improving the inference speed and computational efficiency of the model.

[0007] In some embodiments, the image data With picture data The following conditions are met:

[0008]

[0009] in, represents the attenuation coefficient, Represents the target noise.

[0010] In the above technical solution, The reconstruction goal is still , that is, the above relationship must hold. The relationship ensures that the reconstructed target is consistent with the original image data. If the relationship does not hold, the reconstructed image may be significantly different from the original image, resulting in reconstruction failure.

[0011] According to another aspect of the present invention, there is provided a training device for the compliance of generated content with prompt words by staggered diffusion optimization, comprising: Acquisition module: used to obtain image data from a data set and image data And the text data corresponding to the image data and text data ; The first noise processing module is used to process the image data Fusion First Noise Get image data , and the fused image data With text data Input the diffusion generation network model to obtain the first predicted noise; restore the image data based on the first predicted noise , get the image data ; Analyze the image data With picture data , obtain the target noise; The second noise processing module is used to process the image data Fusion First Noise Get image data , and the image data With text data Input the diffusion generation network model again to obtain the second prediction noise; Training module: used to train the neural network model based on the target noise and the second predicted noise.

[0012] In the above technical solution, in order to better use the above method, the present application proposes a training device for the compliance degree of the generated content with the prompt words by staggered diffusion optimization. Each module corresponds to each step of the above method. The specific principle has been described above and will not be repeated here.

[0013] According to another aspect of the present invention, a stable diffusion model is provided, which is trained based on the above training method.

[0014] In the above technical solution, the model relies on the training method, and the model obtained based on the training method improves the prompt word compliance of the model, reduces the ambiguity and uncertainty of the generated image, and thus improves the practicality and reliability in the application of accurate image generation. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.

[0015] According to another aspect of the present invention, there is provided a training device for the compliance of content generated by staggered diffusion optimization with prompt words, comprising: at least one processor and a memory in communication connection with the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above method.

[0016] In the above technical solution, in order to better run and process the method, the above method is stored in a memory, and the stored method is executed by a processor. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.

[0017] According to the last aspect of the present invention, there is provided a computer-readable storage medium storing a computer program, wherein the computer program implements the above method when executed by a processor.

[0018] In the above technical solution, in order to better run and use the method, the above method is stored in a computer-readable storage medium, and the above method is implemented by a processor. It should be noted that the principle and effect of each step have been described above and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flow chart of an embodiment of a method for training the compliance degree of content generated by staggered diffusion optimization to prompt words according to the present invention; Figure 2 It is a training framework diagram of an embodiment of a training method for the compliance degree of generated content with prompt words by staggered diffusion optimization according to the present invention; Figure 3 It is a structural schematic diagram of an embodiment of a training device for the compliance degree of content generated by staggered diffusion optimization to prompt words in the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present invention, but are not intended to limit the scope of the present invention. Similarly, the following examples are only partial embodiments of the present invention rather than all embodiments, and all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0022] The present invention provides a training method for the compliance of generated content with prompt words by staggered diffusion optimization, which can strengthen the training process of Stable Diffusion, so that the generated image results are closer to the prompt words. By staggered use of different diffusion steps and prompt word optimization during the training process, the prompt word compliance of the model is improved, the ambiguity and uncertainty of the generated image are reduced, thereby improving the practicality and reliability in the application of accurate image generation.

[0023] Embodiment 1 See also Figure 1 A training method for the compliance degree of generated content with prompt words by staggered diffusion optimization is provided, the method comprising the following steps: S1. Get image data from a dataset and image data And the text data corresponding to the image data and text data ; S2, the picture data Fusion First Noise (The first noise is a randomly generated noise) Get image data , and the fused image data With text data Input the diffusion generative network model (text + noise, generating an image is the standard process of diffusion, the text is to constrain and specify the generated content to simulate deviation so that the training process can learn a "ability to get back on track", which is a setting scheme that the standard training method does not have), and obtain the first prediction noise (the first prediction noise is The noise predicted by the network at the time of input); restore the image data based on the first predicted noise , get the image data ; Analyze the image data With picture data , obtain the target noise (the target noise is the calculated final learning target); this step simulates the deviation in the standard process and accurately defines the new learning target required to correct the deviation, so that the network can master the ability to automatically correct the deviation S3, the picture data Fusion First Noise Get image data , and the image data With text data Input the diffusion generation network model again to obtain the second prediction noise (the second prediction noise is the noise predicted by the network when At* is input); S4. Training a neural network model based on the target noise and the second predicted noise.

[0024] In this embodiment, the interlaced diffusion training method of this case is intended to enhance the training efficiency of the Stable Diffusion model to generate images that better meet the prompt word requirements. This method significantly improves the model's compliance with the prompt word by alternating different diffusion steps and prompt word optimization strategies during the training phase, effectively reduces the fuzziness and uncertainty of the generated image, and thus enhances the practicality and reliability of the model in the field of accurate image generation. Specifically, this method injects noise into the image data and trains the model to recover the original image from these noises, thereby enhancing the robustness of the model to various types of noise and improving its generalization ability in a variety of practical application scenarios. In addition, the image data and the corresponding text data are input into the model together, so that the model can learn and master the intrinsic connection between the image content and the text description, thereby achieving better performance in the image generation or text-to-image conversion tasks. Introducing noise during the training process not only enables the model to make accurate predictions under interference conditions, but also has important significance for dealing with various interferences and non-ideal conditions that may be encountered in practical applications. By comparing the predicted noise with the target noise, the performance of the model can be evaluated more accurately, thereby achieving a more targeted optimization of the model training process. This method is highly adaptable and can be applied to different data sets and task types. It does not rely on a specific data distribution, but instead learns how to extract and recover information from noise, thereby comprehensively improving the performance of the model. This flexibility and efficiency make the interleaved diffusion training method proposed in this patent have broad application prospects in the field of image generation.

[0025] In this embodiment, the noise adding method adopts the following steps: In time When the noise-fused image data is multiplied by an attenuation factor, the original image data and the first noise Multiply by another factor Add together to get; in, represents the attenuation coefficient, as time goes by As time goes by, it gradually decreases.

[0026] In this embodiment, the standard noise adding method is a fusion of random noise and graphics, which is used as input, and the network is required to predict this "random noise added to the photo". The method of this case redefines the input, which simulates the deviation from the right track. Furthermore, this case redefines the prediction target, no longer predicting the "random noise added to the picture", but predicting the specific noise that can correct the deviation. The solution formula for the specific noise is derived in detail below and will not be repeated here. Compared with the traditional noise adding method, this method gradually integrates the noise into the data by introducing a time-dependent attenuation factor. This process simulates the natural accumulation of noise in the physical world, which greatly promotes the model's in-depth learning of the underlying structure and noise pattern of the data. Through the attenuation coefficient, the noise level can be adjusted at different time points, providing the model with all-round adaptive training from mild to severe noise environments. Compared with those methods that only rely on simple forward and reverse processes, the step-by-step denoising strategy of this method can generate higher quality images, ensuring the clarity and realism of the generated results. In addition, through this progressive denoising method, the model can handle data distribution more smoothly, effectively reducing the risk of discriminator overfitting, thereby improving the model's reasoning speed and computational efficiency.

[0027] In this embodiment, the picture data With picture data The following conditions are met:

[0028]

[0029] in, represents the attenuation coefficient, Represents the target noise.

[0030] In this embodiment, The reconstruction goal is still , that is, the above relationship must hold. The relationship ensures that the reconstructed target is consistent with the original image data. If the relationship does not hold, the reconstructed image may be significantly different from the original image, resulting in reconstruction failure.

[0031] In order to better explain the present invention, the following will be further described. Figure 2 , the steps are as follows: 1. Extract unequal data A and B from data set Z, and For pictures, , To describe A0 and 's text.

[0032]

[0033]

[0034] 2. By adding noise, get

[0035]

[0036] 3. Neural Networks In text condition Next recover , simulating the situation when deviating from the correct path

[0037] 4. Then from Adding noise gives

[0038]

[0039] 5. The reconstruction goal is still , that is, it must be established

[0040]

[0041] in, represents the attenuation coefficient, Represents the target noise.

[0042] 6. This changes the training objectives of the neural network net:

[0043]

[0044]

[0045]

[0046] in, For learnable parameters loss of supervision; Calculate the MSE loss between the network prediction and the convergence target; is the diffuse noise of At; is the target noise; is the diffusion time step; are learnable parameters; the new learning objective constructs a distribution deviation (using Condition as a reconstruction guide), the neural network net must be based on Following the corrective path, the retrained model will be significantly more consistent with prompt guidance than the baseline.

[0047] Embodiment 2 See also Figure 3 , a training device for generating content to comply with prompt words by staggered diffusion optimization, comprising: Acquisition module: used to obtain image data from a data set and image data And the text data corresponding to the image data and text data ; The first noise processing module is used to process the image data Fusion First Noise Get image data , and the fused image data With text data Input the diffusion generation network model to obtain the first predicted noise; restore the image data based on the first predicted noise , get the image data ; Analyze the image data With picture data , obtain the target noise; The second noise processing module is used to process the image data Fusion First Noise Get image data , and the image data With text data Input the diffusion generation network model again to obtain the second prediction noise; Training module: used to train the neural network model based on the target noise and the second predicted noise.

[0048] In this embodiment, in order to better use the above method, the present application proposes a training device for the compliance degree of the generated content with the prompt words through staggered diffusion optimization. Each module corresponds to each step of the above method. The specific principle has been described above and will not be repeated here.

[0049] Embodiment 3 A stable diffusion model is obtained by training based on the training method described in one of the embodiments.

[0050] In the above technical solution, the model relies on the training method, and the model obtained based on the training method improves the prompt word compliance of the model, reduces the ambiguity and uncertainty of the generated image, and thus improves the practicality and reliability in the application of accurate image generation. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.

[0051] Embodiment 4 A training device for the compliance degree of content generated by staggered diffusion optimization to prompt words, comprising: at least one processor and a memory connected to the at least one processor in communication; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in one of the embodiments.

[0052] In this embodiment, in order to better run and process the method described in one of the embodiments, the above method is stored in a memory, and a processor is used to execute the stored method. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.

[0053] Embodiment 5 A computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0054] In this embodiment, in order to better run and use the method described in one of the embodiments, the method is stored in a computer-readable storage medium and implemented by a processor. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.

[0055] The above descriptions are only some embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent device or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A training method for the compliance of generated content with prompt words by staggered diffusion optimization, characterized in that: The method comprises the following steps: Get image data from a dataset and image data And the text data corresponding to the image data and text data ; The image data Fusion First Noise Get image data , and the fused image data With text data Input the diffusion generation network model to obtain the first predicted noise; restore the image data based on the first predicted noise , get the image data ; Analyze the image data With picture data , obtain the target noise; The image data Fusion First Noise Get image data , and the image data With text data Input the diffusion generation network model again to obtain the second prediction noise; The neural network model is trained based on the target noise and the second predicted noise.

2. A training method for the compliance of content to prompt words by staggered diffusion optimization as claimed in claim 1, characterized in that: The noise adding method adopts the following steps: In time When the noise-fused image data is multiplied by an attenuation factor, the original image data and the first noise Multiply by another factor Add together to get; in, represents the attenuation coefficient, as time goes by As time goes by, it gradually decreases.

3. A training method for the compliance of content to prompt words by staggered diffusion optimization as claimed in claim 1, characterized in that: The picture data With picture data The following conditions are met: in, represents the attenuation coefficient, Represents the target noise.

4. A training device for the degree of compliance of generated content with prompt words by staggered diffusion optimization, characterized in that: include: Acquisition module: used to obtain image data from a data set and image data And the text data corresponding to the image data and text data ; The first noise processing module is used to process the image data Fusion First Noise Get image data , and the fused image data With text data Input the diffusion generation network model to obtain the first predicted noise; restore the image data based on the first predicted noise , get the image data ; Analyze the image data With picture data , obtain the target noise; The second noise processing module is used to process the image data Fusion First Noise Get image data , and the image data With text data Input the diffusion generation network model again to obtain the second prediction noise; Training module: used to train the neural network model based on the target noise and the second predicted noise.

5. A training device for the compliance of content generated by staggered diffusion optimization to prompt words as claimed in claim 4, characterized in that: The noise adding method adopts the following steps: In time When the noise-fused image data is multiplied by an attenuation factor, the original image data and the first noise Multiply by another factor Add together to get; in, represents the attenuation coefficient, as time goes by As time goes by, it gradually decreases.

6. A training device for the compliance of content generated by staggered diffusion optimization to prompt words as claimed in claim 4, characterized in that: The picture data With picture data The following conditions are met: in, represents the attenuation coefficient, Represents the target noise.

7. A stable diffusion model, characterized in that: The method is obtained by training based on the training method described in any one of claims 1 to 3.

8. A training device for the compliance of generated content with prompt words by staggered diffusion optimization, characterized in that: include: at least one processor and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as claimed in any one of claims 1 to 3.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images