Deep learning-based text map diffusion model training method

By performing stage pre-training and joint training of the literary and biographic diffusion model, combining the BERT model and the generative adversarial network, the problem of insufficient accuracy of the existing Chinese biographic diffusion model in noise reduction processing and text and picture alignment scores is solved, and high quality is achieved image generation and model training efficiency improvement.

CN119048615AActive Publication Date: 2024-11-29HUAZHONG UNIV OF SCI & TECH +1

Patent Information

Application Number
CN202410930748.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-11
Publication Date
2024-11-29
Estimated Expiration
2044-07-11

AI Technical Summary

Technical Problem

The existing diffusion model of literature and art image has insufficient accuracy in noise reduction processing and text image alignment scores, resulting in poor quality of generated images, poor learning effect of the model on complex samples, and lack of flexibility in parameter adjustment.

Method used

Using the deep learning-based literary graph diffusion model training method, the text encoder and image generator are pre-trained separately, combined with joint training data for multi-step iterative training, and high-quality semantic vectors are extracted using BERT model and generative adversarial network. , and dynamically adjust the learning rate through the Adam optimizer and the learning rate scheduler until the model meets the expected standards.

Benefits of technology

It significantly improves the performance of the model in the initial stage, improves the semantic consistency and clarity of image generation, enhances image details and texture, ensures the quality of generated images, and improves training efficiency and comprehensive capabilities of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048615A_ABST
    Figure CN119048615A_ABST
Patent Text Reader

Abstract

The invention relates to a deep learning-based text map diffusion model training method, and relates to the technical field of artificial intelligence. The deep learning-based text map diffusion model training method comprises the steps of obtaining text training data, image training data and joint training data; based on the text training data and the image training data, stage pre-training is carried out on the text image diffusion model, wherein the stage pre-training comprises text stage pre-training and image stage pre-training; and carrying out joint training on the text map diffusion model after stage pre-training based on the joint training data, and evaluating and adjusting each joint training result until the text map diffusion model meets an expected standard. According to the method, through multi-stage pre-training, combined training data is utilized, the BERT model and the generative adversarial network are combined, the diffusion process is introduced to gradually remove noise and dynamically adjust the learning rate, and the training precision of the diffusion model of the text graph and the quality of the generated image are remarkably improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a training method for a Vincent graph diffusion model based on deep learning. Background Art

[0002] In the field of image generation, methods based on deep learning have made significant progress in recent years. Text-to-Image generation technology is a technology that generates corresponding images by inputting text descriptions. This technology has important value in many application fields, such as automatic image generation, content creation, and virtual reality. However, the existing text-to-image generation technology still faces many challenges in practical applications. Therefore, a training method for the text-to-image diffusion model based on deep learning has emerged.

[0003] The prior art, such as the invention patent application with announcement number: CN116051668B, discloses a training method for a Vincent graph diffusion model and a text-based image generation method, wherein the steps are as follows: the Vincent graph diffusion model performs denoising on a noisy sample image according to a sample text to generate a denoised sample image; a first text-image alignment score is obtained according to a first representation vector of the denoised sample image and a second representation vector of the sample text, and a first training sample is selected from a current batch of training samples based on the first text-image alignment score; a first loss function of the Vincent graph diffusion model is determined according to an original sample image and a denoised sample image of the sample text in the first training sample, and the Vincent graph diffusion model is adjusted based on the first loss function; training is continued using the next batch of training samples until the target Vincent graph diffusion model is obtained after the training is completed, thereby improving the training accuracy of the Vincent graph diffusion model.

[0004] Based on the above solution, it is found that the limitations of the existing technology include at least the following problems. First, there are deficiencies in the noise reduction process, which easily leads to poor quality of generated images and inaccurate text-image alignment scores, affecting the training effect of the model. At the same time, the screening of difficult samples is insufficient, which easily leads to poor learning effect of the model on complex samples, and the adjustment of model parameters lacks flexibility and effectiveness, affecting the final performance. Summary of the invention

[0005] In view of the technical problems existing in the prior art, the present invention provides a training method for a Wensheng graph diffusion model based on deep learning, which solves the problems.

[0006] The technical solution of the present invention to solve the above technical problems is as follows: a training method for a deep learning-based vincent graph diffusion model, comprising the following steps: obtaining text training data, image training data and joint training data; performing stage pre-training on the vincent graph diffusion model based on the text training data and the image training data, including text stage pre-training and image stage pre-training; performing joint training on the vincent graph diffusion model after stage pre-training based on the joint training data, and evaluating and adjusting the results of each joint training until the vincent graph diffusion model meets the expected standards.

[0007] Furthermore, the text training data specifically includes a number of text information and corresponding text semantic vectors for training the text encoder in the text-graph diffusion model, the image training data specifically includes a number of text semantic vectors and corresponding text semantic image data for training the image generator in the text-graph diffusion model, and the joint training data specifically includes a number of text information and corresponding text image data.

[0008] Furthermore, the text encoder in the text graph diffusion model is specifically a BERT model, and the specific process of pre-training the text phase of the text graph diffusion model based on the text training data is as follows: reading a number of text information used to train the text encoder in the text graph diffusion model and pre-processing; establishing a text information data set based on the pre-processed text information and the corresponding text semantic vectors, and dividing it into a text information training set and a text information verification set; dividing the text information training set into a number of batch training sets, and each batch training set contains a number of text information; performing forward propagation processing on each batch training set in the BERT model to obtain the CLS vector of each text information in each batch training set, and calculating the text loss function; iteratively training the BERT model, evaluating each training result of the BERT model based on the back propagation algorithm and the text information verification set, and adjusting the model parameters according to the verification results until the CLS vector of the text information predicted by the model meets the corresponding text semantic vector.

[0009] Furthermore, the text loss function is specifically calculated using the cosine similarity formula, and its specific formula is as follows: Among them, L is the text loss function, CLS ij is the CLS vector of the jth text information in the i-th batch training set, WY ij is the text semantic vector corresponding to the jth text information in the i-th batch training set, i = 1, 2, 3, ..., N, N is the number of divided batch training sets, j = 1, 2, 3, ..., M, N is the number of text information in the batch training set.

[0010] Furthermore, the text encoder in the text-graph diffusion model is specifically a generative adversarial network, which includes a generator and a discriminator. The specific process of pre-training the text-graph diffusion model at the image stage based on the image training data is as follows: initializing the generative adversarial network; reading a number of text semantic vectors and corresponding text semantic image data used to train the image generator in the text-graph diffusion model, and establishing an image data set, which is divided into an image data training set and an image data verification set; performing cyclic training on the generative adversarial network based on the image data training set, specifically, the generator in the generative adversarial network receives each text semantic vector and generates a corresponding text image; performing discriminative training on the discriminator in the generative adversarial network based on the text image and the corresponding text semantic image; performing adversarial training on the generator and discriminator in the generative adversarial network, and calculating the generator loss function and the discriminator loss function respectively; performing performance evaluation on each training result of the generative adversarial network based on the image data verification set, and adjusting the model parameters according to the verification results until the text image predicted by the generative adversarial network meets the corresponding text semantic image.

[0011] Furthermore, the specific formulas for calculating the generator loss function and the discriminator loss function are as follows: Among them, S is the generator loss function, WY u is the u-th text semantic vector received by the generator in the generative adversarial network, G(WY u ) is the text image generated by the u-th text semantic vector, ZS u is the text semantic image corresponding to the u-th text semantic vector received by the generator in the generative adversarial network, P is the discriminator loss function, and y u is the image label of the u-th text semantic vector received by the discriminator in the generative adversarial network, u = 1, 2, 3, ..., V, V is the number of text semantic vectors obtained for training the image generator in the text-image diffusion model.

[0012] Furthermore, the text-to-image diffusion model after stage pre-training is jointly trained based on the joint training data, and the results of each joint training are evaluated and adjusted until the text-to-image diffusion model meets the expected standards. Specifically: read a number of text information and corresponding text image data, and merge them into a joint dataset; divide the joint dataset into a joint training set and a joint validation set; load the pre-trained BERT model and generative adversarial network, and perform batch training on the text-to-image diffusion model based on the joint training set, and calculate the diffusion loss function; perform backpropagation on the diffusion loss function and calculate the gradient; use the Adam optimizer to optimize the model parameters according to the calculated gradient, and after a set number of training epochs, use the joint validation set to evaluate the model and adjust the learning rate based on the evaluation results; repeat the training process until the text-to-image diffusion model meets the expected standards.

[0013] Furthermore, the specific process of performing batch training on the text-to-image diffusion model based on the joint training set is as follows: for each batch, process the text information based on the BERT model to generate high-dimensional semantic vectors; generate a random noise image based on the generative adversarial network, and this noise image serves as the starting point of the diffusion model; use the text semantic vectors as conditions, input the text semantic vectors into the generator of the generative adversarial network to gradually remove the noise, and the generator of the generative adversarial network repeatedly applies the conditional noise removal process.

[0014] Furthermore, the specific formula for calculating the diffusion loss function is as follows: where Ks is the diffusion loss function, q(x t |x0) is the conditional probability distribution of the forward process, p(x t |x t-1 ) is the conditional probability distribution of the reverse process, x0 is the initial image data, t is the time step, x t is the image data at the t-th time step, x t-1 is the image data at the (t - 1)-th time step.

[0015] Furthermore, the specific process of using the Adam optimizer to optimize the model parameters according to the calculated gradient, and after a set number of training epochs, using the joint validation set to evaluate the model and adjusting the learning rate based on the evaluation results is as follows: initialize the Adam optimizer and obtain the preset initial learning rate and hyperparameters; after each training batch, adjust the value of each parameter according to the gradient information calculated by backpropagation to minimize the diffusion loss function; after the end of the set number of training epochs, use the joint validation set to evaluate the model and calculate the quality metrics of image generation; dynamically adjust the learning rate based on the validation results.

[0016] The beneficial effects of the present invention are as follows: by pre-training the text encoder and the image generator in stages respectively, the performance of the model in the initial stage is significantly improved; by merging text data and image data into a joint data set for training, the comprehensive ability of the model in processing multimodal data is improved, so that the generated image is more semantically consistent with the text description; the BERT model is used to extract high-quality text semantic vectors, and the noise is gradually removed in a multi-step iterative process in combination with a generative adversarial network to generate high-quality images, thereby improving the clarity of the image and the consistency with the text description; a diffusion model is introduced in the image generation process, and the noise is gradually removed through multi-step iterations, which not only enhances the image details and texture, but also ensures the quality of the generated image; by using an Adam optimizer and a learning rate scheduler, the learning rate is dynamically adjusted according to the verification result, so as to ensure that the model can be continuously optimized during the training process, thereby improving the training efficiency and the quality of the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of a training method for a deep learning-based Wensheng graph diffusion model of the present invention.

[0018] Figure 2 The present invention is a flowchart of the specific steps of performing text stage pre-training on a text graph diffusion model based on text training data in a training method of a text graph diffusion model based on deep learning. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0020] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0021] In the description of the present application, the term "for example" is used to mean "used as an example, illustration or description". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid unnecessary details to obscure the description of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0022] The overall idea of ​​the problem in the embodiment of this application is as follows:

[0023] First, obtain text training data, image training data and joint training data, preprocess these data to generate text semantic vectors and image feature vectors. Secondly, based on the text training data, the text encoder, namely the BERT model, is pre-trained at the text stage, and based on the image training data, the image generator, namely the generative adversarial network, is pre-trained at the image stage. Secondly, the preprocessed text data and image data are merged into a joint data set and divided into a joint training set and a joint validation set. The pre-trained BERT model and the generative adversarial network are loaded, batch training is performed based on the joint training set, the diffusion loss function is calculated, the diffusion loss function is back-propagated, the gradient is calculated, and the Adam optimizer is used for parameter optimization. Finally, after the set training cycle, the model is evaluated using the joint validation set, the learning rate is dynamically adjusted according to the evaluation results, and the training process is repeated until the text-image diffusion model meets the expected standards.

[0024] See also Figure 1 The embodiment of the present invention provides a technical solution: a training method for a deep learning-based vincent graph diffusion model, comprising the following steps: obtaining text training data, image training data, and joint training data; performing stage pre-training on the vincent graph diffusion model based on the text training data and the image training data, including text stage pre-training and image stage pre-training; performing joint training on the vincent graph diffusion model after stage pre-training based on the joint training data, and evaluating and adjusting each joint training result until the vincent graph diffusion model meets the expected standard.

[0025] The text training data specifically refers to a number of text information and corresponding text semantic vectors used to train the text encoder in the text-image diffusion model. The image feature vector specifically refers to the feature vector of the real image generated corresponding to the text information. The image training data specifically refers to a number of text semantic vectors and corresponding text semantic image data used to train the image generator in the text-image diffusion model. The joint training data specifically refers to a number of text information and corresponding text image data.

[0026] Specifically, Figure 2 As shown in the figure, the text encoder in the text-graph diffusion model is specifically a BERT model, and the specific process of pre-training the text stage of the text-graph diffusion model based on the text training data is as follows: a number of text information used to train the text encoder in the text-graph diffusion model is read and pre-processed; a text information data set is established based on the pre-processed text information and the corresponding text semantic vectors, and is divided into a text information training set and a text information verification set; the text information training set is divided into a number of batch training sets, and each batch training set contains a number of text information; each batch training set is forward propagated in the BERT model to obtain the CLS vector of each text information in each batch training set, and the text loss function is calculated; the BERT model is iteratively trained, and each training result of the BERT model is evaluated based on the back propagation algorithm and the text information verification set, specifically minimizing the text loss function, and adjusting the model parameters according to the verification results until the CLS vector of the text information predicted by the model meets the corresponding text semantic vector.

[0027] The text loss function is calculated using the cosine similarity formula, which is as follows: Among them, L is the text loss function, CLS ij is the CLS vector of the jth text information in the i-th batch training set, WY ij is the text semantic vector corresponding to the jth text information in the i-th batch training set, |CLS ij | is the modulus of the CLS vector of the jth text information in the i-th batch training set, |WY ij | is the modulus of the text semantic vector corresponding to the jth text information in the i-th batch training set, i = 1, 2, 3, ..., N, N is the number of divided batch training sets, j = 1, 2, 3, ..., M, N is the number of text information in the batch training set.

[0028] In this implementation scheme, a number of text messages used to train the text encoder in the text graph diffusion model are read and preprocessed. A text information dataset is established based on the preprocessed text messages and the corresponding text semantic vectors, and is divided into a text information training set and a text information verification set. The preprocessing and dataset construction ensure the quality and consistency of the data and improve the reliability of model training. The generation of text semantic vectors enables text information to be efficiently input into the model for training. By systematically constructing training and verification datasets, stable basic data is provided to ensure the effectiveness of the model training process and the accuracy of the evaluation. The text information training set is divided into a number of batch training sets, and each batch training set contains a number of text messages. Batch training can improve training efficiency and ensure that a stable gradient is obtained each time the model parameters are updated during training. Degree estimation, batch training helps to accelerate model training and reduce memory consumption. Through batch division, the memory limitation problem in large-scale data training is solved, and the efficiency and stability of model training are improved. In the BERT model, each batch training set is forward propagated to obtain the CLS vector of each text information in each batch training set, and the text loss function is calculated. Forward propagation can extract the deep semantic features of each text information. The CLS vector captures the global information of the text. The calculation of the text loss function ensures that the training objectives of the model are clear and the optimization direction is clear. Through forward propagation and CLS vector calculation, the model's ability to understand text semantics is enhanced, and the accuracy and consistency of text-to-image generation are improved. The BERT model is iteratively trained, and the training results of each BERT model are evaluated based on the back propagation algorithm and the text information verification set. Specifically, the text loss function is minimized, and the model parameters are adjusted according to the verification results until the CLS vector of the text information predicted by the model meets the corresponding text semantic vector. Iterative training and back propagation ensure that the model can be continuously optimized and the training accuracy is gradually improved. The evaluation and parameter adjustment based on the verification set ensure the generalization ability and practical application effect of the model. Through iterative training and verification, the model can continuously learn and optimize, gradually improve the quality of the generated image and the consistency with the text semantics, and dynamically adjust the parameters so that the model can maintain the best performance at different stages, solving the overfitting or underfitting problems that may occur during the model training process. The text loss function is specifically calculated using the cosine similarity formula, which can effectively measure the similarity between text vectors and improve the consistency of text to image generation. Cosine similarity can eliminate the influence of vector size, focus on the similarity of direction, and enhance the model's ability to capture semantic information. Through the use of the cosine similarity loss function, the accuracy of text encoding and the quality of generated images are significantly improved, which solves the problem of inaccurate loss functions in the prior art and improves the training and generation effects of the model.

[0029] Specifically, the text encoder in the text image diffusion model is a generative adversarial network, which includes a generator and a discriminator. The specific process of pre-training the text image diffusion model at the image stage based on the image training data is as follows: initialize the generative adversarial network, specifically initialize the generator and discriminator of the generative adversarial network, create a network structure of the generator and the discriminator, the generator is responsible for generating images, and the discriminator is responsible for judging whether the image is a real image; read several text semantic vectors and corresponding text semantic image data used to train the image generator in the text image diffusion model, and establish an image data set, and divide it into image data training and training. set and image data verification set; perform cyclic training on the generative adversarial network based on the image data training set, specifically, the generator in the generative adversarial network receives each text semantic vector and generates a corresponding text image; perform discriminant training on the discriminator in the generative adversarial network based on the text image and the corresponding text semantic image; perform adversarial training on the generator and discriminator in the generative adversarial network, and calculate the generator loss function and the discriminator loss function respectively; perform performance evaluation on each training result of the generative adversarial network based on the image data verification set, and adjust the model parameters according to the verification results until the text image predicted by the generative adversarial network meets the corresponding text semantic image.

[0030] The specific formulas for calculating the generator loss function and the discriminator loss function are as follows: Among them, S is the generator loss function, WY u is the u-th text semantic vector received by the generator in the generative adversarial network, G(WY u ) is the text image generated by the u-th text semantic vector, ZS u is the text semantic image corresponding to the u-th text semantic vector received by the generator in the generative adversarial network, D[G(WY u )] is the image G(WY generated by the discriminator to the generator u ) output, specifically, D[G(WY u )] indicates that the discriminator judges the image G(WY u ) is the probability of a “real image”, logD[G(WY u )] represents the logarithm of the discriminator output. The logarithmic function is used to amplify the impact of the output with a probability close to 1, because the logarithmic function is very sensitive to changes when the input is close to 1. P is the discriminator loss function, y u is the image label of the u-th text semantic vector received by the discriminator in the generative adversarial network. For the real image, the label y u is 1, for the generated image, the label is 0, D(ZS u ) and D[G(WY u)] are the judgment probabilities of the discriminator for the real image and the generated image, logD(ZS u ) is the logarithm of the probability that the discriminator correctly identifies the real image as real, log{1-D[G(WY u )]} is the logarithm of the probability that the discriminator correctly identifies the generated image as fake. The negative sign indicates that by minimizing this combined loss, the discriminator can be trained to correctly distinguish between real and fake images. u = 1, 2, 3, ..., V, V is the number of text semantic vectors obtained for training the image generator in the text-image diffusion model.

[0031] In this implementation scheme, the generator and discriminator of the generative adversarial network are initialized, and the network structure of the generator and the discriminator is created. The generator is responsible for generating images, and the discriminator is responsible for judging whether the image is a real image. The initialization process ensures the reasonable design of the generator and discriminator structures, laying the foundation for subsequent training. Clarifying the responsibilities of the generator and the discriminator helps to improve the quality and authenticity of the generated images. By reasonably designing the network structure of the generator and the discriminator, the generation and discrimination capabilities of the model are improved, laying the foundation for the overall performance of the model. Several text semantic vectors and corresponding text semantic image data used to train the image generator in the text-image diffusion model are read to establish an image dataset. , the image data set is divided into an image data training set and an image data verification set. Data reading and data set division ensure the quality and independence of training data and verification data, improve the training effect and evaluation accuracy of the model, and divide the training set and verification set to effectively prevent model overfitting and improve the generalization ability of the model. By reasonably dividing the data set, a stable training and verification environment is provided to ensure that the model can effectively learn and evaluate during the training process. The generative adversarial network is trained in a loop based on the image data training set. Specifically, the generator receives each text semantic vector and generates a corresponding text image. The discriminator is trained based on the generated text image and the corresponding text semantic image. The discriminator determines whether the generated image is a real image. By cyclically training the generator and the discriminator, the quality and authenticity of the generated image are continuously optimized. The adversarial training mechanism can make the images generated by the generator more realistic and improve the generation ability of the model. Through the adversarial training of the generator and the discriminator, the quality of the generated image and the accuracy of the discrimination are improved, and the overall performance of the model is enhanced. The reasonable design of the generator loss function and the discriminator loss function is conducive to the effective training of the model, ensuring the quality of the generated image and the accuracy of the discrimination. Through the optimization of the adversarial loss function, the images generated by the generator can be made more realistic, and the discriminator can have a stronger ability to distinguish between real and generated images. Through reasonable design and optimization of the loss function, the discriminator can make the real image more realistic and the discriminator can distinguish between real and generated images. The function significantly improves the training effect of the generative adversarial network, improves the quality of the generated images and the discrimination ability of the model, performs performance evaluation on each training result of the generative adversarial network based on the image data verification set, and adjusts the model parameters according to the verification results until the text image predicted by the generative adversarial network meets the corresponding text semantic image. The performance evaluation ensures the performance of the model on the verification set, promptly discovers and corrects the problems of the model, and dynamically adjusts the model parameters according to the evaluation results, which can effectively prevent the model from overfitting or underfitting and improve the generalization ability of the model. Through regular evaluation and parameter adjustment, it ensures that the model can maintain the best performance at different stages, which solves the problem of inflexible model parameter adjustment in the existing technology.

[0032] Specifically, the text-to-image diffusion model after stage pre-training is jointly trained based on the joint training data, and the results of each joint training are evaluated and adjusted until the text-to-image diffusion model meets the expected standards. Specifically: read a number of text information and corresponding text image data, and merge them into a joint dataset. Among them, each text semantic vector has corresponding image data, forming a one-to-one correspondence; divide the joint dataset into a joint training set and a joint validation set; load the pre-trained BERT model and generative adversarial network, and perform batch training on the text-to-image diffusion model based on the joint training set, and calculate the diffusion loss function. Each batch contains a set of text semantic vectors and their corresponding real images; perform backpropagation on the diffusion loss function and calculate the gradient. Specifically, the loss is passed from the output layer to the input layer layer by layer, and the gradients of the loss for each layer's parameters, such as weights and biases, are calculated; use the Adam optimizer to optimize the model's parameters according to the calculated gradients, and after a set number of training epochs, use the joint validation set to evaluate the model and adjust the learning rate based on the evaluation results; repeat the training process until the text-to-image diffusion model meets the expected standards.

[0033] The specific formula for calculating the diffusion loss function is as follows: Where Ks is the diffusion loss function, q(x t |x0) is the conditional probability distribution of the forward process, p(x t |x t-1 ) is the conditional probability distribution of the reverse process, denotes the expected value under the distribution of q(x t |x0), which is used to measure the difference between the predicted reverse process of the model and the actual forward process. x0 is the initial image data, which is the noiseless image at the beginning in the diffusion process and is the starting point of the diffusion process. t is the time step, x t is the image data at the t-th time step, which represents the noisy image at the t-th step in the diffusion process. As t increases, the noise gradually increases in the image. x t-1 is the image data at the (t - 1)-th time step, which represents the noisy image at the (t - 1)-th step and is used to predict the image at the t-th step.

[0034] In this implementation scheme, a number of text information and its corresponding text image data are read, and the read text information and its corresponding image data are merged into a joint data set. Each text semantic vector forms a one-to-one correspondence with its corresponding image data. By ensuring that the text semantic vector forms a one-to-one correspondence with the corresponding image data, multimodal data can be fully utilized to improve the integrity and consistency of training data. This process provides basic data for subsequent model training, ensures the quality and relevance of the data, and provides a comprehensive joint data set, so that the model can simultaneously learn the relationship between text and image, improve the generation ability and consistency of the model, and divide the joint data set into a joint training set. And the joint validation set. By dividing the training set and the validation set, they can be used for model training and performance evaluation respectively to prevent overfitting and improve the generalization ability of the model. The training set is used to update the model parameters, and the validation set is used to monitor the model performance and adjust the training strategy to ensure that the model can obtain stable training data during the training process. At the same time, the model performance is evaluated through the validation set, the training process is optimized, and the generalization ability of the model is improved. The pre-trained BERT model and the generative adversarial network are loaded, and the text-graph diffusion model is batch trained based on the joint training set. Each batch contains a set of text semantic vectors and their corresponding real images. Through batch training, the training efficiency and stability can be improved to ensure that each parameter update All have reliable gradient estimates. The diffusion loss function measures the difference between the reverse process predicted by the model and the actual forward process, so that the model can more accurately recover clear images from noisy data, improving the training efficiency and stability of the model. By optimizing the diffusion loss function, the images generated by the model are clearer and more accurate. The diffusion loss function is back-propagated and the gradient is calculated. The loss is transferred from the output layer to the input layer layer by layer, and the parameters of each layer, such as weights and biases, are calculated. The gradient of the loss. The back-propagation algorithm enables the model to continuously optimize the parameters according to the feedback of the loss function, improve the training effect, and calculate the gradient accurately, ensuring that the direction and amplitude of the model parameter update are appropriate, improving the training efficiency, and through Back propagation and gradient calculation, the model can effectively learn and adjust parameters, gradually improve the quality of generated images and consistency with text semantics, use Adam optimizer to optimize model parameters according to the calculated gradient, initialize Adam optimizer, and obtain pre-set initial learning rate and hyperparameters. After each training batch, adjust the value of each parameter according to the gradient information calculated by back propagation to minimize the diffusion loss function. After the set training cycle, use the joint validation set to evaluate the model, and adjust the learning rate based on the evaluation results. If the performance on the validation set does not improve within multiple evaluation cycles, it may be necessary to reduce the learning rate. Use the learning rate scheduler to adjust the learning rate according to evaluation indicators such as F ID score. The Adam optimizer dynamically adjusts the learning rate of each parameter to improve the stability and efficiency of training. Dynamically adjusting the learning rate based on the verification results can prevent overfitting or underfitting.Ensure that the model can maintain optimal performance at different stages. By using the Adam optimizer and learning rate scheduler, the model can efficiently optimize parameters, improve training results and the quality of generated images.

[0035] Specifically, the specific process of batch training the text-graph diffusion model based on the joint training set is as follows: for each batch, the text information is processed based on the BERT model to generate a high-dimensional semantic vector; a random noise image is generated based on the generative adversarial network, and this noise image is used as the starting point of the diffusion model. This image contains a high level of random noise and has no meaningful visual content; using the text semantic vector as a condition, the text semantic vector is input into the generator of the generative adversarial network to gradually remove the noise. This step is iterative, and each iteration uses the output of the previous iteration as input to refine the image details and reduce noise, and the generator of the generative adversarial network repeatedly applies the conditional noise removal process. Each step further reduces the noise in the image on the basis of the previous step and enhances the visual details that match the text description. After each iteration, the clarity and quality of the image are gradually improved, which is closer to the actual image appearance. As the diffusion step proceeds, not only the noise is removed, but the details and textures in the image are also gradually enhanced, so that the final generated image is visually highly consistent with the text content.

[0036] In this implementation scheme, for each batch, text information is processed based on the BERT model to generate a high-dimensional semantic vector. By using the BERT model to process text information, the deep semantic features of the text can be captured and high-quality semantic vectors can be generated. The high-dimensional semantic vector provides rich text information, which helps to better match the text description when generating images, improves the representation ability of text information, and enables the model to more accurately understand and generate images that match the text semantics. A random noise image is generated based on the generative adversarial network. This noise image serves as the starting point of the diffusion model. This image contains a high level of random noise and does not have any meaningful visual content. By generating a high-level random noise image, it is ensured that the diffusion model has a unified starting point, which is convenient for the subsequent noise removal process. The random noise image provides an initial noise environment, so that the diffusion process can gradually optimize the quality of the generated image. The unified noise starting point improves the stability of the model training and ensures the consistency and controllability of each training process. Using the text semantic vector as a condition, the text semantic vector is input into the generator of the generative adversarial network to gradually remove the noise. This step is Iteratively, each iteration uses the output of the previous iteration as input to refine image details and reduce noise. The iterative denoising process can gradually improve the clarity and quality of the image, making the generated image more realistic and detailed. The refinement process of each iteration ensures that the generated image gradually approaches the actual image appearance and enhances the matching degree with the text description. Through the iterative denoising process, the resolution and visual quality of the generated image are improved, so that the model can generate higher quality images. The generator of the generative adversarial network repeatedly applies the conditional noise removal process. Each step further reduces the noise in the image on the basis of the previous step and enhances the visual details that match the text description. As the diffusion step proceeds, not only the noise is removed, but the details and textures in the image are also gradually enhanced. Repeated application of the conditional noise removal process can continuously enhance the image details and textures, so that the final generated image is visually highly consistent with the text content. This process can gradually optimize the image quality, so that the generated image is not only high in clarity, but also rich in details. By continuously enhancing the image details and textures, the quality and realism of the model-generated images are improved, solving the problem of low image quality in the prior art.

[0037] Specifically, the Adam optimizer is used to optimize the model parameters according to the calculated gradient, and after a set training cycle, the joint validation set is used to evaluate the model, and the specific process of adjusting the learning rate based on the evaluation results is as follows: initialize the Adam optimizer and obtain the pre-set initial learning rate and hyperparameters; after each training batch, adjust the value of each parameter according to the gradient information calculated by back propagation to minimize the diffusion loss function; after the set training cycle, such as each epoch, the joint validation set is used to evaluate the model and calculate the quality indicator FID of image generation; dynamically adjust the learning rate based on the verification results, specifically: if the performance on the validation set has not improved within multiple evaluation cycles, it may be necessary to reduce the learning rate. Use the learning rate scheduler to adjust the learning rate according to the evaluation indicators, such as the FID score, to prevent overfitting or underfitting. By reducing the learning rate, the model can adjust the parameters more finely and improve the quality of the generated image.

[0038] In this implementation scheme, the Adam optimizer is used, and a preset initial learning rate and hyperparameters are obtained. The Adam optimizer combines the advantages of Momentum and RMSprop, and can adaptively adjust the learning rate during the training process, effectively avoiding the problems of gradient disappearance and gradient explosion. The preset initial learning rate and hyperparameters make the optimization process more stable and efficient. By using the Adam optimizer, the optimization process is more efficient, the complexity of hyperparameter adjustment is reduced, and the convergence speed and stability of the model are improved. After each training batch, back propagation is used to calculate the gradient information, and the value of each parameter is adjusted according to the calculated gradient information to minimize the diffusion loss function. The back propagation algorithm enables the model to accurately calculate the gradient of each parameter. The optimizer can adjust the parameters according to the gradient information to improve the convergence speed and accuracy of the model. The adaptive learning rate adjustment feature of the Adam optimizer can effectively reduce the oscillation and convergence time during the training process, improve the efficiency and accuracy of parameter updating, and enable the model to converge to a better solution within a fewer number of iterations, reducing training time and computing resource consumption. After the set training cycle, such as each epoch, the model is evaluated using the joint validation set, and the quality index FI of the image generation is calculated. D. FID score is a standard indicator for evaluating the quality of generated images. It can quantify the difference between generated images and real images, ensure the diversity and realism of generated images, and regularly evaluate model performance, which helps to timely discover and adjust problems in model training and ensure the stability and effect of the model. Using FID score as the evaluation standard enhances the scientificity and accuracy of model evaluation, improves the quality and realism of generated images, and solves the problem of insufficient traditional indicators. The learning rate is dynamically adjusted based on the verification results. The specific operations are as follows: If the performance on the verification set does not improve within multiple evaluation cycles, it may be necessary to reduce the learning rate and use learning rate scheduling The device adjusts the learning rate according to evaluation indicators such as FID score to prevent overfitting or underfitting. By reducing the learning rate, the model can adjust parameters more finely and improve the quality of generated images. Dynamic adjustment of the learning rate can avoid the problem of too high or too low learning rate during training, so that the model can be trained at the optimal learning rate to achieve optimal performance and prevent overfitting and underfitting. By reasonably adjusting the learning rate, the generalization ability and stability of the model are improved. The introduction of the learning rate scheduling mechanism enhances the training flexibility and adaptability of the model, ensuring that the model can effectively adjust the learning strategy at different training stages and improve the quality and diversity of generated images.

[0039] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0040] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0041] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0042] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0043] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0044] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0045] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A training method for the Wensheng graph diffusion model based on deep learning, It is characterized in that The following steps are involved: Obtain text training data, image training data, and joint training data; The text-graph diffusion model is pre-trained in stages based on text training data and image training data, including text stage pre-training and image stage pre-training. Based on the joint training data, the pre-trained Vincent graph diffusion model is jointly trained, and the results of each joint training are evaluated and adjusted until the Vincent graph diffusion model meets the expected standards.

2. The training method of the deep learning-based Wensheng graph diffusion model according to claim 1, It is characterized in that The text training data specifically includes a number of text information and corresponding text semantic vectors used to train the text encoder in the text-graph diffusion model, the image training data specifically includes a number of text semantic vectors and corresponding text semantic image data used to train the image generator in the text-graph diffusion model, and the joint training data specifically includes a number of text information and corresponding text image data.

3. The training method of the deep learning-based Wensheng graph diffusion model according to claim 2, It is characterized in that The text encoder in the text-graph diffusion model is specifically a BERT model. The specific process of performing text-stage pre-training on the text-graph diffusion model based on text training data is as follows: Read and preprocess several text information used to train the text encoder in the text graph diffusion model; A text information dataset is established based on the preprocessed text information and the corresponding text semantic vectors, and is divided into a text information training set and a text information verification set; The text information training set is divided into a number of batch training sets, and each batch training set contains a number of text information; In the BERT model, each batch of training sets is forward propagated to obtain the CLS vector of each text information in each batch of training sets, and the text loss function is calculated; The BERT model is trained iteratively, and the training results of each BERT model are evaluated based on the back-propagation algorithm and the text information verification set. The model parameters are adjusted according to the verification results until the CLS vector of the text information predicted by the model meets the corresponding text semantic vector.

4. The training method of the deep learning-based Wensheng graph diffusion model according to claim 3, It is characterized in that The text loss function is specifically calculated using the cosine similarity formula, and its specific formula is as follows: Among them, L is the text loss function, CLS ij is the CLS vector of the jth text information in the i-th batch training set, WY ij is the text semantic vector corresponding to the jth text information in the i-th batch training set, i = 1, 2, 3, ..., N, N is the number of divided batch training sets, j = 1, 2, 3, ..., M, N is the number of text information in the batch training set.

5. The training method of the deep learning-based Wensheng graph diffusion model according to claim 2, It is characterized in that The text encoder in the text-graph diffusion model is specifically a generative adversarial network, which includes a generator and a discriminator. The specific process of performing image-stage pre-training on the text-graph diffusion model based on image training data is as follows: Initialize the generative adversarial network; Read several text semantic vectors and corresponding text semantic image data used to train the image generator in the text image diffusion model, and establish an image data set, which is divided into an image data training set and an image data verification set; The generative adversarial network is cyclically trained based on the image data training set, specifically, the generator in the generative adversarial network receives each text semantic vector and generates a corresponding text image; Perform discriminative training on the discriminator in the generative adversarial network based on the text image and the corresponding text semantic image; Conduct adversarial training on the generator and discriminator in the generative adversarial network, and calculate the generator loss function and the discriminator loss function respectively; The performance of each training result of the generative adversarial network is evaluated based on the image data validation set, and the model parameters are adjusted according to the verification results until the text image predicted by the generative adversarial network meets the corresponding text semantic image.

6. The training method of the deep learning-based Wensheng graph diffusion model according to claim 5, It is characterized in that The specific formulas for calculating the generator loss function and the discriminator loss function are as follows: Among them, S is the generator loss function, WY u is the u-th text semantic vector received by the generator in the generative adversarial network, G(WY u ) is the text image generated by the u-th text semantic vector, ZS u is the text semantic image corresponding to the u-th text semantic vector received by the generator in the generative adversarial network, P is the discriminator loss function, and y u is the image label of the u-th text semantic vector received by the discriminator in the generative adversarial network, u = 1, 2, 3, ..., V, V is the number of text semantic vectors obtained for training the image generator in the text-image diffusion model.

7. The training method of the deep learning-based Wensheng graph diffusion model according to claim 2, It is characterized in that Based on the joint training data, the pre-trained Vincent graph diffusion model is jointly trained, and the results of each joint training are evaluated and adjusted until the Vincent graph diffusion model meets the expected standards. Specifically: Read several text information and corresponding text image data, and merge them into a joint data set; Divide the joint dataset into a joint training set and a joint validation set; Load the pre-trained BERT model and generative adversarial network, perform batch training on the text graph diffusion model based on the joint training set, and calculate the diffusion loss function; Back-propagate the diffusion loss function and calculate the gradient; Use the Adam optimizer to optimize the model parameters according to the calculated gradients, and after the set training cycle, use the joint validation set to evaluate the model and adjust the learning rate based on the evaluation results; The training process is repeated until the Vincent graph diffusion model meets the expected standards.

8. The training method of the deep learning-based Wensheng graph diffusion model according to claim 7, It is characterized in that The specific process of batch training the Wensheng graph diffusion model based on the joint training set is as follows: For each batch, the text information is processed based on the BERT model to generate a high-dimensional semantic vector; Generate a random noise image based on the generative adversarial network, and use this noise image as the starting point of the diffusion model; Using the text semantic vector as a condition, the text semantic vector is input into the generator of the generative adversarial network to gradually remove noise, and the generator of the generative adversarial network repeatedly applies the conditional noise removal process.

9. The training method of the deep learning-based Wensheng graph diffusion model according to claim 7, It is characterized in that The specific formula for calculating the diffusion loss function is as follows: Among them, Ks is the diffusion loss function, q(x t |x 0 ) is the conditional probability distribution of the forward process, p(x t |x t-1 ) is the conditional probability distribution of the reverse process, x 0 is the initial image data, t is the time step, x t is the image data at the t-th time step, x t-1 is the image data at the (t - 1)-th time step.

10. The training method of the deep learning-based Wensheng graph diffusion model according to claim 7, It is characterized in that The Adam optimizer is used to optimize the model parameters according to the calculated gradient. After the set training cycle, the joint validation set is used to evaluate the model. The specific process of adjusting the learning rate based on the evaluation results is as follows: Initialize the Adam optimizer and obtain the preset initial learning rate and hyperparameters; After each training batch, the value of each parameter is adjusted according to the gradient information calculated by back-propagation to minimize the diffusion loss function; After the set training cycle is completed, the model is evaluated using the joint validation set and the quality indicators of image generation are calculated; The learning rate is dynamically adjusted based on the validation results.

Citation Information

Patent Citations

  • Text-based image diffusion model training method and text-based image generation method

    CN116051668A

  • Image generation model compression and acceleration method and system based on diffusion model

    CN116542321A

  • Deep learning-based text image generation method and device and readable storage medium

    CN117611944A

  • Figure graph and model training method and device, electronic equipment and storage medium

    CN118155023A

  • Figure graph model training method and text graph method

    CN118194958A

Cited By

  • Product testing method and system based on AI large model and machine learning

    CN119377085A

  • Zero-sample custom image generation method, control device, storage medium and system

    CN119559290A

  • Medical image enhancement processing method and system based on deep learning

    CN119671884A

  • Vehicle cabin adjusting method and device, vehicle and storage medium

    CN120439965A

  • Image generation training method for AIGC large model

    CN121437670A