A training method of a text-to-image diffusion model based on deep learning

Abstract

The application relates to a training method of a text-to-image diffusion model based on deep learning and relates to the technical field of artificial intelligence. The training method of the text-to-image diffusion model based on deep learning acquires text training data, image training data and joint training data; text-to-image diffusion models are respectively subjected to stage pre-training based on the text training data and the image training data, including text stage pre-training and image stage pre-training; the text-to-image diffusion models subjected to stage pre-training are subjected to joint training based on the joint training data, and each joint training result is evaluated and adjusted until the text-to-image diffusion model meets an expected standard; the application significantly improves the training precision of the text-to-image diffusion model and the quality of generated images through multi-stage pre-training, utilization of joint training data, combination of a BERT model and a generative adversarial network, introduction of a diffusion process for gradually removing noise and dynamic adjustment of a learning rate.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a training method for a text-based graph diffusion model based on deep learning. Background Technology

[0002] In the field of image generation, deep learning-based methods have made significant progress in recent years. Text-to-Image (TTO) generation technology is a technique that generates corresponding images from input text descriptions. This technology has important value in many application areas, such as automatic image generation, content creation, and virtual reality. However, existing TTO generation technologies still face many challenges in practical applications. Therefore, a training method for a deep learning-based TTO diffusion model has emerged.

[0003] Existing technologies, such as the text-based image diffusion model training method and text-based image generation method disclosed in invention patent application CN116051668B, involve the following steps: the text-based image diffusion model denoises a noisy sample image based on the sample text to generate a denoised sample image; a first text-image alignment score is obtained based on the first representation vector of the denoised sample image and the second representation vector of the sample text, and a first training sample is selected from the current batch of training samples based on the first text-image alignment score; a first loss function of the text-based image diffusion model is determined based on the original sample image and the denoised sample image of the sample text in the first training sample, and the text-based image diffusion model is adjusted based on the first loss function; training continues using the next batch of training samples until the training is completed and the target text-based image diffusion model is obtained, thus improving the training accuracy of the text-based image diffusion model.

[0004] Based on the above solutions, the limitations of the existing technology include at least the following problems: First, there are deficiencies in noise reduction processing, which can easily lead to poor image quality and inaccurate text-image alignment scores, affecting the training effect of the model. Second, the selection of difficult samples is insufficient, which can easily lead to poor learning effect of the model on complex samples. Third, the adjustment of model parameters lacks flexibility and effectiveness, affecting the final performance. Summary of the Invention

[0005] This invention addresses the technical problems existing in the prior art by providing a training method for a text-based image diffusion model based on deep learning, thus solving the problems.

[0006] The technical solution of this invention to solve the above-mentioned technical problems is as follows: a training method for a text-based image diffusion model based on deep learning, comprising the following steps: acquiring text training data, image training data, and joint training data; performing stage pre-training on the text-based image diffusion model based on the text training data and image training data, including text stage pre-training and image stage pre-training; performing joint training on the stage pre-trained text-based image diffusion model based on the joint training data, and evaluating and adjusting the results of each joint training until the text-based image diffusion model meets the expected standards.

[0007] Furthermore, the text training data specifically comprises several text information and corresponding text semantic vectors used to train the text encoder in the text-to-image diffusion model; the image training data specifically comprises several text semantic vectors and corresponding text semantic image data used to train the image generator in the text-to-image diffusion model; and the joint training data specifically comprises several text information and corresponding text image data.

[0008] Furthermore, the text encoder in the text-based image diffusion model is specifically a BERT model. The specific process of pre-training the text-based image diffusion model based on text training data is as follows: Several text information pieces used to train the text encoder in the text-based image diffusion model are read and preprocessed; a text information dataset is established based on the preprocessed text information pieces and the corresponding text semantic vectors, and divided into a text information training set and a text information validation set; the text information training set is divided into several batch training sets, and each batch training set contains several text information pieces; forward propagation processing is performed on each batch training set in the BERT model to obtain the CLS vector of each text information piece in each batch training set, and the text loss function is calculated; the BERT model is iteratively trained, and the training results of the BERT model are evaluated based on the backpropagation algorithm and the text information validation set, and the model parameters are adjusted according to the validation results until the CLS vector of the text information predicted by the model matches the corresponding text semantic vector.

[0009] Furthermore, the text loss function is specifically calculated using the cosine similarity formula, the specific formula of which is as follows: ;in, For text loss function, For the first The first batch of training sets A CLS vector of text information. For the first The first batch of training sets The text semantic vector corresponding to each piece of text information. , The number of batches in the training set. , This represents the number of text messages in the batch training set.

[0010] Furthermore, the text encoder in the text-to-image diffusion model is specifically a generative adversarial network (GAN), which includes a generator and a discriminator. The specific process of image-stage pre-training of the text-to-image diffusion model based on image training data is as follows: Initialize the GAN; read several text semantic vectors and corresponding text semantic image data used to train the image generator in the text-to-image diffusion model, and establish an image dataset, which is divided into an image data training set and an image data validation set; perform iterative training on the GAN based on the image data training set, specifically, the generator in the GAN receives each text semantic vector and generates the corresponding text image; perform discriminative training on the discriminator in the GAN based on the text image and the corresponding text semantic image; perform adversarial training on the generator and discriminator in the GAN, and calculate the generator loss function and discriminator loss function respectively; evaluate the performance of each training result of the GAN based on the image data validation set, and adjust the model parameters according to the validation results until the text image predicted by the GAN matches the corresponding text semantic image.

[0011] Furthermore, the specific formulas for calculating the generator loss function and the discriminator loss function are as follows: ;in, Let be the generator loss function. For the generator in a generative adversarial network, the first... A text semantic vector, For the first Text images generated from text semantic vectors For the generator in a generative adversarial network, the first... The text semantic image corresponding to each text semantic vector. Let the discriminator loss function be... To generate the first received by the discriminator in the adversarial network Image labels for text semantic vectors, , This refers to the number of text semantic vectors obtained for training the image generator in the text-to-image diffusion model.

[0012] Furthermore, the text-image diffusion model, after pre-training, is jointly trained based on joint training data. The results of each joint training session are evaluated and adjusted until the text-image diffusion model meets the expected criteria. Specifically: several text information sets and their corresponding text-image data are read and merged into a joint dataset; the joint dataset is divided into a joint training set and a joint validation set; the pre-trained BERT model and generative adversarial network are loaded, and the text-image diffusion model is batch-trained based on the joint training set, with the diffusion loss function calculated; backpropagation is performed on the diffusion loss function, and the gradient is calculated; the Adam optimizer is used to optimize the model parameters based on the calculated gradient, and after a set training period, the model is evaluated using the joint validation set, with the learning rate adjusted based on the evaluation results; the training process is repeated until the text-image diffusion model meets the expected criteria.

[0013] Furthermore, the specific process of batch training the text-based image diffusion model based on the joint training set is as follows: For each batch, text information is processed based on the BERT model to generate high-dimensional semantic vectors; a random noise image is generated based on the generative adversarial network, and this noise image serves as the starting point of the diffusion model; using the text semantic vector as a condition, the text semantic vector is input into the generator of the generative adversarial network to gradually remove noise, and the generator of the generative adversarial network repeatedly applies the conditional noise removal process.

[0014] Furthermore, the specific formula for calculating the diffusion loss function is as follows: ;in, Let be the diffusion loss function. This represents the conditional probability distribution for the forward process. This represents the conditional probability distribution for the reverse process. For initial image data, For time steps, For the first Image data at each time step For the first Image data at each time step.

[0015] Furthermore, the specific process of using the Adam optimizer to optimize the model parameters based on the calculated gradients, evaluating the model using a joint validation set after a set training period, and adjusting the learning rate based on the evaluation results is as follows: Initialize the Adam optimizer and obtain the pre-set initial learning rate and hyperparameters; after each training batch, adjust the value of each parameter based on the gradient information calculated by backpropagation to minimize the diffusion loss function; after the set training period ends, evaluate the model using a joint validation set and calculate the quality index of image generation; dynamically adjust the learning rate based on the validation results.

[0016] The beneficial effects of this invention are as follows: By performing staged pre-training on the text encoder and image generator respectively, the performance of the model in the initial stage is significantly improved. By merging text data and image data into a joint dataset for training, the comprehensive ability of the model in processing multimodal data is enhanced, making the generated images more semantically consistent with the text description. The BERT model is used to extract high-quality text semantic vectors, and combined with a generative adversarial network to gradually remove noise in a multi-step iteration process, generating high-quality images and improving the clarity of the images and their consistency with the text description. A diffusion model is introduced in the image generation process to gradually remove noise through multi-step iteration, which not only enhances the image details and textures but also ensures the quality of the generated images. By using the Adam optimizer and learning rate scheduler, the learning rate is dynamically adjusted according to the validation results, ensuring that the model can be continuously optimized during training, thereby improving training efficiency and the quality of generated images. Attached Figure Description

[0017] Figure 1 This is a flowchart of a training method for a text-based graph diffusion model based on deep learning, according to the present invention.

[0018] Figure 2 This is a flowchart illustrating the specific steps of pre-training the text-based image diffusion model using text training data in the training method of the deep learning-based text-based image diffusion model of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0021] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0022] The problem addressed in this application's embodiments can be summarized as follows:

[0023] First, text training data, image training data, and joint training data are acquired and preprocessed to generate text semantic vectors and image feature vectors. Second, the text encoder (BERT model) is pre-trained on the text training data for the text stage, and the image generator (Generative Adversarial Network) is pre-trained on the image stage based on the image training data. Third, the preprocessed text and image data are merged into a joint dataset and divided into a joint training set and a joint validation set. The pre-trained BERT model and Generative Adversarial Network are loaded, and batch training is performed on the joint training set. The diffusion loss function is calculated, backpropagation is performed on the diffusion loss function, the gradient is calculated, and the parameters are optimized using the Adam optimizer. Finally, after a set training period, the model is evaluated using the joint validation set, and the learning rate is dynamically adjusted based on the evaluation results. The training process is repeated until the text-to-image diffusion model meets the expected criteria.

[0024] Please see Figure 1 This invention provides a technical solution: a training method for a text-based image diffusion model based on deep learning, comprising the following steps: acquiring text training data, image training data, and joint training data; performing stage pre-training on the text-based image diffusion model based on the text training data and image training data, including text stage pre-training and image stage pre-training; performing joint training on the stage pre-trained text-based image diffusion model based on the joint training data, and evaluating and adjusting the results of each joint training until the text-based image diffusion model meets the expected standards.

[0025] The text training data specifically consists of several text information and corresponding text semantic vectors used to train the text encoder in the text-to-image diffusion model. The image feature vector specifically consists of the feature vectors of the real images generated corresponding to the text information. The image training data specifically consists of several text semantic vectors and corresponding text semantic image data used to train the image generator in the text-to-image diffusion model. The joint training data specifically consists of several text information and corresponding text image data.

[0026] Specifically, such as Figure 2 As shown, the text encoder in the text-based graph diffusion model is specifically the BERT model. The specific process of pre-training the text-based graph diffusion model based on text training data is as follows: Several text messages used to train the text encoder in the text-based graph diffusion model are read and preprocessed; a text information dataset is established based on the preprocessed text messages and corresponding text semantic vectors, and divided into a text information training set and a text information validation set; the text information training set is divided into several batch training sets, and each batch training set contains several text messages; forward propagation is performed on each batch training set in the BERT model to obtain the CLS vector of each text message in each batch training set, and the text loss function is calculated; the BERT model is iteratively trained, and the training results of the BERT model are evaluated based on the backpropagation algorithm and the text information validation set, specifically by minimizing the text loss function and adjusting the model parameters according to the validation results until the CLS vector of the predicted text information conforms to the corresponding text semantic vector.

[0027] The text loss function is specifically calculated using the cosine similarity formula, which is as follows: ;in, For text loss function, For the first The first batch of training sets A CLS vector of text information. For the first The first batch of training sets The text semantic vector corresponding to each piece of text information. For the first The first batch of training sets The magnitude of the CLS vector of a text message. For the first The first batch of training sets The modulus of the text semantic vector corresponding to each piece of text information , The number of batches in the training set. , This represents the number of text messages in the batch training set.

[0028] In this implementation scheme, several text information pieces used to train the text encoder in the text-to-image diffusion model are read and preprocessed. A text information dataset is then established based on the preprocessed text information and its corresponding semantic vectors, and divided into a training set and a validation set. Preprocessing and dataset construction ensure data quality and consistency, improving the reliability of model training. The generation of semantic vectors allows text information to be efficiently input into the model for training. Systematic construction of training and validation datasets provides stable foundational data, ensuring the effectiveness of the model training process and the accuracy of the evaluation. The text information training set is divided into several batch training sets, each containing several text information pieces. Batch training improves training efficiency and ensures a stable gradient is obtained each time the model parameters are updated. Degree estimation and batch training help accelerate model training and reduce memory consumption. By dividing into batches, the memory limitation problem in large-scale data training is solved, improving the efficiency and stability of model training. In the BERT model, forward propagation is performed on each batch of training sets to obtain the CLS vector of each text information in each batch of training sets, and the text loss function is calculated. Forward propagation can extract the deep semantic features of each text information, and the CLS vector captures the global information of the text. The calculation of the text loss function ensures that the training objective of the model is clear and the optimization direction is clear. Through forward propagation and CLS vector calculation, the model's ability to understand text semantics is enhanced, and the accuracy and consistency of text-to-image generation are improved. The BERT model is iteratively trained, and the training results of the BERT model are evaluated based on the backpropagation algorithm and the text information validation set. Specifically, the model minimizes the text loss function and adjusts the model parameters based on the validation results until the CLS vector of the predicted text information matches the corresponding text semantic vector. Iterative training and backpropagation ensure that the model can continuously optimize and gradually improve training accuracy. Evaluation and parameter adjustment based on the validation set ensure the model's generalization ability and practical application effect. Through iterative training and validation, the model can continuously learn and optimize, gradually improving the quality of the generated image and its consistency with the text semantics. Dynamic parameter adjustment allows the model to maintain optimal performance at different stages, solving the overfitting or underfitting problems that may occur during model training. The text loss function is specifically calculated using the cosine similarity formula, which can effectively measure the similarity between text vectors and improve the consistency of text-to-image generation. Cosine similarity can eliminate the influence of vector size and focus on directional similarity, improving the model's ability to capture semantic information. By using the cosine similarity loss function, the accuracy of text encoding and the quality of generated images are significantly improved, solving the problem of insufficient accuracy of loss functions in existing technologies and improving the training and generation effects of the model.

[0029] Specifically, the text encoder in the text-based image diffusion model is a generative adversarial network (GAN), which includes a generator and a discriminator. The specific process of image-stage pre-training of the text-based image diffusion model based on image training data is as follows: Initialization of the GAN is performed, specifically initializing the generator and discriminator, creating the network structures of the generator and discriminator. The generator is responsible for generating images, and the discriminator is responsible for determining whether an image is a real image. Several text semantic vectors and corresponding text semantic image data used to train the image generator in the text-based image diffusion model are read, and an image dataset is established, which is then divided into image data training sections. The training and validation sets of the generative adversarial network (GAN) are used. The GAN is trained iteratively using the image data training set. Specifically, the generator in the GAN receives each text semantic vector and generates the corresponding text image. The discriminator in the GAN is trained based on the text image and its corresponding text semantic image. The generator and discriminator in the GAN are then trained adversarially, and the generator loss function and discriminator loss function are calculated respectively. The performance of each training result of the GAN is evaluated based on the image data validation set, and the model parameters are adjusted according to the validation results until the text image predicted by the GAN matches the corresponding text semantic image.

[0030] The specific formulas for calculating the generator loss function and the discriminator loss function are as follows: ;in, Let be the generator loss function. For the generator in a generative adversarial network, the first... A text semantic vector, For the first Text images generated from text semantic vectors For the generator in a generative adversarial network, the first... The text semantic image corresponding to each text semantic vector. This is the discriminator's view of the image generated by the generator. The output, specifically... This indicates that the discriminator judges the image generated by the generator. The probability of it being a "real image" This represents taking the logarithm of the discriminator's output. The logarithmic function is used to amplify the effect of outputs with a probability close to 1, because the logarithmic function is very sensitive to changes when the input is close to 1. Let the discriminator loss function be... To generate the first received by the discriminator in the adversarial network For a real image, the image label is a text semantic vector. The label is 1; for the generated image, the label is 0. and These represent the probability of the discriminator judging the real image and the generated image, respectively. This is the logarithm of the probability that the discriminator correctly identifies a real image as real. This is the logarithm of the probability that the discriminator correctly identifies a generated image as fake. The negative sign indicates that by minimizing this combined loss, the discriminator can be trained to correctly distinguish between real and fake images. , This refers to the number of text semantic vectors obtained for training the image generator in the text-to-image diffusion model.

[0031] In this implementation scheme, the generator and discriminator of the generative adversarial network are initialized, and their network structures are created. The generator is responsible for generating images, and the discriminator is responsible for determining whether an image is real. The initialization process ensures the reasonable design of the generator and discriminator structures, laying the foundation for subsequent training. Clarifying the responsibilities of the generator and discriminator helps improve the quality and realism of the generated images. By reasonably designing the network structures of the generator and discriminator, the model's generation and discrimination capabilities are improved, laying the foundation for the overall performance of the model. Several text semantic vectors and corresponding text semantic image data used to train the image generator in the text-to-image diffusion model are read to establish an image dataset. The image dataset is divided into a training set and a validation set. This data reading and dataset partitioning ensures the quality and independence of the training and validation data, improving the model's training performance and evaluation accuracy. Partitioning the training and validation sets effectively prevents overfitting and enhances the model's generalization ability. A stable training and validation environment is provided through reasonable dataset partitioning, ensuring the model can effectively learn and evaluate during training. The generative adversarial network is trained iteratively based on the image training set. Specifically, the generator receives each text semantic vector and generates a corresponding text image. The discriminator is then trained based on the generated text images and their corresponding semantic images. The discriminator determines whether the generated image is real. Through iterative training of the generator and discriminator, the quality and realism of the generated images are continuously optimized. Adversarial training mechanisms can make the generated images more realistic, improving the model's generation ability. Adversarial training between the generator and discriminator improves the quality of the generated images and the accuracy of discrimination, enhancing the overall performance of the model. The reasonable design of the generator and discriminator loss functions helps in the effective training of the model, ensuring the quality of the generated images and the accuracy of discrimination. Optimizing the adversarial loss function makes the generated images more realistic, and the discriminator's ability to distinguish between real and generated images is stronger. This is achieved through the reasonable design and optimization of the loss function. This function significantly improves the training effect of generative adversarial networks (GANs), enhances the quality of generated images and the model's discriminative ability. It evaluates the performance of each training result of the GAN based on an image data validation set, and adjusts the model parameters according to the validation results until the text images predicted by the GAN match the corresponding text semantic images. Performance evaluation ensures the model's performance on the validation set, allowing for timely identification and correction of model problems. Dynamically adjusting model parameters based on evaluation results effectively prevents overfitting or underfitting, improves the model's generalization ability, and ensures that the model maintains optimal performance at different stages through regular evaluation and parameter adjustment, solving the problem of inflexible model parameter adjustment in existing technologies.

[0032] Specifically, the text-image diffusion model is jointly trained based on the pre-trained joint training data, and the results of each joint training are evaluated and adjusted until the text-image diffusion model meets the expected standards. Specifically: several text information sets and their corresponding text-image data are read and merged into a joint dataset, where each text semantic vector has a corresponding image data set, forming a one-to-one correspondence; the joint dataset is divided into a joint training set and a joint validation set; the pre-trained BERT model and generative adversarial network are loaded, and the text-image diffusion model is batch-trained based on the joint training set, calculating the diffusion loss function. Each batch contains a set of text semantic vectors and their corresponding real images; the diffusion loss function is backpropagated, and the gradient is calculated, specifically by passing the loss layer from the output layer to the input layer, calculating the gradient of each layer's parameters, such as weights and biases, with respect to the loss; the Adam optimizer is used to optimize the model parameters based on the calculated gradients, and after a set training period, the model is evaluated using the joint validation set, and the learning rate is adjusted based on the evaluation results; the training process is repeated until the text-image diffusion model meets the expected standards.

[0033] The specific formula for calculating the diffusion loss function is as follows: ;in, Let be the diffusion loss function. This represents the conditional probability distribution for the forward process. This represents the conditional probability distribution for the reverse process. Indicates to Expected value under the distribution Used to measure the difference between the model's predicted inverse process and the actual forward process. The initial image data, during the diffusion process, represents the first noise-free image and is the starting point of the diffusion process. For time steps, For the first Image data at time step n, during the diffusion process, represents the image data at time step n. Noisy images during step, as As the noise level increases, the noise gradually increases in the image. For the first Image data at the nth time step, representing the image data at the nth time step. The noisy image at step 1 is used to predict the 2nd step. Images taken during the step.

[0034] In this implementation scheme, several text messages and their corresponding text-image data are read and merged into a joint dataset. Each text semantic vector has a one-to-one correspondence with its corresponding image data. By ensuring this one-to-one correspondence, multimodal data can be fully utilized, improving the completeness and consistency of the training data. This process provides foundational data for subsequent model training, ensuring data quality and relevance, and providing a comprehensive joint dataset that enables the model to learn the relationship between text and images simultaneously, improving the model's generative ability and consistency. The joint dataset is divided into a joint training set and a joint validation set. The training set and validation set can be used for model training and performance evaluation, respectively, to prevent overfitting and improve the model's generalization ability. The training set is used for updating model parameters, while the validation set is used to monitor model performance and adjust training strategies, ensuring that the model obtains stable training data during training. Simultaneously, the validation set is used to evaluate model performance, optimize the training process, and improve the model's generalization ability. A pre-trained BERT model and a generative adversarial network are loaded, and batch training of the text-to-image diffusion model is performed based on the joint training set. Each batch contains a set of text semantic vectors and their corresponding real images. Batch training improves training efficiency and stability, ensuring reliable gradient estimation for each parameter update. The diffusion loss function measures the difference between the model's predicted backward process and the actual forward process. The differences in processing enable the model to more accurately recover clear images from noisy data, improving the model's training efficiency and stability. Optimizing the diffusion loss function makes the generated images clearer and more accurate. Backpropagation is performed on the diffusion loss function, and gradients are calculated, passing the loss layer by layer from the output layer to the input layer. Parameters for each layer, such as weights and biases, are calculated, along with the gradient of the loss. The backpropagation algorithm allows the model to continuously optimize parameters based on feedback from the loss function, improving training effectiveness. Accurate gradient calculation ensures appropriate direction and magnitude of model parameter updates, improving training efficiency. Through backpropagation and gradient calculation, the model can effectively learn and adjust parameters, gradually improving the quality of generated images and their consistency with text semantics. Using Ada... The Adam optimizer optimizes model parameters based on the calculated gradients. It initializes the Adam optimizer with a pre-set initial learning rate and hyperparameters. After each training batch, it adjusts the value of each parameter based on the gradient information calculated by backpropagation to minimize the diffusion loss function. After a set number of training periods, the model is evaluated using a joint validation set, and the learning rate is adjusted based on the evaluation results. If performance on the validation set does not improve over multiple evaluation periods, the learning rate may need to be reduced. A learning rate scheduler is used to adjust the learning rate based on evaluation metrics such as the FID score. The Adam optimizer dynamically adjusts the learning rate for each parameter, improving training stability and efficiency. Dynamically adjusting the learning rate based on validation results can prevent overfitting or underfitting.To ensure the model maintains optimal performance at different stages, the Adam optimizer and learning rate scheduler are used to efficiently optimize parameters, improving training effectiveness and the quality of generated images.

[0035] Specifically, the batch training process of the text-based image diffusion model based on the joint training set is as follows: For each batch, the text information is processed based on the BERT model to generate a high-dimensional semantic vector; a random noise image is generated based on the generative adversarial network (GAN), which serves as the starting point for the diffusion model. This image contains a high level of random noise and no meaningful visual content; using the text semantic vector as a condition, the text semantic vector is input into the generator of the GAN to gradually remove noise. This step is iterative, with each iteration using the output of the previous iteration as input to refine image details and reduce noise. The generator of the GAN repeatedly applies the conditional noise removal process, further reducing noise in the image and enhancing visual details that match the text description based on the previous step. After each iteration, the image clarity and quality gradually improve, becoming closer to the actual image appearance. As the diffusion step proceeds, not only is noise removed, but the details and textures in the image are also gradually enhanced, making the final generated image highly consistent with the text content visually.

[0036] In this implementation scheme, for each batch, text information is processed based on the BERT model to generate high-dimensional semantic vectors. Using the BERT model to process text information captures deep semantic features of the text, generating high-quality semantic vectors. These high-dimensional semantic vectors provide rich textual information, helping to better match text descriptions when generating images, thus improving the representational power of text information. This allows the model to more accurately understand and generate images that match the text semantics. A random noise image is generated based on a generative adversarial network (GAN). This noise image serves as the starting point for the diffusion model. This image contains a high level of random noise and no meaningful visual content. Generating a high-level random noise image ensures that the diffusion model has a unified starting point, facilitating subsequent noise removal. The random noise image provides an initial noise environment, allowing the diffusion process to progressively optimize the quality of the generated images. A unified noise starting point improves the stability of model training, ensuring consistency and controllability in each training process. Using the text semantic vector as a condition, the text semantic vector is input into the generator of the GAN to progressively remove noise. This step is... Iteratively, each iteration uses the output of the previous iteration as input to refine image details and reduce noise. The iterative denoising process gradually improves the image's clarity and quality, making the generated image more realistic and detailed. Each iteration's refinement process ensures that the generated image gradually approximates the actual image appearance, enhancing its matching degree with the text description. Through the iterative denoising process, the resolution and visual quality of the generated image are improved, enabling the model to generate higher-quality images. The generator of the generative adversarial network repeatedly applies the conditional noise removal process, further reducing noise in the image at each step based on the previous step and enhancing visual details that match the text description. As the diffusion step proceeds, not only is noise removed, but the details and textures in the image are also gradually enhanced. Repeatedly applying the conditional noise removal process can continuously enhance image details and textures, making the final generated image highly consistent with the text content visually. This process can gradually optimize image quality, making the generated image not only highly clear but also rich in detail. By continuously enhancing image details and textures, the quality and realism of the model-generated images are improved, solving the problem of low image quality in existing technologies.

[0037] Specifically, the process of using the Adam optimizer to optimize model parameters based on the calculated gradients, evaluating the model using a joint validation set after a set training period, and adjusting the learning rate based on the evaluation results is as follows: Initialize the Adam optimizer and obtain the pre-set initial learning rate and hyperparameters; after each training batch, adjust the value of each parameter based on the gradient information calculated by backpropagation to minimize the diffusion loss function; after each set training period, such as each epoch, evaluate the model using a joint validation set and calculate the image generation quality index (FID); dynamically adjust the learning rate based on the validation results. Specifically, if the performance on the validation set does not improve over multiple evaluation periods, it may be necessary to reduce the learning rate. Use the learning rate scheduler to adjust the learning rate according to the evaluation index, such as the FID score, to prevent overfitting or underfitting. By reducing the learning rate, the model can finely adjust its parameters and improve the quality of the generated images.

[0038] In this implementation, the Adam optimizer is used, and a pre-set initial learning rate and hyperparameters are obtained. The Adam optimizer combines the advantages of Momentum and RMSprop, enabling adaptive adjustment of the learning rate during training to effectively avoid gradient vanishing and gradient exploding problems. Pre-setting the initial learning rate and hyperparameters makes the optimization process more stable and efficient. By using the Adam optimizer, the optimization process is more efficient, reducing the complexity of hyperparameter tuning and improving the model's convergence speed and stability. After each training batch, backpropagation is used to calculate gradient information, and the value of each parameter is adjusted based on the calculated gradient information to minimize the diffusion loss function. The backpropagation algorithm allows the model to accurately calculate the gradient of each parameter, and the optimizer can adjust the parameters based on this gradient information, improving the model's convergence speed and accuracy. The adaptive learning rate adjustment feature of the Adam optimizer effectively reduces oscillations and convergence time during training, improving the efficiency and accuracy of parameter updates, enabling the model to converge to a better solution in fewer iterations, reducing training time and computational resource consumption. After each set training period, such as each epoch, the model is evaluated using a joint validation set, and the image generation quality index FI is calculated. FID (Fixed Image ID) score is a standard metric for evaluating the quality of generated images. It quantifies the difference between generated and real images, ensuring the diversity and realism of generated images. Regularly evaluating model performance helps to identify and adjust problems during model training in a timely manner, ensuring the stability and effectiveness of the model. Using FID score as an evaluation standard enhances the scientific rigor and accuracy of model evaluation, improves the quality and realism of generated images, and addresses the shortcomings of traditional metrics. The learning rate is dynamically adjusted based on validation results. Specifically, if performance on the validation set does not improve over multiple evaluation periods, the learning rate may need to be reduced using learning rate scheduling. The learning rate is adjusted based on evaluation metrics, such as FID score, to prevent overfitting or underfitting. By reducing the learning rate, the model can finely adjust parameters, improving the quality of generated images. Dynamically adjusting the learning rate avoids problems of excessively high or low learning rates during training, allowing the model to be trained at the optimal learning rate to achieve best performance and prevent overfitting and underfitting. By reasonably adjusting the learning rate, the model's generalization ability and stability are improved. The introduction of a learning rate scheduling mechanism enhances the model's training flexibility and adaptability, ensuring that the model can effectively adjust its learning strategy at different training stages, thereby improving the quality and diversity of generated images.

[0039] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0040] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0041] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0042] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0043] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0044] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0045] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A training method for a text-based image diffusion model based on deep learning, characterized in that, Includes the following steps: Acquire text training data, image training data, and joint training data; The text-to-image diffusion model is pre-trained in stages based on text training data and image training data, including text stage pre-training and image stage pre-training. The text-based image diffusion model is jointly trained based on the joint training data, and the results of each joint training are evaluated and adjusted until the text-based image diffusion model meets the expected standards. The text encoder in the text-based image diffusion model is specifically a generative adversarial network (GAN), which includes a generator and a discriminator. The specific process of image-stage pre-training of the text-based image diffusion model based on image training data is as follows: Initialize the generative adversarial network; Read several text semantic vectors and corresponding text semantic image data used to train the image generator in the text-to-image diffusion model, and establish an image dataset, which is divided into an image data training set and an image data validation set. The generative adversarial network is trained cyclically based on the image data training set. Specifically, the generator in the generative adversarial network receives each text semantic vector and generates the corresponding text image. The discriminator in the generative adversarial network is trained based on the text image and the corresponding text semantic image. Adversarial training is performed on the generator and discriminator in the generative adversarial network, and the loss functions of the generator and discriminator are calculated respectively. The performance of each training result of the generative adversarial network is evaluated based on the image data validation set, and the model parameters are adjusted according to the validation results until the text image predicted by the generative adversarial network matches the corresponding text semantic image. The specific formulas for calculating the generator loss function and the discriminator loss function are as follows: ； in, Let be the generator loss function. For the generator in a generative adversarial network, the first... A text semantic vector, For the first Text images generated from text semantic vectors For the discriminator to compare the images generated by the generator The output, For the generator in a generative adversarial network, the first... The text semantic image corresponding to each text semantic vector. and These represent the probability of the discriminator judging the real image and the generated image, respectively. Let the discriminator loss function be... To generate the first received by the discriminator in the adversarial network Image labels for text semantic vectors, , This refers to the number of text semantic vectors obtained for training the image generator in the text-to-image diffusion model.

2. The training method for the text-based image diffusion model based on deep learning according to claim 1, characterized in that, The text training data specifically comprises several text information and corresponding text semantic vectors used to train the text encoder in the text-to-image diffusion model. The image training data specifically comprises several text semantic vectors and corresponding text semantic image data used to train the image generator in the text-to-image diffusion model. The joint training data specifically comprises several text information and corresponding text image data.

3. The training method for the text-based image diffusion model based on deep learning according to claim 2, characterized in that, The text encoder in the text-generated image diffusion model is specifically a BERT model. The specific process of pre-training the text-generated image diffusion model based on text training data is as follows: Read several text information from the text encoder used to train the text-generated image diffusion model and preprocess them; A text information dataset is established based on several preprocessed text information and their corresponding text semantic vectors, and is divided into a text information training set and a text information validation set. The text information training set is divided into several batch training sets, and each batch training set contains several text information sets. In the BERT model, forward propagation is performed on each batch of training sets to obtain the CLS vector of each text information in each batch of training sets, and the text loss function is calculated. The BERT model is trained iteratively. The training results of the BERT model are evaluated based on the backpropagation algorithm and the text information validation set. The model parameters are adjusted according to the validation results until the CLS vector of the text information predicted by the model matches the corresponding text semantic vector.

4. The training method for the text-based image diffusion model based on deep learning according to claim 3, characterized in that, The text loss function is specifically calculated using the cosine similarity formula, and the specific formula is as follows: ； in, For text loss function, For the first The first batch of training sets A CLS vector of text information. For the first The first batch of training sets The text semantic vector corresponding to each piece of text information. , The number of batches in the training set. , This represents the number of text messages in the batch training set.

5. The training method for the text-based image diffusion model based on deep learning according to claim 2, characterized in that, The text-based image diffusion model, after phased pre-training, is jointly trained based on joint training data. The results of each joint training are evaluated and adjusted until the text-based image diffusion model meets the expected criteria. Specifically: Read several text messages and their corresponding text image data, and merge them into a joint dataset; The joint dataset is divided into a joint training set and a joint validation set. Load the pre-trained BERT model and generative adversarial network, batch train the text image diffusion model based on the joint training set, and calculate the diffusion loss function; Backpropagation is performed on the diffusion loss function, and the gradient is calculated; The Adam optimizer is used to optimize the model parameters based on the calculated gradients, and after a set training period, the model is evaluated using a joint validation set, and the learning rate is adjusted based on the evaluation results. Repeat the training process until the text-based image diffusion model meets the expected target.

6. The training method for the text-based image diffusion model based on deep learning according to claim 5, characterized in that, The specific process of batch training the text image diffusion model based on the joint training set is as follows: For each batch, text information is processed based on the BERT model to generate high-dimensional semantic vectors; A random noise image is generated based on a generative adversarial network, and this noise image serves as the starting point for the diffusion model. Using text semantic vectors as conditions, the text semantic vectors are input into the generator of the generative adversarial network to gradually remove noise, and the generator of the generative adversarial network repeatedly applies the conditional noise removal process.

7. The training method for the text-based image diffusion model based on deep learning according to claim 5, characterized in that, The specific formula for calculating the diffusion loss function is as follows: ； in, Let be the diffusion loss function. This represents the conditional probability distribution for the forward process. This represents the conditional probability distribution for the reverse process. Indicates to Expected value under the distribution For initial image data, For time steps, For the first Image data at each time step For the first Image data at each time step.

8. The training method for the text-based image diffusion model based on deep learning according to claim 5, characterized in that, The specific process of using the Adam optimizer to optimize model parameters based on the calculated gradients, evaluating the model using a joint validation set after a set training period, and adjusting the learning rate based on the evaluation results is as follows: Initialize the Adam optimizer and obtain the pre-set initial learning rate and hyperparameters; After each training batch, the value of each parameter is adjusted based on the gradient information calculated by backpropagation to minimize the diffusion loss function; After the set training period ends, the model is evaluated using a joint validation set, and the quality metrics of the generated images are calculated. The learning rate is dynamically adjusted based on the validation results.

Images