Image generation model training method and device, computer device, and storage medium

Abstract

The application provides an image generation model training method and device, computer equipment and a storage medium. The image generation model training method comprises the following steps: predefining a plurality of image-text pairs; training an evaluation model of an initial generation model based on the plurality of image-text pairs; the evaluation model is used for evaluating the image quality and semantic consistency of an output image of the initial generation model; encoding triad information of an original input text to obtain a first text feature vector, encoding the original input text to obtain a second text feature vector, and fusing the first text feature vector and the second text feature vector to obtain a fused text feature vector; inputting the fused text feature vector into the initial generation model to obtain a target image; and optimizing model parameters of the initial generation model according to an evaluation value of the target image to obtain a trained image generation model. The application can accurately capture the semantic interaction relationship in the text content, so that the semantic expression of the generated image is more accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image generation, in particular to an image generation model training method and device, computer equipment and a storage medium. BACKGROUND

[0002] The core of text-to-image technology is to convert natural language text into visual images. This process model needs to not only understand the surface meaning of the text, but also capture the deep semantics and contextual relationships therein.

[0003] Many existing methods perform well when dealing with simple descriptions, but often struggle when faced with complex semantic interactions. This is because most methods mainly focus on the surface features of the text, such as contours, colors, etc., while ignoring the implicit semantic interactions in the text. This neglect leads to deficiencies in semantic consistency and details in the generated images. For example, when the text description involves multiple objects and their interactions, existing models may not accurately capture the relationships between these objects, leading to problems of semantic inaccuracy in the generated images. SUMMARY

[0004] Therefore, the present application proposes an image generation model training method, device, computer equipment and storage medium to solve the problem of inaccurate semantic expression of generated images due to the inability to accurately capture semantic interaction relationships in text content in related technologies.

[0005] The first aspect of the present application proposes an image generation model training method, which comprises:

[0006] Predefine a plurality of image-text pairs; the image information in each image-text pair matches the text information;

[0007] An evaluation model of an initial generation model is trained based on the plurality of image-text pairs; the evaluation model is used to evaluate the image quality and semantic consistency of the output image of the initial generation model;

[0008] Encode the triple information of the original input text to obtain a first text feature vector, encode the original input text to obtain a second text feature vector, and fuse the first text feature vector and the second text feature vector to obtain a fused text feature vector; the original input text refers to the text information in any image-text pair; the triple information includes subject text information, predicate text information and object text information;

[0009] inputting the fusion text feature vector into the initial generation model to obtain a target image; the initial generation model is used to perform multi-step image processing on a random noise image to obtain the target image; the image processing includes image denoising and image evaluation; image denoising of each step is performed based on a denoised image of a previous step and the fusion text feature vector, and image evaluation of each step refers to evaluating an image quality and semantic consistency of a denoised image of a current step by using the evaluation model to obtain an evaluation value; the evaluation value of the target image is greater than a preset evaluation threshold value;

[0010] optimizing model parameters of the initial generation model according to the evaluation value of the target image to obtain a trained image generation model.

[0011] In the embodiments of the present application, a plurality of image-text pairs are predefined, including:

[0012] obtaining a plurality of image-text pairs; each image-text pair includes image information and text information corresponding to the image information;

[0013] For any image-text pair, encoding the text information in the image-text pair to obtain a text feature vector, and encoding the image information in the image-text pair to obtain an image feature vector;

[0014] calculating a similarity between the text feature vector and the image feature vector, and if the similarity is greater than a preset similarity threshold value, determining that the text information and the image information are matched;

[0015] According to the matching degree of the text information and the image information, the plurality of image-text pairs are filtered out from the plurality of image-text pairs.

[0016] In the embodiments of the present application, after obtaining a plurality of image-text pairs, the method includes:

[0017] In response to a manual annotation instruction, a target image-text pair corresponding to the manual annotation instruction is filtered out from the plurality of image-text pairs;

[0018] According to a text-image matching score in the manual annotation instruction, a first score of the target image-text pair is determined;

[0019] The data source of the target image-text pair is taken as a weight factor to perform weighted calculation on the first score to obtain a second score of the target image-text pair.

[0020] In the embodiments of the present application, an evaluation model of an initial generation model is trained based on the plurality of image-text pairs, including:

[0021] inputting text information in the image-text pair into the initial generation model to obtain a generated image;

[0022] calculating an evaluation value of the generated image to obtain evaluation values of a plurality of generated images corresponding to the plurality of image-text pairs;

[0023] training the evaluation model by using a preset loss function, wherein the preset loss function is used to maximize the sum of the evaluation values of the plurality of generated images.

[0024] In the embodiments of the present application, the evaluation value of the generated image is calculated, comprising:

[0025] comparing the generated image with image information in the image-text pair to obtain an image quality score;

[0026] calculating a semantic consistency score of a first feature vector of the generated image and a second feature vector of the text information in the image-text pair;

[0027] According to the image quality score, the first weight coefficient corresponding to the image quality score, the semantic consistency score, and the second weight coefficient corresponding to the semantic consistency score, the evaluation value of the generated image is calculated.

[0028] In the embodiments of the present application, the method further comprises generating triple information of the original input text by the following steps:

[0029] performing part-of-speech tagging on each word in the original input text, wherein the part-of-speech tagging is used to identify the grammatical role of each word in a sentence;

[0030] constructing a syntax tree by taking each word in the original input text as a node and taking the relationship between words as a directed edge between nodes;

[0031] filtering predicate information from the syntax tree, and filtering subject information and object information associated with the predicate information from context nodes of the predicate information;

[0032] generating the triple information by using the predicate information, the subject information, and the object information.

[0033] In the embodiments of the present application, the model parameters include a first low-rank matrix and a second low-rank matrix decomposed from a weight matrix of the initial generation model;

[0034] optimizing the model parameters of the initial generation model according to the evaluation value of the target image, comprising:

[0035] adjusting model parameters of the initial generative model from the first low-rank matrix to a target first low-rank matrix according to a first gradient and a preset learning rate; the first gradient refers to a gradient of the evaluation value with respect to the first low-rank matrix;

[0036] adjusting model parameters of the initial generative model from the second low-rank matrix to a target second low-rank matrix according to a second gradient and the preset learning rate; the second gradient refers to a gradient of the evaluation value with respect to the second low-rank matrix.

[0037] Embodiments of the second aspect of the application provide an image generative model training device, comprising:

[0038] an image-text pair definition module configured to predefine a plurality of image-text pairs; image information in each image-text pair matches text information;

[0039] an evaluation model training module configured to train an evaluation model of an initial generative model based on the plurality of image-text pairs; the evaluation model is configured to evaluate image quality and semantic consistency of an output image of the initial generative model;

[0040] a vector fusion module configured to encode triad information of an original input text to obtain a first text feature vector, encode the original input text to obtain a second text feature vector, and fuse the first text feature vector and the second text feature vector to obtain a fused text feature vector; the original input text refers to text information in any image-text pair; the triad information includes subject text information, predicate text information and object text information;

[0041] a target image generation module configured to input the fused text feature vector into the initial generative model to obtain a target image; the initial generative model is configured to perform multi-step image processing on a random noise image to obtain the target image; the image processing includes image denoising and image evaluation; image denoising at each step is based on a denoised image at a previous step and the fused text feature vector; image evaluation at each step refers to evaluating image quality and semantic consistency of a denoised image at a current step by the evaluation model to obtain an evaluation value; the evaluation value of the target image is greater than a preset evaluation threshold;

[0042] a model optimization module configured to optimize model parameters of the initial generative model according to the evaluation value of the target image to obtain a trained image generative model.

[0043] The embodiment of the third aspect of the present application provides a computer device, comprising a memory and a processor, which are connected in communication with each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the image generation model training method in the first aspect.

[0044] The embodiment of the fourth aspect of the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to make a computer execute the image generation model training method in the first aspect.

[0045] Additional aspects and advantages of the present application will be made apparent by the following description of the preferred embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0046] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are intended to illustrate preferred embodiments of the present application, and should not be considered limiting of the present application. Indeed, the drawings themselves can comprise embodiments of the present application that can possibly be realized in the absence of corresponding text in the specification. Like reference numerals are used to designate corresponding parts throughout the accompanying drawings.

[0047] Figure 1 A flowchart of an image generation model training method provided by an embodiment of the present application is shown;

[0048] Figure 2 A flowchart of an evaluation model training method provided by an embodiment of the present application is shown;

[0049] Figure 3 A flowchart of another image generation model training method provided by an embodiment of the present application is shown;

[0050] Figure 4 A flowchart of model iterative training provided by an embodiment of the present application is shown;

[0051] Figure 5 A structural diagram of an image generation model training device provided by an embodiment of the present application is shown;

[0052] Figure 6 A structural diagram of a computer device provided by an embodiment of the present application is shown;

[0053] Figure 7 A structural diagram of a storage medium provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0054] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it is understood that the present application can be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0055] It should be noted that unless otherwise specified, technical or scientific terms used in the present application should be understood as their common meaning to those skilled in the art to which the present application pertains.

[0056] The technical scenarios related to the embodiments of the present application are described below.

[0057] Text-to-image technology, as a kind of multi-modal technology, has made remarkable progress. The initial image generation technology mainly relies on hand-designed features and rules, and the quality and diversity of the generated images are limited. With the rise of deep learning, especially the emergence of generative adversarial networks and diffusion models, image generation technology has been greatly improved. These models can generate high-quality and diverse images by learning a large amount of image data.

[0058] The core of text-to-image technology is to convert natural language text into visual images, and this process requires the model to not only understand the surface meaning of the text, but also capture the deep semantics and contextual relationships. Many existing methods perform well when dealing with simple descriptions, but they often struggle when faced with complex semantic interactions. This is because most methods mainly focus on the surface features of the text, such as contours, colors, etc., while ignoring the implicit semantic interactions in the text. This neglect leads to a lack of semantic consistency and details in the generated images. For example, when the text description involves multiple objects and their interactions, existing models may not accurately capture the relationships between these objects, resulting in a lack of semantic accuracy in the generated images.

[0059] To address these issues, new methods need to better understand the semantic interactions in the text and effectively utilize this information during image generation. Limited by existing training loss functions that cannot supervise such highly detailed semantic associations, more accurate semantic understanding modules and reward models need to be designed to provide more precise feedback and guidance during the generation process. By introducing a semantic understanding module, the model can extract the subject-predicate-object structure in the text, identify the objects described in the text and their relationships, and thus better maintain semantic consistency during image generation. In addition, the introduction of a reward model can evaluate the quality and semantic consistency of the generated image during the generation process and provide feedback to guide the model to optimize. In this way, the model can continuously adjust and improve during the generation process, ultimately generating high-quality images that better match the text description. This method not only improves the visual quality of the image but also enhances its semantic consistency, making the generated image more accurately reflect the complex semantic relationships in the text.

[0060] According to an embodiment of the present application, an image generation model training method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown.

[0061] In this embodiment, an image generation model training method is provided, Figure 1 is a flowchart of the image generation model training method according to an embodiment of the present application, as Figure 1 shown, the flow includes the following steps:

[0062] Step S101, predefine a plurality of image-text pairs.

[0063] Specifically, each image-text pair contains an image information and a text information matching the image information, which can be understood as the object mentioned in the text information has a corresponding visual presentation in the image information. For example: the text information is described as "a black cat sitting on the sofa", then the image information should have a black cat, and it should be sitting on the sofa.

[0064] In some specific embodiments, the above step S101 includes steps S1011-S1014:

[0065] Step S1011, obtain a plurality of image-text pairs.

[0066] Specifically, each image-text pair includes an image and corresponding text information. The image and text information in these pairs may have a high degree of matching or a low degree of matching. These image-text pairs may originate from reliable generative models or manually drawn models, or they may originate from low-parameter, small models with low reliability; therefore, they need to be screened. During the process of obtaining image-text pairs, for example... Figure 2 As shown, text information can be obtained first through online web resources, and then image information corresponding to the text information can be obtained through pre-trained generative models (such as diffusion-based generative models (SD, Stable Diffusion)) or publicly available image-text pair datasets.

[0067] Step S1012: For any image-text pair, the text information in the image-text pair is encoded to obtain a text feature vector, and the image information in the image-text pair is encoded to obtain an image feature vector.

[0068] Specifically, such as Figure 2 As shown, a text feature vector can be obtained by text encoding the text information in a text-image pair using a text encoder, and an image feature vector can be obtained by image encoding the image information in a text-image pair using an image encoder.

[0069] Step S1013: Calculate the similarity between the text feature vector and the image feature vector. If the similarity is greater than a preset similarity threshold, then determine that the text information and the image information match.

[0070] Specifically, similarity can be understood as cosine similarity, and the text feature vector is calculated using the following formula. A and the image feature vector B cosine similarity :

[0071]

[0072] More specifically, the preset similarity threshold can be set according to the actual situation, and no specific limitation is made here.

[0073] Step S1014: Select the multiple image-text pairs from the multiple image-text pairs based on the degree of matching between the text information and the image information.

[0074] Specifically, the matching degree can be understood as the following two: meeting the matching requirement and not meeting the matching requirement, wherein the meeting the matching requirement refers to the similarity between the text information and the image information being greater than a preset similarity threshold, and the not meeting the matching requirement refers to the similarity between the text information and the image information being less than or equal to the preset similarity threshold. That is to say: when the similarity is greater than the preset similarity threshold, it can be determined that the text information and the image information are matched, and they can be taken as an image-text pair; on the contrary, when the similarity is less than or equal to the preset similarity threshold, it can be determined that the text information and the image information are not matched, and they cannot be taken as an image-text pair. Thus, the matched image-text pairs are filtered from the multiple image-text pairs in this way.

[0075] In some specific embodiments, after obtaining the multiple image-text pairs, the method comprises steps a1-a3:

[0076] Step a1, in response to the manual annotation instruction, a target image-text pair corresponding to the manual annotation instruction is filtered from the multiple image-text pairs.

[0077] Step a2, a first score of the target image-text pair is determined according to the image-text matching score in the manual annotation instruction.

[0078] Step a3, the data source of the target image-text pair is taken as a weight factor to weight the first score to obtain a second score of the target image-text pair.

[0079] In the embodiments of the present application, the image-text label of the target image-text pair and the image-text matching score of the target image-text pair are contained in the manual annotation instruction sent by the user end; wherein the image-text label can be used to filter the target image-text pair corresponding to the image-text label from the multiple image-text pairs; the image-text matching score can be understood as the score of manually judging whether the image information and the text information in the target image-text pair match.

[0080] In the embodiments of the present application, the data source includes but is not limited to: a high-confidence generative model or manual drawing, and a low-confidence low-parameter small model. When the data source is from a high-confidence generative model or manual drawing, it can be further determined that the image information and the text information in the target image-text pair match, i.e. the score is improved, i.e. the second score is greater than the first score; on the contrary, when the data source is from a low-confidence low-parameter small model, it can be further determined that the image information and the text information in the target image-text pair do not match, i.e. the score is reduced, i.e. the second score is less than the first score.

[0081] Step S102, training an evaluation model of an initial generative model based on the multiple image-text pairs.

[0082] Specifically, the evaluation model is used to evaluate the image quality and semantic consistency of the output image of the initial generation model. Wherein, the initial generation model is used to output a generated image corresponding to the text description; the semantic consistency can be understood as the matching degree of the generated image in semantics with the text description, that is, whether the visual content presented in the image can accurately reflect the semantic information in the text description.

[0083] In some specific embodiments, the above step S102 includes steps S1021-S1023:

[0084] Step S1021, for any image-text pair in the plurality of image-text pairs, input the text information in the image-text pair into the initial generation model to obtain a generated image.

[0085] Step S1022, calculate the evaluation value of the generated image, to obtain a plurality of evaluation values of the generated images corresponding to the plurality of image-text pairs.

[0086] Specifically, the evaluation value of the generated image can be calculated by calculating the image quality score and the semantic consistency score of the generated image.

[0087] In some specific embodiments, the above step S1022 further includes steps b1-b3:

[0088] Step b1, compare the generated image with the image information in the image-text pair to obtain an image quality score.

[0089] Specifically, a convolutional neural network can be used, and a pre-trained multi-modal deep learning model (CLIP, Contrastive Language-Image Pre-training) can be used as a feature extractor to extract the image feature vector of the generated image and the image feature vector of the image information in the image-text pair; and the feature distance between the two image feature vectors is calculated as the image quality score. Wherein, the feature distance can be calculated by the following formula:

[0090]

[0091] Wherein, represents the feature distance between the two image feature vectors, that is, the image quality score; represents the image feature vector of the generated image, represents the image feature vector of the image information in the image-text pair, represents the Euclidean distance. By minimizing the feature distance, the evaluation model can identify the generated result closer to the high-quality image (i.e. the image information in the above image-text pair).

[0092] Step b2, calculating a semantic consistency score of the first feature vector of the generated image and a second feature vector of the text information in the image-text pair.

[0093] Specifically, the first feature vector can be obtained by image encoding the generated image through an image encoder based on a multi-modal deep learning model (CLIP, Contrastive Language-Image Pre-training), and the second feature vector can be obtained by text encoding the text information in the image-text pair through a text encoder based on the multi-modal deep learning model (CLIP, Contrastive Language-Image Pre-training); and the cosine similarity of the first feature vector and the second feature vector is calculated by the following formula, and the cosine similarity is taken as the semantic consistency score:

[0094]

[0095] wherein, is the first feature vector of the generated image, is the second feature vector of the text information in the image-text pair, denotes the norm of the vector. By maximizing the cosine similarity, it is evaluated that the model can identify the generated image that is semantically consistent with the input text description.

[0096] Step b3, calculating the evaluation value of the generated image according to the image quality score, the first weight coefficient corresponding to the image quality score, the semantic consistency score, and the second weight coefficient corresponding to the semantic consistency score.

[0097] Specifically, the evaluation value of the generated image can be calculated by the following formula:

[0098]

[0099] wherein, denotes the image quality score; denotes the semantic consistency score; denotes the first weight coefficient corresponding to the image quality score, denotes the second weight coefficient corresponding to the semantic consistency score, and are used to measure the importance of image quality and semantic consistency.

[0100] Step S1023, obtaining the evaluation model by model training through a preset loss function; wherein the preset loss function is used to maximize the sum of the evaluation values of the plurality of generated images.

[0101] Specifically, during the training process, the parameters of the evaluation model are optimized through backpropagation and gradient descent algorithm. Specifically, the following loss function is minimized:

[0102]

[0103] wherein, is the number of image-text pairs, is the th generated image, denotes the sum of evaluation values of the plurality of generated images. By minimizing the loss function (i.e., maximizing the sum of evaluation values of the plurality of generated images), the evaluation model can gradually improve the evaluation accuracy of the quality and semantic consistency of the generated images.

[0104] In some specific embodiments, in order to prevent overfitting, L2 regularization is introduced. L2 regularization limits the complexity of the model by adding the sum of squares of weights in the loss function:

[0105]

[0106] wherein, is the regularization coefficient, is the parameter of the model, is the total number of parameters. Through regularization, the evaluation model can maintain good performance on different data sets.

[0107] In the embodiments of the present application, after the training is completed, the evaluation model can predict the quality of unseen images, which is used to guide the optimization of the generation model subsequently. Specifically, the initial generation model will evaluate the intermediate generated results (i.e., the denoised images at each step below) in the process of generating images through the evaluation model, and adjust the generation strategy according to the evaluation value.

[0108] In some specific embodiments, before step S103, the method further comprises steps c1-c4:

[0109] Step c1, part-of-speech tagging is performed on each word in the original input text.

[0110] Specifically, part-of-speech tagging is used to identify the grammatical role of each word in a sentence. By analyzing each word in a sentence, its grammatical role in the sentence is determined. For example, a word can be a noun, a verb, an adjective, an adverb, a preposition, etc. Part-of-speech tagging can better understand the structure and semantics of a sentence.

[0111] More specifically, part-of-speech tagging of each word can be achieved by a POS tagger.

[0112] Step c2, constructing a syntax tree by taking each word in the original input text as a node and taking the relationship between words as a directed edge between nodes.

[0113] Specifically, the dependency relationship between words in a sentence (i.e., the directed edge between the above nodes) can be identified by a grammar analysis tool (Stanford Parser) of the sentence, for example, in the sentence "Xiaoming likes running", "likes" is a verb, "Xiaoming" is the subject, and "running" is the object, and there is a dependency relationship between them.

[0114] Step c3, filtering predicate information from the syntax tree, and filtering subject information and object information associated with the predicate information from the context nodes of the predicate information.

[0115] Specifically, the predicate verb (V) is filtered from the syntax tree, and the subject (S) and object (O) directly related to the predicate verb (V) are identified, where direct relationship can be understood as filtering subject information and object information associated with predicate information from the context nodes of the predicate information.

[0116] Step c4, generating the triple information by the predicate information, the subject information and the object information.

[0117] Step S103, encoding the triple information of the original input text to obtain a first text feature vector, encoding the original input text to obtain a second text feature vector, and fusing the first text feature vector and the second text feature vector to obtain a fused text feature vector.

[0118] Specifically, for example Figure 3 As shown: the original input text "a woman is holding a microphone and giving a speech" is text encoded by a text encoder to obtain a second text feature vector, and the triple information of the original input text "<woman [perform] speech>, <woman [hold] microphone>" is text encoded by a text encoder to obtain a first text feature vector, and finally the first text feature vector and the second text feature vector are fused to obtain a fused text feature vector.

[0119] Similarly, when the original input text is in English, the text encoder can be used to text encode "A woman is holding a microphone and giving a speech" to obtain a second text feature vector, and text encode the triple information "woman [holding] microphone, woman [giving] speech" of the original input text to obtain a first text feature vector, and finally fuse the first text feature vector and the second text feature vector to obtain a fused text feature vector.

[0120] Specifically, the original input text refers to text information in any image-text pair; the triple information includes subject text information, predicate text information, and object text information.

[0121] More specifically, the first text feature vector is obtained by encoding as shown in the following formula:

[0122]

[0123] wherein Concat(·) represents a vector concatenation operation, and is used to concatenate the vector of the subject text information , the vector of the predicate text information , and the vector of the object text information to obtain the first text feature vector t. These vectors store semantic interaction relationships in the original input text.

[0124] The embodiments of the present application fuse the first text feature vector obtained by encoding the triple information and the second text feature vector obtained by encoding the original input text to obtain a fused text feature vector, and input the fused text feature vector into the initial generation model, so that the initial generation model can better understand semantic interaction information by using additional semantic information (i.e., the first text feature vector obtained by encoding the triple information).

[0125] In step S104, the fused text feature vector is input into the initial generation model to obtain a target image.

[0126] Specifically, the initial generation model is used to perform multi-step image processing on a random noise image to obtain the target image; the image processing includes image denoising and image evaluation; the image denoising at each step is based on the denoised image at the previous step and the fused text feature vector, and the image evaluation at each step refers to evaluating the image quality and semantic consistency of the denoised image at the current step by the evaluation model to obtain an evaluation value; the evaluation value of the target image is greater than a preset evaluation threshold.

[0127] In the embodiments of the present application, the initial generation model belongs to an improved diffusion model. In a normal diffusion model, a random noise image is gradually denoised to obtain a generated image. For example, a random noise image is denoised in the first step to obtain a denoised image 1, the denoised image 1 is denoised in the second step to obtain a denoised image 2, and so on, and the denoised image n is denoised in the n+1 step to obtain a target image. However, in the initial generation model of the present application, the image of each denoising step is evaluated, that is, the image quality and semantic consistency of the denoised image of the current step are evaluated by the evaluation model to obtain an evaluation value. If the evaluation value of the denoised image is greater than a preset evaluation threshold, the denoised image can be used as the target image. Otherwise, if the evaluation value of the denoised image is less than or equal to the preset evaluation threshold, the denoised image needs to be denoised again.

[0128] In some specific embodiments, the denoising of each step in the initial generation model is based on the denoised image of the previous step and the fused text feature vector. When the initial generation model denoises at each step, it reduces the noise in the denoised image of the previous step while preserving the image features related to the fused text feature vector. This mechanism enables the model to gradually generate high-quality images (for example Figure 3 “Image 1, Image 2, Image 3, Image 4, …, Image n”) in multiple iterations, while ensuring the consistency of the generated image with the text prompt.

[0129] In some specific embodiments, during the denoising process, the evaluation model evaluates that the details of a tree in the generated image are not clear enough, and the shape of the trunk and branches is somewhat blurred. In order to improve this, some controlled noise will be added to the area of the tree, and then the initial generation model enters a new round of denoising stage. In the new denoising stage, the initial generation model will use these noises to reconstruct the details of the tree, making the shape of the tree clearer and more natural. Through multiple iterations of this kind, the final generated image will be better in terms of details and semantic coherence.

[0130] In some specific embodiments, during the process of the initial generation model obtaining the target image through gradual denoising, the gradual denoising can be used to represent the transition from the current state to the previous state . This process is modeled by a Gaussian distribution, and the formula is:

[0131]

[0132] where, and are the mean and covariance, respectively, parameterizing the transition from to .

[0133] For the semantic encoding of the image before and after the different denoising steps in the above denoising process, the following distributions are defined:

[0134]

[0135]

[0136] where the known part (p) is the information obtained from the original input or prior knowledge, including the initial image features. The unknown part (q) is the content that needs to be predicted or generated by the model. Subsequently, to ensure the quality of the generated results, an evaluation model (f) is used to evaluate the generated q. The evaluation model gives a score according to the details and semantic consistency of the generated image. If the evaluation value (f(q)) is lower than the set threshold (f(q) < f), the re-diffusion mechanism is repeated to sample until the generated state satisfies the condition:

[0137]

[0138] If f(q) < f, the following steps are repeated: sampling q, generating q until f(q) > f. In addition, a gradient adjustment term is introduced to further optimize the generation strategy:

[0139]

[0140] where a is the learning rate, and g is the gradient of the reward model, used to guide the optimization of the generation process.

[0141] The embodiments of the present application give dynamic feedback in the denoising process through the evaluation model, so that the generated image meets the standard in terms of quality and semantic consistency. Preferably, compared with the denoising process in the prior art, which can only denoise to the last step to realize image generation, the present technical solution can filter out images with high quality and semantic consistency at an early stage by evaluating the image after each denoising step. If the denoised image at a certain step already meets the quality requirement (the evaluation value is greater than the preset evaluation threshold), it can be directly used as the target image, avoiding unnecessary subsequent denoising steps, thereby saving computing resources and ensuring the quality of the generated image.

[0142] Step S105, optimizing the model parameters of the initial generation model according to the evaluation value of the target image to obtain a trained image generation model.

[0143] Specifically, for example​​​​​​​​​​​Figure 4 As shown, the decoded image (i.e., the target image mentioned above) is used for LoRA (Low-Rank Adaptation) training to enhance the generative ability of the initial generative model. LoRA is a technique that optimizes neural network weights through low-rank decomposition, which can improve model performance without significantly increasing computational complexity. The LoRA training method is shown below:

[0144] The model parameters of the initial generative model refer to the first low-rank matrix obtained by decomposing the weight matrix of the initial generative model. A Second low-rank matrix B As shown in the following formula:

[0145]

[0146] in, Represents the original weight matrix. This represents the updated weight matrix. Denotes the first low-rank matrix. Let the second low-rank matrix satisfy... This LoRA training method can improve the model's adaptability and generative ability without significantly increasing computational complexity.

[0147] In some specific embodiments, step S105 above includes steps S1051-S1052:

[0148] Step S1051: Adjust the model parameters of the initial generated model from the first low-rank matrix to the target first low-rank matrix according to the first gradient and the preset learning rate.

[0149] Specifically, the first gradient refers to the gradient of the evaluation value with respect to the first low-rank matrix.

[0150] Step S1052: Adjust the model parameters of the initial generated model from the second low-rank matrix to the target second low-rank matrix according to the second gradient and the preset learning rate.

[0151] Specifically, the second gradient refers to the gradient of the evaluation value with respect to the second low-rank matrix.

[0152] In steps S1051-S1052, the gradient descent algorithm can be used to adjust the parameters of the low-rank matrix based on the feedback from the evaluation value.

[0153]

[0154]

[0155] in, a learning rate, a first gradient, a second gradient. Through this adaptive optimization process, the generation model can gradually improve the quality and detail performance of the generated images. The feedback signal of the evaluation model ensures that the generation model can dynamically adjust the parameters during the generation process to adapt to different input features and generation requirements.

[0156] The embodiments of the present application significantly improve the image generation capability of the image generation model under the input of text containing object interaction relationship by training the evaluation model based on the self-built data set (i.e., the above-mentioned pre-defined multiple image-text pairs) and the self-iterative training enhanced based on the semantic interaction relationship (i.e., multi-step image processing on random noise images). The method first constructs a high-quality semantic interaction image-text data set and evaluates and feeds back the generation results through the training of the evaluation model. Subsequently, a semantic interaction perception module (i.e., constructing a triple) is designed and a re-diffusion mechanism (i.e., evaluating the intermediate images in the denoising process through the trained evaluation model. When it is found that the generated image has possible defects in details or semantic coherence, a new round of denoising stage is applied. Through reciprocating iteration, the local detail information in the image is better expressed), and finally through self-iterative training, the model can continuously optimize the generation strategy and gradually improve the details and accuracy of the generated image. Compared with traditional methods, the present application not only can generate more realistic images, but also can effectively capture and enhance the semantic interaction relationship during the generation process, and at the same time, the generation effect is supervised in the denoising process. The method is suitable for various image generation application scenarios such as virtual reality, film production and intelligent design, and has wide application prospect and commercial value.

[0157] Corresponding to the implementation mode of the above image generation model training method, the embodiments of the present application also provide an image generation model training device for executing the image generation model training method described in the above embodiments. As shown in the Figure 5 image generation model training device includes:

[0158] An image-text pair definition module is configured to predefine a plurality of image-text pairs; the image information in each image-text pair matches the text information;

[0159] An evaluation model training module is configured to train an evaluation model of an initial generation model based on the plurality of image-text pairs; the evaluation model is configured to evaluate the image quality and semantic consistency of the output image of the initial generation model;

[0160] a vector fusion module configured to encode triadic information of an original input text to obtain a first text feature vector, encode the original input text to obtain a second text feature vector, and fuse the first text feature vector and the second text feature vector to obtain a fused text feature vector; the original input text refers to text information in any image-text pair; the triadic information includes subject text information, predicate text information, and object text information;

[0161] a target image generation module configured to input the fused text feature vector into the initial generation model to obtain a target image; the initial generation model is configured to perform multi-step image processing on a random noise image to obtain the target image; the image processing includes image denoising and image evaluation; image denoising at each step is performed based on a denoised image at a previous step and the fused text feature vector; image evaluation at each step refers to evaluating image quality and semantic consistency of a denoised image at a current step by using the evaluation model to obtain an evaluation value; the evaluation value of the target image is greater than a preset evaluation threshold;

[0162] a model optimization module configured to optimize model parameters of the initial generation model according to the evaluation value of the target image to obtain a trained image generation model.

[0163] Optionally, the image-text pair definition module is further configured to obtain a plurality of image-text pairs; each image-text pair includes image information and text information corresponding to the image information; for any image-text pair, the text information in the image-text pair is encoded to obtain a text feature vector, and the image information in the image-text pair is encoded to obtain an image feature vector; a similarity between the text feature vector and the image feature vector is calculated, and if the similarity is greater than a preset similarity threshold, it is determined that the text information and the image information match; the plurality of image-text pairs are filtered according to a matching degree of the text information and the image information to obtain the plurality of image-text pairs.

[0164] Optionally, the apparatus further includes an artificial annotation module configured to, in response to an artificial annotation instruction, filter a target image-text pair corresponding to the artificial annotation instruction from the plurality of image-text pairs; determine a first score of the target image-text pair according to a text-image matching score in the artificial annotation instruction; and calculate a second score of the target image-text pair by weighting the first score with a data source of the target image-text pair as a weight factor.

[0165] Optionally, the evaluation model training module is further configured to, for any image-text pair in the plurality of image-text pairs, input text information in the image-text pair into the initial generation model to obtain a generated image; calculate an evaluation value of the generated image to obtain a plurality of evaluation values of a plurality of generated images corresponding to the plurality of image-text pairs one by one; and train the evaluation model by using a preset loss function to maximize the sum of the plurality of evaluation values of the generated images.

[0166] Optionally, the evaluation model training module is further configured to compare the generated image with image information in the image-text pair to obtain an image quality score; calculate a semantic consistency score of a first feature vector of the generated image and a second feature vector of the text information in the image-text pair; and calculate the evaluation value of the generated image according to the image quality score, a first weight coefficient corresponding to the image quality score, the semantic consistency score, and a second weight coefficient corresponding to the semantic consistency score.

[0167] Optionally, the apparatus further comprises a triple generation module configured to perform part-of-speech tagging on each word in the original input text, the part-of-speech tagging being used to identify the grammatical role of each word in a sentence; construct a syntax tree by taking each word in the original input text as a node and taking the relationship between words as a directed edge between nodes; filter predicate information from the syntax tree, and filter subject information and object information associated with the predicate information from context nodes of the predicate information; and generate the triple information by using the predicate information, the subject information, and the object information.

[0168] Optionally, the model optimization module is further configured to adjust model parameters of the initial generation model from the first low-rank matrix to a target first low-rank matrix according to a first gradient and a preset learning rate, the first gradient being a gradient of the evaluation value with respect to the first low-rank matrix; and adjust model parameters of the initial generation model from the second low-rank matrix to a target second low-rank matrix according to a second gradient and the preset learning rate, the second gradient being a gradient of the evaluation value with respect to the second low-rank matrix.

[0169] The image generation model training apparatus provided by the above embodiments of the present application and the image generation model training method provided by the embodiments of the present application have the same beneficial effects as the method adopted, run, or implemented by the application program stored therein.

[0170] The embodiments of the present application further provide a computer device for executing the above image generation model training method. Please refer to Figure 6 , which shows a schematic diagram of a computer device provided by some embodiments of the present application. As shown inFigure 6 As shown, the computer device 6 comprises a processor 600, a memory 601, a bus 602 and a communication interface 603, the processor 600, the communication interface 603 and the memory 601 are connected through the bus 602; the memory 601 stores a computer program capable of running on the processor 600, and the processor 600 runs the computer program to execute the image generation model training method provided in the foregoing embodiments of the present application.

[0171] The memory 601 can include a high-speed random access memory (RAM) and can also include a non-volatile memory such as at least one disk memory. The communication between the system network element and at least one other network element is realized through at least one communication interface 603 (which can be wired or wireless), and the Internet, a wide area network, a local network, a metropolitan area network, etc. can be used.

[0172] The bus 602 can be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 601 is used to store programs, and the processor 600 executes the programs after receiving execution instructions. The image generation model training method disclosed in the foregoing embodiments can be applied to the processor 600 or implemented by the processor 600.

[0173] The processor 600 can be an integrated circuit chip with a processing capability of signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 600 or the instruction in the form of software. The processor 600 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a ready programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware coding processor for execution, or a combination of hardware and software modules in the coding processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 601, and the processor 600 reads the information in the memory 601, and combines the hardware to complete the steps of the above method.

[0174] The computer device provided by the embodiments of the present application and the image generation model training method provided by the embodiments of the present application have the same beneficial effects as the method adopted, run or implemented.

[0175] The embodiments of the present application also provide a computer readable storage medium corresponding to the image generation model training method provided by the preceding embodiments. Please refer to Figure 7 The computer readable storage medium shown is an optical disc 30, and a computer program (i.e. program product) is stored on the optical disc 30. When the computer program is run by a processor, the image generation model training method provided by any of the preceding embodiments is executed.

[0176] It should be noted that examples of the computer readable storage medium can also include, but are not limited to, a phase change memory (PRAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), other types of random access memory (RAM), a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a flash memory or other optical, magnetic storage medium, which will not be described one by one here.

[0177] The computer readable storage medium provided by the above embodiments of the present application has the same beneficial effects as the method adopted, run or implemented by the application program stored therein, based on the same inventive concept as the image generation model training method provided by the embodiments of the present application.

[0178] It should be noted that:

[0179] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

[0180] Similarly, it is to be understood that the various features of the application can be used in any combination and / or sub-combination, and are not limited to the specific embodiments described herein. In particular, the various features of the application can be used in any combination and / or sub-combination in the methods of the application.

[0181] Furthermore, those skilled in the art will recognize that boundaries between the functionality of the above described embodiments are merely illustrative and that the functionality of each can be combined in a single embodiment and / or distributed in additional embodiments. For example, the functionality of two or more of the above described embodiments can be combined in a single embodiment.

[0182] The above description is only the preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any changes or substitutions easily conceived by those skilled in the art within the technical scope disclosed by the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for training an image generation model, characterized in that, The method includes: Multiple image-text pairs are predefined; the image information in each image-text pair is matched with the text information. An evaluation model for the initial generation model is obtained by training based on the multiple image-text pairs; the evaluation model is used to evaluate the image quality and semantic consistency of the output image of the initial generation model. The first text feature vector is obtained by encoding the triple information of the original input text; the second text feature vector is obtained by encoding the original input text; and the first text feature vector and the second text feature vector are fused to obtain a fused text feature vector. The original input text refers to the text information in any image-text pair; the triple information includes subject text information, predicate text information, and object text information. The fused text feature vector is input into the initial generation model to obtain the target image; the initial generation model is used to perform multi-step image processing on the random noise image to obtain the target image; the image processing includes image denoising and image evaluation; each step of image denoising is based on the denoised image of the previous step and the fused text feature vector, and each step of image evaluation refers to evaluating the image quality and semantic consistency of the denoised image of the current step through the evaluation model to obtain an evaluation value; the evaluation value of the target image is greater than a preset evaluation threshold; The model parameters of the initial generation model are optimized based on the evaluation value of the target image to obtain a trained image generation model; An evaluation model for the initial generation model trained based on the multiple image-text pairs includes: For any one of the plurality of image-text pairs, the text information of the image-text pair is input into the initial generation model to obtain the generated image; Calculate the evaluation value of the generated image to obtain the evaluation values ​​of multiple generated images that correspond one-to-one with the multiple image-text pairs; The evaluation model is obtained by training the model using a preset loss function; wherein, the preset loss function is used to maximize the sum of the evaluation values ​​of the multiple generated images; Calculating the evaluation value of the generated image includes: The generated image is compared with the image information in the image-text pair to obtain an image quality score; Calculate the semantic consistency score between the first feature vector of the generated image and the second feature vector of the text information in the image-text pair; The evaluation value of the generated image is calculated based on the image quality score, the first weight coefficient corresponding to the image quality score, the semantic consistency score, and the second weight coefficient corresponding to the semantic consistency score.

2. The method according to claim 1, characterized in that, Several image-text pairs are predefined, including: Multiple image-text pairs are obtained; each image-text pair includes an image information and text information corresponding to the image information. For any image-text pair, the text information in the image-text pair is encoded to obtain a text feature vector, and the image information in the image-text pair is encoded to obtain an image feature vector; Calculate the similarity between the text feature vector and the image feature vector. If the similarity is greater than a preset similarity threshold, then determine that the text information and the image information match. The plurality of image-text pairs are selected from the plurality of image-text pairs based on the degree of matching between the text information and the image information.

3. The method according to claim 2, characterized in that, After acquiring multiple image-text pairs, the method includes: In response to a manual annotation instruction, a target image-text pair corresponding to the manual annotation instruction is selected from the plurality of image-text pairs; The first score of the target image text pair is determined based on the image-text matching score in the manual annotation instruction; The data source of the target image-text pair is used as a weighting factor to calculate the weighted score of the first score, thereby obtaining the second score of the target image-text pair.

4. The method according to claim 1 or 2, characterized in that, The method further includes: generating triplet information of the original input text through the following steps: Each word in the original input text is tagged with part-of-speech tags, which are used to identify the grammatical role of each word in the sentence. Each word in the original input text is treated as a node, and the relationships between words are treated as directed edges between nodes to construct a syntax tree; Predicate information is filtered from the syntax tree, and subject and object information associated with the predicate information are filtered from the context nodes of the predicate information. The triplet information is generated using the predicate information, the subject information, and the object information.

5. The method according to claim 1 or 2, characterized in that, The model parameters include: a first low-rank matrix and a second low-rank matrix obtained by decomposing the weight matrix of the initial generated model; Optimizing the model parameters of the initial generation model based on the evaluation value of the target image includes: The model parameters of the initially generated model are adjusted from the first low-rank matrix to the target first low-rank matrix based on the first gradient and the preset learning rate; the first gradient refers to the gradient of the evaluation value with respect to the first low-rank matrix; The model parameters of the initially generated model are adjusted from the second low-rank matrix to the target second low-rank matrix based on the second gradient and the preset learning rate; the second gradient refers to the gradient of the evaluation value with respect to the second low-rank matrix.

6. An image generation model training device, characterized in that, The device includes: The image-text pair definition module is used to predefine multiple image-text pairs; the image information in each image-text pair is matched with the text information. An evaluation model training module is used to evaluate the initial generation model trained based on the multiple image-text pairs; the evaluation model is used to evaluate the image quality and semantic consistency of the output image of the initial generation model. The vector fusion module is used to encode the triple information of the original input text to obtain a first text feature vector, encode the original input text to obtain a second text feature vector, and fuse the first text feature vector and the second text feature vector to obtain a fused text feature vector; the original input text refers to the text information in any image-text pair; the triple information includes subject text information, predicate text information, and object text information; A target image generation module is used to input the fused text feature vector into the initial generation model to obtain a target image; the initial generation model is used to perform multi-step image processing on a random noise image to obtain the target image; the image processing includes image denoising and image evaluation; each step of image denoising is based on the denoised image of the previous step and the fused text feature vector, and each step of image evaluation refers to evaluating the image quality and semantic consistency of the denoised image of the current step through the evaluation model to obtain an evaluation value; the evaluation value of the target image is greater than a preset evaluation threshold; The model optimization module is used to optimize the model parameters of the initial generation model based on the evaluation value of the target image, so as to obtain a trained image generation model; An evaluation model for the initial generation model trained based on the multiple image-text pairs includes: For any one of the plurality of image-text pairs, the text information of the image-text pair is input into the initial generation model to obtain the generated image; Calculate the evaluation value of the generated image to obtain the evaluation values ​​of multiple generated images that correspond one-to-one with the multiple image-text pairs; The evaluation model is obtained by training the model using a preset loss function; wherein, the preset loss function is used to maximize the sum of the evaluation values ​​of the multiple generated images; Calculating the evaluation value of the generated image includes: The generated image is compared with the image information in the image-text pair to obtain an image quality score; Calculate the semantic consistency score between the first feature vector of the generated image and the second feature vector of the text information in the image-text pair; The evaluation value of the generated image is calculated based on the image quality score, the first weight coefficient corresponding to the image quality score, the semantic consistency score, and the second weight coefficient corresponding to the semantic consistency score.

7. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the image generation model training method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the image generation model training method according to any one of claims 1 to 5.

Images