Figure generation model training method, image generation method, equipment, medium and product
By extracting entity and attribute information in literary graph model training and introducing comparative learning loss, the problem that Unet model is difficult to capture text description information is solved, and the accuracy of image generation is improved.
Patent Information
- Application Number
- CN202411807056.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-09
- Publication Date
- 2025-04-11
AI Technical Summary
In the literary and graphic model, the Unet model is difficult to accurately capture the guiding information in the text description, resulting in poor accuracy of the generated images.
During the model training process, the entity and attribute information in the prompt word sample is extracted, and the diffusion model is adjusted by comparing the learning loss, so that the model can compare the attention map of the positive and negative sample pairs, reduce information pollution, and improve the accuracy of the attention map.
By introducing comparative learning losses, the model can more accurately capture the guiding information in text descriptions, avoid the loss of entity or attribute information, and improve the accuracy of image generation.
Smart Images
Figure CN119810614A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a text-based graph model training method, image generation method, device, medium and product. Background Art
[0002] The text-based graph model can analyze the input text description, extract key information and features, and then convert this information into visual elements to generate images.
[0003] In related technologies, text-generated graph models are usually implemented based on diffusion models. The Unet model in the diffusion model performs interactive attention learning on the encoded text features and the encoded image features to generate corresponding images based on the text description. However, since text features and image features belong to different modalities, there are natural differences between the two, which makes it difficult for the attention graph obtained by the Unet model to accurately capture the guiding information in the text description, which in turn leads to poor accuracy in the image generated by the model. Summary of the invention
[0004] In view of the above problems, the embodiments of the present application provide a cultural graph model training method, image generation method, device, medium and product to overcome the above problems or at least partially solve the above problems.
[0005] In a first aspect of an embodiment of the present application, a method for training a cultural graph model is provided, the method comprising:
[0006] When using the image-text annotation dataset to train the diffusion model, extracting entity and attribute information from each prompt word sample, the image-text annotation dataset includes each prompt word sample and its corresponding image sample;
[0007] Extracting the attention map corresponding to each entity and the attention map corresponding to each attribute information from the attention maps generated by the Unet model in the diffusion model for each prompt word sample and its corresponding image sample;
[0008] For each of the prompt word samples, a combination of an entity in the prompt word sample and the attribute information of the entity itself is determined as a positive sample pair, and a combination of different entities, a combination of different attribute information, and a combination of an entity and attribute information of other entities in the prompt word sample are respectively determined as negative sample pairs;
[0009] Determine the contrastive learning loss corresponding to each of the prompt word samples according to the similarities of the attention maps of each positive sample pair and the similarities of the attention maps of each negative sample pair associated with each of the prompt word samples, wherein the contrastive learning loss is positively correlated with the similarities of the attention maps of the positive sample pairs, and the contrastive learning loss is negatively correlated with the similarities of the attention maps of the negative sample pairs;
[0010] The parameters of the diffusion model are adjusted according to the contrastive learning losses corresponding to the respective prompt word samples to obtain a trained text graph model.
[0011] A second aspect of the embodiments of the present application provides an image generation method, the method comprising:
[0012] Determine a text description input by a user as a prompt word, wherein the text description is used to characterize the content required to be included in the image to be generated;
[0013] Inputting the prompt word into a pre-trained text image model to generate an image corresponding to the prompt word;
[0014] The pre-trained Vincent graph model is obtained by the Vincent graph model training method as described in the first aspect.
[0015] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the Vincent graph model training method as described in the first aspect are implemented, or the steps of the image generation method as described in the second aspect are implemented.
[0016] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the Wensheng graph model training method as described in the first aspect are implemented, or the steps of the image generation method as described in the second aspect are implemented.
[0017] In a fifth aspect of an embodiment of the present application, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the Wensheng graph model training method as described in the first aspect, or implements the steps of the image generation method as described in the second aspect.
[0018] The embodiments of the present application include the following advantages: during the training process of the diffusion model, the entity and attribute information in each prompt word sample are extracted, the combination of the entity and its own attribute information is used as a positive sample pair, the combination of different entities, the combination of different attribute information, and the combination of the entity and the attribute information of other entities are used as negative sample pairs, and then the parameters of the diffusion model are adjusted based on the contrastive learning loss, so that the model can perform contrastive learning on the attention maps of the positive and negative sample pairs, ensuring that the relevant attention maps determined by the model for the positive sample pairs can be close to each other, and the relevant attention maps determined for the negative sample pairs can be far away from each other, thereby reducing the information pollution on the attention map, so that the attention map obtained by the model can accurately capture the guiding information in the text description (such as the prompt word sample), avoiding the loss of entity or attribute information or inaccurate number of entities in the model-generated image, thereby improving the accuracy of the model-generated image. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0020] Figure 1 is an implementation flow chart of a method for training a text graph model in an embodiment of the present application;
[0021] Figure 2 is a schematic diagram of a model training process in an embodiment of the present application;
[0022] Figure 3 is an implementation flow chart of an image generation method in an embodiment of the present application;
[0023] Figure 4 It is a structural schematic diagram of a cultural graph model training device in an embodiment of the present application;
[0024] Figure 5 is a structural schematic diagram of an image generating device in an embodiment of the present application;
[0025] Figure 6 It is a schematic diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION
[0026] To facilitate understanding of the technical solutions provided by the present application, the main technical concepts involved in the embodiments of the present application are briefly described below.
[0027] Stable Diffusion model (SD): A commonly used Wensheng diagram model, including a series of representative models such as SD1.5, SD2.0, and SDXL.
[0028] Unet model: A model structure that uses a symmetrical U-shaped architecture and jump connections, which can effectively capture the global and local features of the image.
[0029] Contrastive Learning: An unsupervised learning method that aims to learn feature representations by comparing similar and dissimilar data samples.
[0030] Text-to-Image Generation: The process of using artificial intelligence technology (such as text-to-image model) to generate corresponding images based on text descriptions.
[0031] The text-generated graph model can analyze the input text description, extract key information and features, and then convert this information into visual elements to generate images that match the text description. This technology is widely used in art creation, advertising design, game development and other fields, and can provide creators with new inspiration and possibilities. With the advancement of technology, the quality and detail of images generated by the text-generated graph model are constantly improving, and it has become an important research direction in the intersection of computer vision and natural language processing.
[0032] In related technologies, text-based graph models are usually implemented based on diffusion models, which mainly include three parts: text encoder, Unet model and image encoder-decoder. The text encoder is used to convert natural language (such as text descriptions entered by users) into text features to guide the model to generate accurate images; the Unet model is used to perform interactive attention learning on the encoded text features and the encoded image features; the image encoder-decoder is used to encode images into latent space features and decode them into images.
[0033] It is understandable that the bridging role of the Unet model is very critical when performing text-to-image tasks based on the diffusion model. However, since text features and image features belong to different modalities, there are natural differences between the two, which leads to the attention map (also called interactive attention map or cross-attention map) obtained by the Unet model based on interactive attention learning. It is difficult to accurately capture the guiding information in the text description, which will lead to various problems. For example, when there are multiple generation targets (i.e., multiple entities) in the text description, the model-generated image loses some generation targets, or the model-generated image loses some attribute information of the generation target, or the number of entities in the model-generated image does not match the number of entities corresponding to the text description, resulting in poor accuracy of the model-generated image.
[0034] In response to the problems existing in the above-mentioned related technologies, the embodiments of the present application propose a text graph model training method, image generation method, device, medium and product. By introducing contrastive learning loss in the model training process, the model can perform contrastive learning on the attention maps of positive and negative sample pairs, thereby ensuring that the relevant attention maps determined by the model for positive sample pairs can be close to each other, and the relevant attention maps determined for negative sample pairs can be far away from each other. This can reduce the information pollution on the attention map, so that the attention map obtained by the model can accurately capture the guiding information in the text description, thereby improving the accuracy of the image generated by the model.
[0035] In combination with the accompanying drawings, a text-based graph model training method, image generation method, device, medium and product provided in the embodiments of the present application are described in detail through some embodiments and their application scenarios.
[0036] First, refer to Figure 1 FIG. 1 is a flowchart of an implementation of a method for training a text graph model provided in an embodiment of the present application, and the method includes the following steps:
[0037] Step S11: when using the image-text annotation dataset to train the diffusion model, extracting the entity and attribute information in each prompt word sample, the image-text annotation dataset includes each prompt word sample and its corresponding image sample.
[0038] In specific implementation, based on part-of-speech analysis, entities (i.e., generated targets, also called subjects) and attribute information of entities in each prompt word sample can be extracted. Specifically, each noun in the prompt word sample can be extracted to obtain each entity, and each adjective in the prompt word sample can be extracted to obtain each attribute information. Taking the prompt word sample "a green schoolbag and a red hat" as an example, by extracting each noun in the prompt word sample, the entities "schoolbag" and "hat" can be obtained, and by extracting each adjective in the prompt word sample, the attribute information "green" of the entity "schoolbag" and the attribute information "red" of the entity "hat" can be obtained.
[0039] Step S12: extracting the attention map corresponding to each entity and the attention map corresponding to each attribute information from the attention maps generated by the Unet model in the diffusion model for each prompt word sample and its corresponding image sample.
[0040] In the specific implementation, for each prompt word sample, tools such as the attention stop class can be used to automatically cache the attention map generated by the Unet model based on the text features corresponding to each word segment (such as entity or attribute information) in the prompt word sample and the image features corresponding to the corresponding image sample. Subsequently, the attention map corresponding to each entity and the attention map corresponding to each attribute information can be extracted from the cache.
[0041] It should be noted that, refer to Figure 2 The schematic diagram of the model training process shown is that when the diffusion model is trained using the image and text annotation dataset (which can be a dataset with billions of annotations), the image codec (such as an autoencoder) in the diffusion model will encode each image sample in the image and text annotation dataset into an image latent space feature (i.e., image feature) and decode it into an image, and the text encoder in the diffusion model will encode each prompt word sample in the image and text annotation dataset into a text latent space feature (i.e., text feature), and then the Unet model in the diffusion model will perform interactive attention learning on the image features and the text features.
[0042] For example, the diffusion model uses a variational auto-encoder (VAE) as an image encoder to convert an image sample x into a feature latent space, that is: z = ε(x), where z represents the image feature (which is a low-dimensional latent space feature), ε represents the image encoder, and all subsequent operations will be performed in the latent space (that is, the feature latent space). Specifically, the diffusion model uses a large language model (such as the CLIP model) as a text encoder to encode the prompt word sample T corresponding to the image sample x into the corresponding text feature c(T). The text feature is in the same latent space as the image feature and interacts with the image feature in the Unet model of the diffusion model. After completing the operation on the latent space features, use the image decoder Transform feature z into Among them, x ′ represents the reconstructed image (i.e., the model generates the image), and the training goal of the diffusion model is to make x≈x′.
[0043] In the above training process, the Unet model interacts between image features and text features through the cross-attention module. Specifically, the Unet model generates attention maps for the text features corresponding to each word in the prompt word sample based on the image features, and then obtains the output after the cross-attention operation on the image features and the text features based on the attention map, and then determines the noise predicted by the model based on the output.
[0044] Exemplarily, the attention map M can be determined by the following formula:
[0045]
[0046] Where, Q = X1W Q ,K=X2W K , X1 represents the text feature corresponding to a certain word segmentation, X2 represents the image feature, Weight Parameters n represents the number of tokens, d represents the feature dimension, and d ′ represents the attention dimension, and softmax represents the normalized exponential function.
[0047] Furthermore, the output Y after the cross-attention operation on the text feature X1 and the image feature X2 can be determined by the following formula:
[0048] Y=MV
[0049] Where V = X2W V , weight parameter d represents the feature dimension and M represents the attention map.
[0050] Step S13: For each of the prompt word samples, a combination of the entity in the prompt word sample and the attribute information of the entity itself is determined as a positive sample pair, and a combination of different entities, a combination of different attribute information, and a combination of an entity and attribute information of other entities in the prompt word sample are respectively determined as negative sample pairs.
[0051] Exemplarily, for the prompt word sample "a green schoolbag and a red hat", the combination of the entity "schoolbag" and its attribute information "green" in the prompt word sample (corresponding to the combination of the attribute information of the entity and the entity itself) is a positive sample pair, the combination of the entity "schoolbag" and the entity "hat" in the prompt word sample (corresponding to the combination of different entities) is a negative sample pair, the combination of the attribute information "green" and the attribute information "red" in the prompt word sample (corresponding to the combination of different attribute information) is a negative sample pair, and the combination of the entity "schoolbag" and the attribute information "red" in the prompt word sample (corresponding to the combination of the attribute information of the entity and other entities) is a negative sample pair.
[0052] Step S14: Determine the contrastive learning loss corresponding to each of the prompt word samples according to the attention map similarities of each positive sample pair and the attention map similarities of each negative sample pair associated with each of the prompt word samples, wherein the contrastive learning loss is positively correlated with the attention map similarity of the positive sample pair, and the contrastive learning loss is negatively correlated with the attention map similarity of the negative sample pair.
[0053] In a specific implementation, the contrastive learning loss corresponding to the relevant prompt word samples can be obtained by subtracting or dividing the attention map similarity (or its weighted sum) of each positive sample pair from the attention map similarity (or its weighted sum) of each negative sample pair.
[0054] Step S15: adjusting the parameters of the diffusion model according to the contrastive learning loss corresponding to each of the prompt word samples to obtain a trained text graph model.
[0055] In specific implementation, the contrastive learning loss and model loss (such as the model loss of the diffusion model or the model loss of the Unet model, etc.) corresponding to each prompt word sample can be combined by weighted sum or other methods, and then the minimization of the combined loss value is used as the training goal, and the parameters of the relevant model (such as the diffusion model as a whole or the part of the diffusion model that at least includes the Unet model) are adjusted. After the parameter adjustment of the relevant model is completed, the diffusion model including the relevant model is used as the trained Vincent graph model.
[0056] It can be understood that in view of the problem that the interactive attention map of the Unet model in the current text graph model is difficult to accurately capture the guiding information in the text description, during the training process of the model, the present application extracts entities and attribute information of entities in the text description (such as prompt word samples), takes the entity and its own attribute information as positive sample pairs, and takes the combination of the attributes of the entity with other entities, the combination between different entities, and the combination of attribute information of different entities as negative sample pairs, and accordingly introduces contrastive learning loss to constrain the cross-attention operation of the Unet model, thereby ensuring that the positive sample pairs are close to each other (that is, ensuring that the similarity of the relevant attention maps determined by the Unet model is high), and the negative sample pairs are far away from each other (that is, ensuring that the similarity of the relevant attention maps determined by the Unet model is low), thereby reducing the information pollution on the attention map, thereby improving the accuracy of the model-generated image.
[0057] Optionally, the trained text graph model can be applied to at least the following scenarios:
[0058] 1. Artistic creation: Artists can use the Wenshengtu model to generate inspiration sketches or complete works to explore new styles and concepts.
[0059] 2. Product display: E-commerce platforms can use the Wenshengtu model to generate high-quality images of products to showcase different angles and configurations of products.
[0060] 3. Fashion design: Designers can use the text graph model to generate clothing design sketches to quickly iterate and create relevant clothing designs.
[0061] By adopting the technical solution of the embodiment of the present application, during the training process of the diffusion model, the entity and attribute information in each prompt word sample are extracted, the combination of the entity and its own attribute information is used as a positive sample pair, the combination of different entities, the combination of different attribute information, and the combination of the entity and the attribute information of other entities are used as negative sample pairs, and then the parameters of the diffusion model are adjusted based on the contrastive learning loss, so that the model can perform contrastive learning on the attention maps of the positive and negative sample pairs, ensuring that the relevant attention maps determined by the model for the positive sample pairs can be close to each other, and the relevant attention maps determined for the negative sample pairs can be far away from each other, thereby reducing the information pollution on the attention map, so that the attention map obtained by the model can accurately capture the guiding information in the text description (such as the prompt word sample), avoiding the loss of entity or attribute information or inaccurate number of entities in the model-generated image, thereby improving the accuracy of the model-generated image.
[0062] As a possible implementation, determining the contrastive learning loss corresponding to each of the prompt word samples according to the attention map similarities of each positive sample pair and the attention map similarities of each negative sample pair associated with each of the prompt word samples includes:
[0063] For each entity or attribute information in each of the prompt word samples, determining the contrastive learning loss corresponding to the entity or attribute information through a target formula;
[0064] According to the contrastive learning losses corresponding to each entity and each attribute information in each prompt word sample, the contrastive learning loss corresponding to each prompt word sample is determined. For example, for each prompt word sample, the weighted sum of the contrastive learning losses corresponding to each entity and each attribute information in the prompt word sample can be determined as the contrastive learning loss corresponding to the prompt word sample.
[0065] The target formula is as follows:
[0066]
[0067] Among them, j, j + 、j - represents a single entity or a single attribute information in a single prompt word sample (such as any one of “school bag”, “hat”, “green” and “red”), and j and j + The combination of j and j is a positive sample pair. - The combination of is a negative sample pair; M j , M n Indicates j, j + , n respectively correspond to the attention map; sim() represents the similarity calculation operation, optionally, u,v represent different attention maps; τ represents the temperature coefficient; Represents the contrastive learning loss corresponding to j.
[0068] In this embodiment, for different entities and attribute information in the text description (such as the prompt word sample), the corresponding attention map M is extracted, for example, the attention map M is extracted. 书包 , M 帽子 Etc., and according to the extracted attention map, the contrastive learning loss corresponding to each entity and attribute information is calculated through the target formula, and then the contrastive learning loss corresponding to the relevant prompt word sample is determined, thereby achieving more refined constraints on the positive and negative sample pairs associated with each entity and attribute information, so as to better achieve the purpose of bringing the positive sample pairs closer and pushing the negative sample pairs away.
[0069] Optionally, to simplify the calculation operation, the contrastive learning losses corresponding to all entities and attribute information in the prompt word sample can be averaged to obtain the final contrastive learning loss (i.e., the contrastive learning loss corresponding to the prompt word sample). Determined by the following formula:
[0070]
[0071] Wherein, N represents the number of entities and attribute information in a single prompt word sample, j represents a single entity or a single attribute information in a single prompt word sample, Denotes the contrastive learning loss corresponding to j.
[0072] As a possible implementation, the step of adjusting the parameters of the diffusion model according to the contrastive learning loss corresponding to each of the prompt word samples includes:
[0073] Determining the target loss corresponding to each of the prompt word samples according to the contrastive learning loss corresponding to each of the prompt word samples and the model loss of the Unet model, wherein the model loss of the Unet model is used to characterize the difference between the noise predicted by the Unet model and the actual noise;
[0074] The parameters of the other parts of the diffusion model except the Unet model are fixed, and the parameters of the Unet model are adjusted according to the target losses corresponding to the respective prompt word samples.
[0075] In the specific implementation, the parameters of other parts of the diffusion model except the Unet model (such as the text encoder and the image codec) are fixed, and the contrast loss is combined with the model loss of the Unet model based on the weighted sum method to obtain the target loss, and then the minimization of the target loss is used as the training goal, and the parameters of the Unet model are adjusted in a targeted manner. In this way, the cross-attention operation of the Unet model is constrained based on contrastive learning, which can effectively avoid the problems of subject loss, attribute leakage and inaccurate number of subjects when the diffusion model generates images for multiple subjects.
[0076] It should be noted that when the Unet model is trained based on the model loss of the Unet model, random Gaussian noise is added to the initial feature z0 in the latent space over time t, thereby obtaining the latent space feature z of the tth step. t . The Cain space feature z t Input to Unet model∈ θ , to train the model to predict the random Gaussian noise (i.e., real noise) added above. For example, the model loss of the Unet model is It can be determined based on the following loss function:
[0077]
[0078] Among them, ∈ represents the real noise, ∈ θ (z t ,c(T),t) represents the noise predicted by the Unet model, represents image features, c(T) represents text features, It means that all possible errors are weighted averaged in the loss function.
[0079] Optionally, the target loss is determined by a target loss function, and the target loss function is expressed as follows:
[0080]
[0081] in, represents the target loss, represents the contrastive learning loss corresponding to a single cue word sample, represents the model loss of the Unet model, and λ represents the preset weight.
[0082] As a possible implementation, extracting entity and attribute information from each prompt word sample includes:
[0083] For each of the prompt word samples, a pre-trained large language model (such as the Qianwen model, the GLM model or the LLaMA model, etc.) is used to perform part-of-speech analysis on each word in the prompt word sample, and entity and attribute information is extracted from each word according to the part-of-speech analysis results.
[0084] In this embodiment, with the help of a pre-trained large language model, the content of the prompt word sample can be automatically and effectively divided, so as to quickly and accurately extract the entity and attribute information in the prompt word sample.
[0085] Second, refer to Figure 3 FIG. 1 is a flowchart of an implementation of an image generation method provided in an embodiment of the present application, and the method comprises the following steps:
[0086] Step S21: determining the text description input by the user as a prompt word, wherein the text description is used to represent the content required to be included in the image to be generated;
[0087] Step S22: inputting the prompt word into a pre-trained text graph model to generate an image corresponding to the prompt word, wherein the pre-trained text graph model is obtained by the text graph model training method described in the first aspect.
[0088] It is understandable that when using the text graph model obtained based on the aforementioned text graph model training method for reasoning, there is no need to extract the attention map, and reasoning can be performed by simply inputting the prompt words into the text graph model, thereby not adding any additional reasoning burden.
[0089] For example, a text description given by a user is used as a prompt word, wherein the text description is used to characterize the content contained in the image that the user wants to obtain, and it may contain multiple complex entities and different entity attribute information; the prompt word is input into the text graph model for reasoning, and an image that meets the prompt word and contains multiple entities with different attributes can be obtained.
[0090] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the order of the actions described, because according to the embodiments of the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.
[0091] Thirdly, Figure 4 : is a structural schematic diagram of a text graph model training device according to an embodiment of the present application, the device comprising:
[0092] A first extraction module 410 is used to extract entity and attribute information from each prompt word sample when training the diffusion model using the image and text annotation dataset, wherein the image and text annotation dataset includes each prompt word sample and its corresponding image sample;
[0093] A second extraction module 420 is used to extract the attention map corresponding to each entity and the attention map corresponding to each attribute information from each attention map generated by the Unet model in the diffusion model for each prompt word sample and its corresponding image sample;
[0094] The data processing module 430 is used to determine, for each of the prompt word samples, a combination of an entity in the prompt word sample and the attribute information of the entity itself as a positive sample pair, and a combination of different entities, a combination of different attribute information, and a combination of an entity and attribute information of other entities in the prompt word sample as negative sample pairs;
[0095] A loss calculation module 440 is used to determine the contrastive learning loss corresponding to each of the prompt word samples according to the attention map similarities of each positive sample pair and the attention map similarities of each negative sample pair associated with each of the prompt word samples, wherein the contrastive learning loss is positively correlated with the attention map similarity of the positive sample pair, and the contrastive learning loss is negatively correlated with the attention map similarity of the negative sample pair;
[0096] The parameter adjustment module 450 is used to adjust the parameters of the diffusion model according to the contrastive learning loss corresponding to each of the prompt word samples, so as to obtain a trained text graph model.
[0097] By adopting the technical solution of the embodiment of the present application, during the training process of the diffusion model, the entity and attribute information in each prompt word sample are extracted, the combination of the entity and its own attribute information is used as a positive sample pair, the combination of different entities, the combination of different attribute information, and the combination of the entity and the attribute information of other entities are used as negative sample pairs, and then the parameters of the diffusion model are adjusted based on the contrastive learning loss, so that the model can perform contrastive learning on the attention maps of the positive and negative sample pairs, ensuring that the relevant attention maps determined by the model for the positive sample pairs can be close to each other, and the relevant attention maps determined for the negative sample pairs can be far away from each other, thereby reducing the information pollution on the attention map, so that the attention map obtained by the model can accurately capture the guiding information in the text description (such as the prompt word sample), avoiding the loss of entity or attribute information or inaccurate number of entities in the model-generated image, thereby improving the accuracy of the model-generated image.
[0098] Optionally, the loss calculation module 440 is further configured to perform the following steps:
[0099] For each entity or attribute information in each of the prompt word samples, determining the contrastive learning loss corresponding to the entity or attribute information through a target formula;
[0100] Determine the contrastive learning loss corresponding to each of the prompt word samples according to the contrastive learning losses corresponding to each of the entities and each of the attribute information in each of the prompt word samples;
[0101] The target formula is as follows:
[0102]
[0103] Among them, j, j + 、j - represent a single entity or a single attribute information in a single prompt word sample, and j and j + The combination of j and j is a positive sample pair. - The combination of is a negative sample pair; M j , M n Indicates j, j + , n respectively correspond to the attention maps; sim() represents the similarity calculation operation; τ represents the temperature coefficient; Denotes the contrastive learning loss corresponding to j.
[0104] Optionally, the contrastive learning loss corresponding to a single cue word sample is Determined by the following formula:
[0105]
[0106] Wherein, N represents the number of entities and attribute information in a single prompt word sample, j represents a single entity or a single attribute information in a single prompt word sample, Denotes the contrastive learning loss corresponding to j.
[0107] Optionally, the parameter adjustment module 450 is further configured to perform the following steps:
[0108] Determining the target loss corresponding to each of the prompt word samples according to the contrastive learning loss corresponding to each of the prompt word samples and the model loss of the Unet model, wherein the model loss of the Unet model is used to characterize the difference between the noise predicted by the Unet model and the actual noise;
[0109] The parameters of the other parts of the diffusion model except the Unet model are fixed, and the parameters of the Unet model are adjusted according to the target losses corresponding to the respective prompt word samples.
[0110] Optionally, the target loss is determined by a target loss function, and the target loss function is expressed as follows:
[0111]
[0112] in, represents the target loss, represents the contrastive learning loss corresponding to a single cue word sample, represents the model loss of the Unet model, and λ represents the preset weight.
[0113] Optionally, the first extraction module 410 is further used to perform part-of-speech analysis on each word in the prompt word sample through a pre-trained large language model, and extract entity and attribute information from each word according to the part-of-speech analysis result.
[0114] Fourthly, Figure 5 : is a schematic diagram of the structure of an image generating device according to an embodiment of the present application, the device comprising:
[0115] An input module 510, used to determine a text description input by a user as a prompt word, wherein the text description is used to represent the content required to be included in the image to be generated;
[0116] The reasoning module 520 is used to input the prompt word into a pre-trained text graph model to generate an image corresponding to the prompt word, wherein the pre-trained text graph model is obtained by the text graph model training method as described in the first aspect.
[0117] It should be noted that the device embodiment is similar to the method embodiment, so the description is relatively simple, and the relevant parts can be referred to the method embodiment.
[0118] The present application also provides an electronic device, referring to Figure 6 , Figure 6 Schematic diagram of an electronic device proposed in an embodiment of the present application. Figure 6 As shown, the electronic device 100 includes: a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus communication. A computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement the steps in the Wensheng graph model training method or image generation method disclosed in the embodiment of the present application.
[0119] The embodiment of the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the method for training the cultural graph model or the method for generating the image as disclosed in the embodiment of the present application is implemented.
[0120] The embodiment of the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the Wensheng graph model training method or the image generation method disclosed in the embodiment of the present application.
[0121] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0122] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0123] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, systems, devices, storage media, and program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0124] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0126] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.
[0127] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0128] The above is a detailed introduction to a text graph model training method, image generation method, device, medium and product provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A method for training a text graph model, characterized in that: The method comprises: When using the image-text annotation dataset to train the diffusion model, extracting entity and attribute information from each prompt word sample, the image-text annotation dataset includes each prompt word sample and its corresponding image sample; Extracting the attention map corresponding to each entity and the attention map corresponding to each attribute information from the attention maps generated by the Unet model in the diffusion model for each prompt word sample and its corresponding image sample; For each of the prompt word samples, a combination of an entity in the prompt word sample and the attribute information of the entity itself is determined as a positive sample pair, and a combination of different entities, a combination of different attribute information, and a combination of an entity and attribute information of other entities in the prompt word sample are respectively determined as negative sample pairs; Determine the contrastive learning loss corresponding to each of the prompt word samples according to the similarities of the attention maps of each positive sample pair and the similarities of the attention maps of each negative sample pair associated with each of the prompt word samples, wherein the contrastive learning loss is positively correlated with the similarities of the attention maps of the positive sample pairs, and the contrastive learning loss is negatively correlated with the similarities of the attention maps of the negative sample pairs; The parameters of the diffusion model are adjusted according to the contrastive learning losses corresponding to the respective prompt word samples to obtain a trained text graph model.
2. The method according to claim 1, characterized in that Determining the contrastive learning loss corresponding to each of the prompt word samples according to the attention map similarities of each positive sample pair and the attention map similarities of each negative sample pair associated with each of the prompt word samples includes: For each entity or attribute information in each of the prompt word samples, determining the contrastive learning loss corresponding to the entity or attribute information through a target formula; Determine the contrastive learning loss corresponding to each of the prompt word samples according to the contrastive learning losses corresponding to each of the entities and each of the attribute information in each of the prompt word samples; The target formula is as follows: Among them, j, j + 、j - represent a single entity or a single attribute information in a single prompt word sample, and j and j + The combination of j and j is a positive sample pair. - The combination of is a negative sample pair; M j , M n Indicates j, j + , n respectively correspond to the attention maps; sim() represents the similarity calculation operation; τ represents the temperature coefficient; Denotes the contrastive learning loss corresponding to j.
3. The method according to claim 2, characterized in that The contrastive learning loss corresponding to a single cue word sample Determined by the following formula: Wherein, N represents the number of entities and attribute information in a single prompt word sample, j represents a single entity or a single attribute information in a single prompt word sample, Denotes the contrastive learning loss corresponding to j.
4. The method according to claim 1, characterized in that: The step of adjusting the parameters of the diffusion model according to the contrastive learning loss corresponding to each of the prompt word samples includes: Determining the target loss corresponding to each of the prompt word samples according to the contrastive learning loss corresponding to each of the prompt word samples and the model loss of the Unet model, wherein the model loss of the Unet model is used to characterize the difference between the noise predicted by the Unet model and the actual noise; The parameters of the other parts of the diffusion model except the Unet model are fixed, and the parameters of the Unet model are adjusted according to the target losses corresponding to the respective prompt word samples.
5. The method according to claim 4, characterized in that The target loss is determined by a target loss function, which is expressed as follows: in, represents the target loss, represents the contrastive learning loss corresponding to a single cue word sample, represents the model loss of the Unet model, and λ represents the preset weight.
6. The method according to any one of claims 1 to 5, characterized in that: The extracting of entity and attribute information from each prompt word sample includes: For each of the prompt word samples, a pre-trained large language model is used to perform part-of-speech analysis on each of the segmented words in the prompt word sample, and entity and attribute information is extracted from each of the segmented words according to the part-of-speech analysis results.
7. An image generation method, characterized in that: The method comprises: Determine a text description input by a user as a prompt word, wherein the text description is used to characterize the content required to be included in the image to be generated; Inputting the prompt word into a pre-trained text image model to generate an image corresponding to the prompt word; Wherein, the pre-trained text graph model is obtained by the text graph model training method according to any one of claims 1-6.
8. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the Wensheng graph model training method according to any one of claims 1 to 6, or to implement the image generation method according to claim 7.
9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the method for training a text graph model as described in any one of claims 1 to 6 is implemented, or the method for generating an image as described in claim 7 is implemented.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for training a text graph model as described in any one of claims 1 to 6 is implemented, or the method for generating an image as described in claim 7 is implemented.