Figure graph model training method and device and text graph method and device

Abstract

The invention provides a text graph model training method and a text graph method and device.The text graph model training method comprises the steps that image-text pairs are input into candidate text graph models, and each image-text pair comprises a sample image and sample description information correspondingly comprising preset relation words and identity information; adding preset noise to the sample image through the candidate text graph model, denoising the noise image based on the sample description information to obtain predicted noise, and obtaining target loss based on noise loss between the predicted noise and the preset noise and comparison loss between the sample description information and preset positive and negative samples, and based on the target loss, carrying out fine tuning on the candidate text graph model to obtain a target text graph model. According to the invention, the candidate text graph model is trained based on the loss of the two aspects of the image and the relation word, the learning ability of the text graph model for the relation and the image is enhanced, and the accuracy of the image output by the text graph model based on the user description is further improved.

Description

Technical Field

[0001] The present invention relates to the field of large model technology, and in particular to a text graph model training method, a text graph method and a device. Background Art

[0002] Text-to-Image is a technology that generates images based on natural language descriptions. In recent years, the technology of text-to-image has made great progress, and the underlying models can be divided into three categories: GAN (Generative Adversarial Network), diffusion model, and autoregressive model.

[0003] At present, the models in the industry are mainly based on the diffusion model. For example, SDM (Stable Diffusion Model) is a widely used model. The underlying model it adopts is the diffusion model. By combining the diffusion model with the latent space, the computational complexity is greatly reduced, and at the same time, good image generation effects can be achieved.

[0004] The current SD model (stable diffusion model) mainly focuses on the appearance and details of the generated samples. The details of the specific people and things themselves are relatively clear, but there are still problems in learning specific relationships. For example, the relationship between people: hugs, handshakes, and back-to-back, and the relationship between things: up, down, and hanging, etc. There may be errors in relationship generation or inconsistency with the facts, resulting in low accuracy of the generated images. Summary of the invention

[0005] In view of this, the embodiments of the present invention provide a method for training a culture graph model, a culture graph method and a device to improve the relationship perception ability of the culture graph model, thereby improving the accuracy of the image generated by the culture graph model.

[0006] According to one aspect of the present invention, a method for training a cultural graph model is provided, the method comprising:

[0007] Obtain a pre-trained candidate text graph model;

[0008] Inputting training data into the candidate text-image model, the training data being image-text pairs, each of the image-text pairs including a sample image and sample description information of the sample image, the sample description information including preset relational words and identity information;

[0009] Adding preset noise to the sample image through the candidate text graph model to obtain a noise image, and denoising the noise image based on the sample description information to obtain predicted noise;

[0010] Obtaining a target loss based on a noise loss between the predicted noise and the preset noise, and a contrast loss between the sample description information and preset positive and negative samples;

[0011] The candidate text graph model is fine-tuned based on the target loss until the target loss converges to obtain a target text graph model.

[0012] In a possible embodiment, the contrast loss is obtained by the following steps:

[0013] Randomly extracting a plurality of comparison samples from a preset sample library, wherein the preset sample library contains positive samples and negative samples set for each preset relational word;

[0014] Calculate the semantic similarity between each of the comparison samples and the sample description information, determine that the comparison samples whose semantic similarity with the sample description information is higher than a preset similarity threshold are positive comparison samples, and determine that the comparison samples whose semantic similarity with the sample description information is not higher than the preset similarity threshold are negative comparison samples;

[0015] The contrast loss is determined based on the sample description information, the semantic similarity between the positive comparison sample and the negative comparison sample.

[0016] In a possible embodiment, adding preset noise to the sample image through the candidate text graph model to obtain a noise image, and denoising the noise image based on the sample description information to obtain predicted noise includes:

[0017] Based on preset Gaussian noise, adding noise to the sample image to obtain a noise image corresponding to the sample image;

[0018] Encoding the noise image using a preset image encoder, and text encoding the sample description information using a preset text encoder to obtain a target image code and a target text code;

[0019] Embedding the target image code and the target text code to obtain a target embedded code; wherein the embedding is to concatenate the parts of the target image code and the target text code with the highest semantic similarity of identity information;

[0020] The noisy image is denoised based on the target embedded coding to obtain predicted noise.

[0021] In a possible embodiment, the method further includes:

[0022] Performing image segmentation on the sample image to obtain a plurality of segmentation results, wherein each of the segmentation results contains information about an object, and the objects include people and animals;

[0023] Extracting image features of each segmentation result to obtain each segmentation feature;

[0024] Performing cross-attention calculation on each of the segmentation features and the target text encoding to obtain a target mixed encoding;

[0025] The noisy image is denoised based on the target hybrid coding and the target embedded coding to obtain predicted noise.

[0026] According to another aspect of the present invention, a method for a Vincent diagram is provided, the method comprising:

[0027] Get target description text and reference image;

[0028] The target description text and the reference image are output to a preset text graph model, so that the text graph model uses a preset text encoder to encode the target description text to obtain a target text code, and encodes the reference image to obtain a target image code; preset noise is added to the target image code to obtain a noisy image, and the target text code and the target image code are embedded and cross-attention calculated to obtain a target embedded code and a target mixed code; the noisy image is denoised based on the target text code, the target embedded code and the target mixed code to obtain a target image, wherein the preset text graph model is pre-trained by any of the above-mentioned text graph model training methods.

[0029] According to another aspect of the present invention, a device for training a cultural graph model is provided, the device comprising:

[0030] A first acquisition module is used to acquire a pre-trained candidate text graph model;

[0031] A first input module, used to input training data into the candidate text-image model, wherein the training data is a picture-text pair, each of which includes a sample image and sample description information of the sample image, and the sample description information includes a preset relational word and identity information;

[0032] A denoising module, configured to add preset noise to the sample image through the candidate text graph model to obtain a noise image, and denoise the noise image based on the sample description information to obtain predicted noise;

[0033] A loss calculation module, used to obtain a target loss based on a noise loss between the predicted noise and the preset noise, and a contrast loss between the sample description information and preset positive and negative samples;

[0034] A training module is used to fine-tune the candidate text graph model based on the target loss until the target loss converges to obtain a target text graph model.

[0035] In a possible embodiment, the contrast loss is obtained by the following steps:

[0036] Randomly extracting a plurality of comparison samples from a preset sample library, wherein the preset sample library contains positive samples and negative samples set for each preset relational word;

[0037] Calculate the semantic similarity between each of the comparison samples and the sample description information, determine that the comparison samples whose semantic similarity with the sample description information is higher than a preset similarity threshold are positive comparison samples, and determine that the comparison samples whose semantic similarity with the sample description information is not higher than the preset similarity threshold are negative comparison samples;

[0038] The contrast loss is determined based on the sample description information, the semantic similarity between the positive comparison sample and the negative comparison sample.

[0039] According to another aspect of the present invention, there is provided a Wensheng diagram device, the device comprising:

[0040] The second acquisition module is used to acquire the target description text and the reference image;

[0041] The third acquisition module is used to output the target description text and the reference image to a preset text graph model, so that the text graph model uses a preset text encoder to encode the target description text to obtain a target text code, encodes the reference image to obtain a target image code; adds preset noise to the target image code to obtain a noise image, and embeds and cross-attention calculates the target text code and the target image code to obtain a target embedded code and a target mixed code; denoises the noise image based on the target text code, the target embedded code and the target mixed code to obtain a target image, wherein the preset text graph model is pre-trained by any of the above-mentioned text graph model training methods.

[0042] According to another aspect of the present invention, there is provided an electronic device, comprising:

[0043] Processor; and

[0044] Memory for storing programs,

[0045] Wherein, the program includes instructions, and when the instructions are executed by the processor, the processor executes any of the above-mentioned Wensheng graph model training methods or Wensheng graph methods.

[0046] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute any of the above-mentioned Vincent graph model training methods or Vincent graph methods.

[0047] One or more technical solutions provided in the embodiments of the present invention fine-tune the pre-trained candidate text graph model by using training data containing preset relationship words, calculate the contrast loss based on the preset relationship words contained in the sample description information and the positive and negative samples, and obtain the denoising loss based on the predicted noise obtained by the candidate text graph model for the sample image and the preset noise added to the known sample image, and train the candidate text graph model based on the target loss obtained by these two losses. By training the candidate text graph model based on the loss of both images and relationship words, the learning ability of the text graph model for relationships and images is enhanced, thereby improving the accuracy of the image output by the text graph model based on the user description. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Further details, features and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0049] Figure 1 A schematic diagram of a flow chart of a method for training a text graph model provided by an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of a process of pre-training a Vincent graph model in a Vincent graph model training method provided in an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of a process of fusion of multiple characters in the method for training a cultural graph model provided by an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of a process of relationship learning in the text graph model training method provided in an embodiment of the present invention;

[0053] Figure 5 A schematic diagram of a flow chart of a Vincent diagram method provided by an embodiment of the present invention;

[0054] Figure 6 A schematic diagram of another process flow of the Wensheng diagram method provided by an embodiment of the present invention;

[0055] Figure 7 A schematic diagram of the structure of a text graph model training device provided by an embodiment of the present invention;

[0056] Figure 8 A schematic diagram of the structure of a Wensheng graph device provided in an embodiment of the present invention;

[0057] Fig. 9 A block diagram of an exemplary electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0058] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0059] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0060] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0061] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0062] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0063] In order to improve the relationship perception ability of the Wensheng graph model, and thus improve the accuracy of the image generated by the Wensheng graph model, an embodiment of the present invention provides a Wensheng graph model training method, a Wensheng graph method, a device, an electronic device and a storage medium. The Wensheng graph model training method provided by the embodiment of the present invention can be applied to any electronic device with a model training function, which can be a server, a computer or a mobile terminal, etc. The scheme of the present invention is described below with reference to the accompanying drawings:

[0064] Figure 1 A flow chart of a method for training a text graph model provided by an embodiment of the present invention may include the following steps:

[0065] S101, obtaining a pre-trained candidate text graph model;

[0066] S102, inputting training data into the candidate text-image model, wherein the training data is a picture-text pair, each of which includes a sample image and sample description information of the sample image, and the sample description information includes a preset relational word and identity information;

[0067] S103, adding preset noise to the sample image through the candidate text graph model to obtain a noise image, and denoising the noise image based on the sample description information to obtain predicted noise;

[0068] S104, obtaining a target loss based on a noise loss between the predicted noise and the preset noise, and a contrast loss between the sample description information and preset positive and negative samples;

[0069] S105 . Fine-tune the candidate text graph model based on the target loss until the target loss converges to obtain a target text graph model.

[0070] By applying the embodiment of the present invention, a pre-trained candidate text graph model is fine-tuned by using training data containing preset relationship words, and the contrast loss is calculated based on the preset relationship words contained in the sample description information and the positive and negative samples, and the denoising loss is obtained based on the predicted noise obtained by the candidate text graph model for the sample image and the preset noise added to the known sample image, and the candidate text graph model is trained by obtaining the target loss based on these two losses. By training the candidate text graph model based on the loss of both images and relationship words, the learning ability of the text graph model for the relationship and the corresponding image is enhanced, thereby improving the accuracy of the image output by the text graph model based on the user description.

[0071] The above S101-S105 are exemplarily described below:

[0072] The above-mentioned pre-trained candidate text-image model is a model pre-trained on large-scale text-image data, and the model can be SDM (Stable Diffusion Model), LDM (Latent Diffusion Model), etc. As a possible implementation method, a massive pre-training data set can be constructed using open source data sets and self-collected data. The above-mentioned open source data sets can include Laion-Art, Laion-HD, Laion-pop, Laion-2B, etc. The size of the pre-training data set can reach 2.5B (billion), and the model can be trained based on the pre-training data set later.

[0073] As a possible implementation, Figure 2 As shown, when pre-training the text-to-image model, after the image-text pair is input into the model, the model encodes the image and text respectively to obtain image encoding and text encoding.

[0074] Specifically, the text graph model can use the encoder of vae (variational auto-encoder) to compress the image code into the latent space, which is a low-dimensional space to avoid the large computational pressure caused by the excessive pixel space of the original image. For the image code, it can be diffused in the latent space. Specifically, the noise is added to the image code multiple times to obtain the noise code. The noise is usually a preset value, which can be Gaussian noise. Then the text code and the noise code are embedded to obtain the embedded code. The embedding here can be a simple splicing of the text code and the noise code. Then the embedded code can be processed by unet, and the noise code can be denoised based on the embedded code. unet (full name U-Net) is a deep learning architecture that can perform semantic segmentation based on embedded coding, and can capture multi-scale feature information during the segmentation process, so as to effectively learn and restore the details of the image. The semantic segmentation result obtained by Unet can be input into the vae decoder, and the latent variables are restored to reconstructed samples through the vae decoding process. Then the decoder decodes the reconstructed sample to obtain the generated image. By training the model based on the loss obtained from the generated image and the input image, a pre-trained candidate text image model can be obtained. The loss can be set according to actual needs, usually a cross entropy loss.

[0075] Unet can reduce noise in the latent space. Specifically, it can predict the noise to be removed based on the semantics of the target text encoding and the noise map, and subtract the noise to be removed from the noisy image to obtain a new image. Specifically, the noise can be removed multiple times according to the preset sampling steps to obtain the encoding of the image to be generated, and the encoding of the image to be generated can be decoded by the decoder to obtain the image to be generated.

[0076] The model is pre-trained in a large-scale pre-training dataset so that the candidate text graph model can better learn text and image features, which ensures the accuracy of the images generated by the text graph model to a certain extent. The candidate text graph model can be fine-tuned subsequently to obtain a target text graph model that is more suitable for the target application scenario.

[0077] In an embodiment of the present invention, an image containing multiple objects and a corresponding description text of the inclusion relationship can be used as a training set to fine-tune the candidate text-generated graph model. In order to distinguish it from a pre-training data set, in an embodiment of the present invention, the training set used in the fine-tuning training process is formed into a fine-tuning training set. Each piece of training data in the fine-tuning training set is also in the form of an image-text pair, wherein the image in the fine-tuning training set can be an image containing multiple objects, and the object can be a person, an animal, or an object, etc. There is a relative relationship between each of the multiple objects, and accordingly, the description text corresponding to the image includes preset relationship words and object identity information.

[0078] The image-text pairs in the pre-training data set may contain a large number of single object images, and their description texts are relatively simple. Therefore, when setting the fine-tuning training set, the description text containing preset relationship words and identity information of multiple objects and the corresponding images can be set in a targeted manner according to the actual application scenario. The preset relationship words refer to words that can describe the relationship between multiple objects contained in the image, such as hugs, handshakes, back-to-back, etc. The identity information can be the corresponding gender, color, clothing and other information contained in the image. For the convenience of description, in the embodiment of the present invention, the images in the fine-tuning training set are referred to as sample images, and the description text of the sample images is referred to as sample description information.

[0079] After inputting each piece of training data included in the fine-tuning training set into the candidate text graph model, the candidate text graph model can encode the sample image and sample description information respectively to obtain the target image code and the target text code, and add preset noise to the target image code, and the preset noise code can be flexibly set according to actual needs. Then, the target text code and the target image code with the preset noise code added are embedded, and the embedded code is denoised to finally generate the target image, wherein the embedding is to splice the part with the highest semantic similarity of the identity information in the target image code and the target text code.

[0080] The Wensheng graph model usually generates a noise map based on an input image and a preset resolution, and performs multiple denoising on the noise map based on the input description text to obtain a generated target image. Each denoising step uses a preset sampling function to sample pixel values ​​from the noise map. The sampling function can be flexibly set according to actual needs. As a possible implementation, the preset sampling function can be α=0.5, where T is the total number of preset denoising times and t is the denoising times identifier.

[0081] In a possible embodiment, preset noise can be added to each sample image in the fine-tuning training set to obtain different noise samples corresponding to each sample image, such as low-noise image and high-noise image samples corresponding to each sample image. Accordingly, during the model training process, each sample image, the noise sample image corresponding to the sample image, and the description information can be input into the candidate text graph model, and the candidate text graph model can denoise the noise sample image and output the target image.

[0082] As a possible implementation, when generating the target text encoding, the candidate text graph model can segment the sample description information, and generate a corresponding vector for each segmentation result, thereby obtaining the target text encoding. Exemplarily, a segmentation vocabulary can be preset, and the segmentation vocabulary can include common words, preset relationship words, etc. When segmenting words, the segmentation can be performed based on the preset segmentation vocabulary. In a possible embodiment, the description information can also be segmented according to a preset segmentation length. Exemplarily, the segmentation length can be set to 4. When segmenting the description information, the segmentation can be performed in groups of 4 words.

[0083] In a possible embodiment, when generating the target code of the sample description information, the segmentation code and position code can be calculated for each segmentation, and the segmentation code and position code of the segmentation can be spliced ​​to obtain the text code of the segmentation, and then the text code of the description information is obtained based on the text code of each segmentation. Among them, the segmentation code can be obtained based on a preset word table, which can include the correspondence between each preset segmentation and the vector, and the position code can identify the order of the segmentation result among all segmentation results. The dimensions of the above segmentation code and position code can be flexibly set according to actual needs.

[0084] For example, for the description information "a girl is running", the description information can be segmented in groups of 4 words to obtain the segmentation results "a girl" "is running". Then, the segmentation code and position code are calculated for "a girl" to obtain the segmentation code of the segmentation, where the position code can be 1; the segmentation code and position code are calculated for "is running" to obtain the segmentation code of the segmentation, where the position code can be 2. The two segmentation codes are spliced ​​to obtain the target text code of the description information.

[0085] In a possible embodiment, the candidate text graph model may include a text encoder, which may be a large language model such as CLIP, T5, or a BERT (Bidirectional Encoder Representationsfrom Transformers) text encoder, wherein CLIP is a neural network that connects text and images, and learns to match images and text descriptions through contrastive learning, in the hope of unifying image features and text features.

[0086] In the embodiment of the present invention, various feasible image encoding methods can be used to perform image encoding on the sample image to obtain the target image encoding. The dimension of the target image encoding can be flexibly set according to actual needs.

[0087] In a possible embodiment, the latent space of the candidate text graph model may include a cross-attention layer, which can perform cross-attention calculations on the target embedding code and the image features of the target image to enhance the learning of the association relationship between the image features and the text code.

[0088] As a possible implementation, taking the object as a person as an example, the target image can be segmented to obtain multiple segmentation results, each of which contains different people. Based on the segmentation results, the image features of each person in the target image can be obtained as segmentation features. After that, cross-attention calculation can be performed based on the target text code and each segmentation feature to obtain a target hybrid code, and then the target image can be output based on the target hybrid code and the target embedded code.

[0089] The cross attention calculation refers to comparing each element in the input sequence with each element in the output sequence and calculating the similarity between them. Then, these similarities are used as weights for weighted averaging, and the weighted average result is used as the representation of each element of the output sequence. For the embodiment of the present invention, the input sequence is the segmentation feature, and the output sequence is the target text encoding.

[0090] like Figure 3 As shown, Figure 3 The present invention is a flowchart of the candidate text graph model in the embodiment of the present invention that integrates characters and text codes. Specifically, a sample image containing multiple characters and corresponding description information are input into the candidate text graph model. The candidate text graph model can segment the sample image to obtain multiple character segmentation results, and use an image encoder to perform image encoding on each task segmentation result to obtain each segmented image code. Then, based on the semantics of each text segmentation word and the semantics of the segmented image code, the segmented image code is embedded into the target text code. The embedding here refers to concat the segmented image code into the corresponding segmentation code. The concat() method is used to connect two or more arrays. Specifically, the segmented image code is spliced ​​into the corresponding segmentation code to obtain the target embedded code.

[0091] At the same time, the candidate text graph model can perform character recognition on the sample image, obtain the segmentation mask of each character, and use the features of each mask as the input sequence and the target embedding code as the output sequence to perform cross-attention calculation and determine the features in the target embedding code corresponding to the mask, thereby achieving further fusion of image and text.

[0092] In a possible embodiment, a plurality of positive and negative samples may be preset for each preset relational word, wherein the positive sample refers to a word that is highly semantically related to the preset relational word, and the negative sample refers to a word that is semantically opposite to the preset relational word or has a different part of speech. For example, if the preset relational word is handshake, its positive sample may be face to face, clapping, etc., and its negative sample may be back to back, man, woman, etc. The positive and negative samples may be pre-stored in a preset database for use in fine-tuning training of the candidate text graph model.

[0093] In the process of denoising a noisy image using a candidate text graph model, samples can be randomly selected from a sample library, and based on the semantics of the samples and the semantics of preset relationship words contained in the description information, samples whose semantics are close to the preset relationship words are determined as positive comparison samples, and samples whose semantics are less than a preset threshold The similarity between the semantics and the semantics of the preset relationship words is determined as negative comparison samples.

[0094] Afterwards, the candidate text graph model can be fine-tuned based on the denoising loss and the contrast loss obtained based on the positive and negative samples, wherein the denoising loss is the loss between the preset noise code and the denoising code obtained in the denoising process, and the loss can be a cross entropy loss. As a possible implementation, the denoising loss L can be obtained by the following formula: denoise :

[0095]

[0096] Among them, t is the number of iterations of noise removal timestep, and f is the sampling function, which can be specifically α=0.5,ε~N(0,1), refers to the Gaussian noise added to the image. θ is the pre-trained text graph model, c is the sample description information, τ θ (c) is the BERT text encoder that encodes the sample description information c, x0 is the initial image, X t is the image obtained after the t-th iteration. The loss is the cross entropy loss between the preset Gaussian noise obtained after t-step sampling and the predicted noise obtained by the candidate text graph model. The predicted noise is obtained by the candidate text graph model based on the number of iterations, the target text encoding, and the image obtained after the iteration.

[0097] The contrast loss is obtained based on the preset relational words and positive and negative samples contained in the description information. As a possible implementation, the contrast loss L can be obtained based on the following formula: steer :

[0098]

[0099] Among them, R is the preset relation word, T is the total number of iteration steps of denoising, is a positive sample, i is the number of positive samples, L is the number of positive samples, is a negative sample, M is the number of negative samples, and γ is a preset parameter, which can be 0.07.

[0100] Then, the total loss L = L can be obtained based on the denoising loss and the contrast loss. steer +L denoise , and fine-tune the candidate text graph model based on the total loss. Exemplarily, if the target loss has not converged, samples can be re-extracted from the sample library or model parameters can be adjusted, such as adjusting the sampling step size and various parameters in the model. When the target loss converges, the target text graph model can be determined. In this way, by training the candidate text graph model with the joint image loss and text loss, the target text graph model can better learn the connection between the image and the relationship, thereby improving the accuracy of image generation.

[0101] like Figure 4 As shown, Figure 4A flow chart of learning preset relation words in an embodiment of the present invention is provided. For the image-text pairs in the fine-tuning training set, noise is added to each sample image according to the preset noise to obtain noise images of different resolutions. Each noise image is output to the Vincent graph model to obtain the predicted noise output by the Vincent graph model. The denoising loss is obtained according to the above formula based on the predicted noise and the added preset noise. For each description text, a relation sample is randomly extracted from a preset sample library, and positive and negative samples are determined based on the semantics of the preset relation words contained in the description text and the semantics of the relation sample. The noise image is denoised based on the text encoding of the positive sample, and the contrast loss is obtained based on the positive and negative samples and the preset relation words. The Vincent graph model is then trained according to the sum of the contrast loss and the denoising loss.

[0102] By applying the embodiments of the present invention, the text graph pre-training module is used to determine the initial parameters of the text graph model, and is trained on a large amount of open source and self-collected data (about 2.5B) to fully learn the associated features of text images, so as to obtain better image generation effects in subsequent small-scale fine-tuning. Furthermore, by strengthening pre-training for the relationship prompts in the text graph model, while paying attention to the pixel-level generation and reconstruction of the image, relationship-guided contrast learning is added to accurately display the relationship between things in the generated image, such as handshakes, hugs, etc. In addition, the personalization of multiple characters is achieved through the character segmentation mask, the cross-attention map and the text condition, and finally the text is generated into a multi-character image, and the characters in the image can show specific relationships.

[0103] Based on the same inventive concept, the embodiment of the present invention also provides a method for generating a text map, such as Figure 5 As shown, the text graph method may include:

[0104] S501, obtaining target description text and reference image;

[0105] S502. Output the target description text and the reference image to a preset text graph model, so that the text graph model uses a preset text encoder to encode the target description text to obtain a target text code, and encodes the reference image to obtain a target image code; adds preset noise to the target image code to obtain a noisy image, and embeds and performs cross-attention calculations on the target text code and the target image code to obtain a target embedded code and a target mixed code; denoises the noisy image based on the target text code, the target embedded code and the target mixed code to obtain a target image, wherein the preset text graph model is pre-trained by any of the above-mentioned text graph model training methods.

[0106] The reference image may be an object for which a textual description that conforms to the target description needs to be generated, such as a person, an animal, or other object. In a possible embodiment, a plurality of reference images may be stored in a preset text image model, and the reference image may include objects of various attributes, such as people of different ages and genders, animals of different types, or other objects, and the like.

[0107] like Figure 6 As shown, Figure 6 A flow chart of generating an image using the text graph model provided by an embodiment of the present invention is provided. Specifically, preset relation words are extracted for the description words input by the user, and the description words are embedded in text using a text encoder to obtain a target text code. For a reference character image, an image encoder is used to obtain image codes of two reference character images, and the two image codes are embedded in the target text code to obtain a target embedded code. The reference character image can be a user input or a reference image pre-stored in the model. The image generation module adds noise to the reference character image based on preset noise, and denoises the noisy image based on the target text code and the target embedded code to generate a target image, wherein in the denoising process, cross-attention calculations are performed on the image, the target embedded code, and the target text code in the latent space, and the noisy image is denoised based on the calculation results.

[0108] According to another aspect of the present invention, a device for training a text graph model is provided. Figure 7 As shown, the apparatus 700 may include:

[0109] The first acquisition module 701 is used to acquire a pre-trained candidate text graph model;

[0110] A first input module 702, configured to input training data into the candidate text-image model, wherein the training data is a text-image pair, each of which includes a sample image and sample description information of the sample image, wherein the sample description information includes a preset relational word and identity information;

[0111] A denoising module 703, configured to add preset noise to the sample image through the candidate text graph model to obtain a noise image, and denoise the noise image based on the sample description information to obtain predicted noise;

[0112] A loss calculation module 704, configured to obtain a target loss based on a noise loss between the predicted noise and the preset noise, and a contrast loss between the sample description information and preset positive and negative samples;

[0113] The training module 705 is used to fine-tune the candidate document graph model based on the target loss until the target loss converges to obtain a target document graph model.

[0114] In a possible embodiment, the contrast loss is obtained by the following steps:

[0115] Randomly extracting a plurality of comparison samples from a preset sample library, wherein the preset sample library contains positive samples and negative samples set for each preset relational word;

[0116] Calculate the semantic similarity between each of the comparison samples and the sample description information, determine that the comparison samples whose semantic similarity with the sample description information is higher than a preset similarity threshold are positive comparison samples, and determine that the comparison samples whose semantic similarity with the sample description information is not higher than the preset similarity threshold are negative comparison samples;

[0117] The contrast loss is determined based on the sample description information, the semantic similarity between the positive comparison sample and the negative comparison sample.

[0118] According to another aspect of the present invention, a Wensheng map device is provided, such as Figure 8 As shown, the apparatus 800 may include:

[0119] The second acquisition module 801 is used to acquire target description text and reference image;

[0120] The third acquisition module 802 is used to output the target description text and the reference image to a preset text graph model, so that the text graph model uses a preset text encoder to encode the target description text to obtain a target text code, encodes the reference image to obtain a target image code; adds preset noise to the target image code to obtain a noise image, and embeds and cross-attention calculates the target text code and the target image code to obtain a target embedded code and a target mixed code; denoises the noise image based on the target text code, the target embedded code and the target mixed code to obtain a target image, wherein the preset text graph model is pre-trained by any of the above-mentioned text graph model training methods.

[0121] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0122] The exemplary embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present invention when executed by the at least one processor.

[0123] Exemplary embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0124] An exemplary embodiment of the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform a method according to an embodiment of the present invention.

[0125] refer to Fig. 9 , a block diagram of an electronic device 900 that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0126] like Fig. 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0127] A plurality of components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, an output unit 907, a storage unit 908, and a communication unit 909. The input unit 906 may be any type of device capable of inputting information to the electronic device 900, and the input unit 906 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 907 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 908 may include, but is not limited to, a disk, an optical disk. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0128] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above. For example, in some embodiments, the above-mentioned Wensheng graph model training method and Wensheng graph method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via the ROM 902 and / or the communication unit 909. In some embodiments, the computing unit 901 may be configured to perform the above-mentioned Wensheng graph model training method and Wensheng graph method by any other appropriate means (e.g., by means of firmware).

[0129] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0130] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0132] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0133] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0134] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A method for training a text graph model, characterized in that: The method comprises: Obtain pre-trained candidate text graph models; Inputting training data into the candidate text-image model, the training data being image-text pairs, each of the image-text pairs including a sample image and sample description information of the sample image, the sample description information including preset relational words and identity information; Adding preset noise to the sample image through the candidate text graph model to obtain a noise image, and denoising the noise image based on the sample description information to obtain predicted noise; Obtaining a target loss based on a noise loss between the predicted noise and the preset noise, and a contrast loss between the sample description information and preset positive and negative samples; The candidate text graph model is fine-tuned based on the target loss until the target loss converges to obtain a target text graph model.

2. The method according to claim 1, characterized in that The contrast loss is obtained by the following steps: Randomly extracting a plurality of comparison samples from a preset sample library, wherein the preset sample library contains positive samples and negative samples set for each preset relational word; Calculate the semantic similarity between each of the comparison samples and the sample description information, determine that the comparison samples whose semantic similarity with the sample description information is higher than a preset similarity threshold are positive comparison samples, and determine that the comparison samples whose semantic similarity with the sample description information is not higher than the preset similarity threshold are negative comparison samples; The contrast loss is determined based on the sample description information, the semantic similarity between the positive comparison sample and the negative comparison sample.

3. The method according to claim 1, characterized in that The step of adding preset noise to the sample image through the candidate text graph model to obtain a noise image, and denoising the noise image based on the sample description information to obtain predicted noise includes: Based on preset Gaussian noise, adding noise to the sample image to obtain a noise image corresponding to the sample image; Encoding the noise image using a preset image encoder, and text encoding the sample description information using a preset text encoder to obtain a target image code and a target text code; Embedding the target image code and the target text code to obtain a target embedded code; wherein the embedding is to concatenate the parts of the target image code and the target text code with the highest semantic similarity of identity information; The noisy image is denoised based on the target embedded coding to obtain predicted noise.

4. The method according to claim 3, characterized in that The method further comprises: Performing image segmentation on the sample image to obtain a plurality of segmentation results, wherein each of the segmentation results contains information about an object, and the objects include people and animals; Extracting image features of each segmentation result to obtain each segmentation feature; Performing cross-attention calculation on each of the segmentation features and the target text encoding to obtain a target mixed encoding; The noisy image is denoised based on the target hybrid coding and the target embedded coding to obtain predicted noise.

5. A Vincent diagram method, characterized in that: The method comprises: Get target description text and reference image; The target description text and the reference image are output to a preset text-based graph model, so that the preset text-based graph model uses a preset text encoder to encode the target description text to obtain a target text code, and encodes the reference image to obtain a target image code; preset noise is added to the target image code to obtain a noise image, and the target text code and the target image code are embedded and cross-attention calculated to obtain a target embedded code and a target mixed code; the noise image is denoised based on the target text code, the target embedded code and the target mixed code to obtain a target image, wherein the preset text-based graph model is pre-trained by the text-based graph model training method described in any one of claims 1 to 4.

6. A device for training a text graph model, characterized in that: The device comprises: A first acquisition module is used to acquire a pre-trained candidate text graph model; A first input module, used to input training data into the candidate text-image model, wherein the training data is a picture-text pair, each of which includes a sample image and sample description information of the sample image, and the sample description information includes a preset relational word and identity information; A denoising module, configured to add preset noise to the sample image through the candidate text graph model to obtain a noise image, and denoise the noise image based on the sample description information to obtain predicted noise; A loss calculation module, used to obtain a target loss based on a noise loss between the predicted noise and the preset noise, and a contrast loss between the sample description information and preset positive and negative samples; A training module is used to fine-tune the candidate text graph model based on the target loss until the target loss converges to obtain a target text graph model.

7. The device according to claim 6, characterized in that The contrast loss is obtained by the following steps: Randomly extracting a plurality of comparison samples from a preset sample library, wherein the preset sample library contains positive samples and negative samples set for each preset relational word; Calculate the semantic similarity between each of the comparison samples and the sample description information, determine that the comparison samples whose semantic similarity with the sample description information is higher than a preset similarity threshold are positive comparison samples, and determine that the comparison samples whose semantic similarity with the sample description information is not higher than the preset similarity threshold are negative comparison samples; The contrast loss is determined based on the sample description information, the semantic similarity between the positive comparison sample and the negative comparison sample.

8. A Wensheng map device, characterized in that: The device comprises: The second acquisition module is used to acquire the target description text and the reference image; The third acquisition module is used to output the target description text and the reference image to a preset text graph model, so that the text graph model uses a preset text encoder to encode the target description text to obtain a target text code, encodes the reference image to obtain a target image code; adds preset noise to the target image code to obtain a noise image, and embeds and cross-attention calculates the target text code and the target image code to obtain a target embedded code and a target mixed code; denoises the noise image based on the target text code, the target embedded code and the target mixed code to obtain a target image, wherein the preset text graph model is pre-trained by the text graph model training method described in any one of claims 1 to 4.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1-4 or 5.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-4 or 5.

Images