Associative text-to-image generation method based on generative pre-training language model

Abstract

This invention discloses an associative text-to-image generation method based on a generative pre-trained language model, comprising: fine-tuning a generative pre-trained model based on a dataset to enable the pre-trained model to acquire semantically preserved existing text information, resulting in a fine-tuned pre-trained model; using ten sentences corresponding to each image in the original dataset as input to the fine-tuned pre-trained model, resulting in a generated dataset output by the model; performing constraint processing and semantic preservation evaluation on the generated dataset to obtain an associative text dataset; and using a DF-GAN-based adversarial generative network model to generate images consistent in cross-modal semantic features between text and images based on the associative text dataset. This invention comprehensively utilizes the associative ability and rich semantic information of the generative pre-trained model, to a certain extent balancing the problem of the imbalance between text and image information in the cross-modal text-to-image generation task of adversarial generative networks.

Description

Technical Field

[0001] This invention relates to the field of image generation technology, and more specifically to an associative text-to-image generation method based on a generative pre-trained language model. Background technology:

[0002] Our experience of the world has gradually shifted from unimodal to multimodal with the development of multimedia technology. Simply put, multimodal refers to information in multiple modalities, including text, images, video, and audio. As the name suggests, multimodal research studies the fusion of these different types of data.

[0003] In the fields of multimodal machine learning and deep learning, text-to-image generation is a promising and increasingly important task. This task has excellent applications in image editing, video editing, stylization generation, and user personalization, and can also assist in design work in the future. For example, users can input requirements to have images drawn to their needs or be helped to complete professional tasks such as designing clothing and modifying patterns based on text content.

[0004] In some existing applications, the training and test sets are derived from the CUB2011 bird dataset, with each CUB image corresponding to 10 descriptions. Recent methods address the issue that previous approaches only selected one text as input to generate the matching target image; however, a single text often only describes a part of an image, which complicates the generation process and leads to low image quality. Therefore, it is necessary to retrieve text closely related to the input text as additional input to enrich the textual information.

[0005] However, methods that expand the text volume by retrieving similar texts contradict the task's objective of supplementing what a single text cannot describe. Furthermore, existing methods limit the retrieval scope to ten sentences; however, "a picture is worth a thousand words," and the text in the dataset may contain inaccuracies or incomplete descriptions. To better describe the image, external knowledge not present in the original text is needed. Additionally, existing methods only implement text-level self-attention mechanisms, neglecting the interaction between image and text information in this cross-modal task. Summary of the Invention

[0006] The purpose of this invention is to provide an associative text-to-image generation method based on a generative pre-trained language model. By introducing a generative pre-trained model to associate and generate richer text information for text-to-image generation, it utilizes a complementation-based fine-tuning method and constructs an adversarial generative network based on cross-modal attention of text and images; the generated images have better semantic consistency.

[0007] Technical solution: To solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0008] Firstly, a method for associative text-to-image generation based on a generative pre-trained language model is provided, including:

[0009] Step S1: Fine-tune the generative pre-trained model based on the dataset to enable the pre-trained model to obtain existing text information with good semantic retention, and obtain the fine-tuned pre-trained model.

[0010] Step S2: Take the ten sentences corresponding to each image in the original dataset as input to the fine-tuned pre-trained model obtained in Step S1 to obtain the generated dataset of the model output; perform constraint processing and semantic preservation evaluation on the generated dataset to obtain the associative text dataset.

[0011] Step S3: Based on the associative text dataset obtained in step S2, generate images that are consistent in cross-modal semantic features of the text images using a DF-GAN-based adversarial generative network model.

[0012] In some embodiments, step S1 includes:

[0013] Step S11: Obtain the dataset and arrange the ten sentences corresponding to each image in the dataset into a sentence string;

[0014] Step S12: Input the sentence strings of the dataset into the pre-trained model for training and fine-tuning to obtain the fine-tuned pre-trained model.

[0015] In some embodiments, step S11, arranging the ten sentences corresponding to each image in the dataset into a sentence string, includes: the dataset includes multiple images, each image corresponding to ten sentences; arranging the ten sentences corresponding to each image into a sentence string according to the following rules:

[0016] The sentence strings are arranged as follows: "$sentence a#sentence b#sentence c#...#sentence 9#sentence 10$";

[0017] The sentence string is divided into two parts: the first part is random initialization, where sentence a, sentence b, and sentence c are three sentences randomly initialized from ten sentences corresponding to an image;

[0018] The second part is the sequential concatenation of the remaining sentences, where "#" and "$" are the separator and start symbol, respectively. GPT-2 generates a structured sentence string. The separator facilitates the decomposition of the generated sentence string, and the start symbol is used to prevent the model from generating sentence strings that are too long or too short.

[0019] In some embodiments, step S12 includes:

[0020] The pre-trained model is a GPT-2 model; the training and fine-tuning methods for the GPT-2 model include:

[0021] Let a given input string of sentences be represented as a sentence sequence [x1,x2,...,xm], where m is the m-th sentence in the string;

[0022] The loss functions for the GPT-2 model during pre-training and fine-tuning are L1(X) and L2(X), respectively, as shown in the following formulas:

[0023]

[0024]

[0025] The pre-training loss function L1(X) adopts the maximum likelihood function, P() represents the conditional probability, Θ is the neural network modeling parameter; i is the traversal value of 0, 1...k; k is less than m, and is the size of the sliding window;

[0026] The fine-tuning process uses supervised learning, and the training samples include sentence sequences [x1, x2, ..., x...]. m The first sentence, x1, is used as the class label; during the fine-tuning of the GPT-2 model, the class label is determined based on the sentence sequence [x1, x2, ..., x...]. m The predicted class label is L2(X);

[0027] The optimization function L3 is a weighted sum of L1 and L2:

[0028] L3=L2+λL1

[0029] Where λ is a hyperparameter, and L1 and L2 are the loss functions of the GPT-2 model during pre-training and fine-tuning, respectively.

[0030] In some embodiments, step S2 involves constraining the generated dataset, including:

[0031] The generated dataset is processed using the proximity principle, format regularization, and sentence selection.

[0032] In some embodiments, step S2, performing a semantic preservation evaluation on the generated dataset, includes:

[0033] The generated dataset is evaluated using the BLEU metric, which includes the BLEU values ​​of samples from the same category but with different poses and backgrounds. a Samples from different categories with significant differences b Samples with similar visual features but belonging to different categories, such as bleu c :

[0034]

[0035] Candidates represents the sentences used to generate the dataset, references are the sentences in the original dataset, and Count represents the count. clip ∑ represents the truncation count, n-gram indicates the number of consecutive words measured in the candidates appearing in the reference, n-gram' indicates the number of consecutive words measured in the candidates, and c, c' are the number of sentences measured simultaneously selected from the dataset; ∑ c∈candidates ,∑ c′∈candidates This indicates that all candidates are included; ∑ n-gram∈c ,∑ n-gram′∈c′ This indicates the number of matching sentences in the candidate variable and the number of specific variables in the reference; Count clip (n-gram) represents the number of sentences that match in the candidates in the reference; Count(n-gram′) represents the number of sentences that match n-gram′ in the candidates;

[0036] Calculate the three metrics bleu for the generated dataset and the original dataset respectively. a bleu b and bleu c ;

[0037] If the ratios of the three metrics of the generated dataset and the original dataset are consistent, it means that the three metrics are semantically consistent with the original dataset. The generated dataset that is semantically consistent with the original dataset is selected as the associative text dataset.

[0038] In some embodiments, in step S3, the DF-GAN-based adversarial generative network model includes: a pre-trained text encoder, a generator, and a discriminator;

[0039] Text Encoder: All text in the Lenovo Text Dataset is encoded by the text encoder, and the output sentence vectors are stored in the text encoding library;

[0040] The generator has two inputs: sentence vectors encoded by a text encoder and random noise from a normal distribution. The random noise is converted to a set size through a fully connected layer and then passes through a series of deep semantic fusion modules to generate image features. In each deep semantic fusion module, the method interacts multiple input sentences with the feature map of the current layer, calculates a cross-modal attention mechanism to distinguish the weight scores of sentences in different generator layers, and then converts the image features into an image through a convolutional layer. Each deep semantic fusion module includes an upsampling layer, a residual block, and a text-image feature fusion block.

[0041] The discriminator uses a series of downsampling layers to convert the image into image features, then concatenates the image features with the sentence vector, and calculates the adversarial loss through a one-step generation process to ensure visual realism and semantic consistency.

[0042] The loss functions for the generator and discriminator are as follows:

[0043]

[0044]

[0045] Among them, L D Let L be the loss function of the discriminator. G Let L be the loss function of the generator. D L G The hinge loss is calculated using the formula, where z is the noise vector sampled by a Gaussian distribution; D is the discriminator, G is the generator, G(z) represents the image generated by the generator, and e is the sentence vector; These represent the distributions of synthetic data, real data, and mismatched data, respectively. Let represent the hinge loss calculation function, x represent the real image, D(x,e) represent the discriminant value of the real image as input to the discriminator, and D(G(z),e) represent the discriminant value of the generated image as input to the discriminator.

[0046] In some embodiments, step S3, the generator's processing includes:

[0047] The generator uses an attention mechanism, let α n The attention mechanism weights corresponding to the nth sentence;

[0048]

[0049] Where X is the input sentence vector, z is the input random noise, s is the attention score function, W represents the number of linear layers that map the sentence vector to the vector in the latent space, and W(z) is the feature map of the image in the current layer of the generator. Given W(z) and X, α is calculated. n .

[0050] In a second aspect, the present invention provides an associative text-to-image generation device based on a generative pre-trained language model, including a processor and a storage medium;

[0051] The storage medium is used to store instructions;

[0052] The processor is configured to operate according to the instructions to perform the steps of the method according to the first aspect.

[0053] Thirdly, the present invention provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0054] The advantages of this invention are as follows: It employs a complementation-based fine-tuning method when fine-tuning the pre-trained model, allowing the generative pre-trained model to retain original semantic information while associating with information not present in the original dataset, thus improving the model's generalization ability. Furthermore, it considers the correlation between the visual information of the input text and the images generated by the current generator layer, constructing a cross-modal attention mechanism based on text and images for image generation. The generated images exhibit better semantic consistency and are closer to real-world images. By comprehensively utilizing the associative ability and rich semantic information of the generative pre-trained model, it balances to some extent the problem of information imbalance between text and images in cross-modal text-to-image generation tasks using generative adversarial networks. It complements missing information in individual text lines while supplementing structured data not present in the training set, and designs a more effective generative adversarial network to generate images, which is beneficial for downstream tasks in natural language processing and cross-modal text generation. Attached Figure Description

[0055] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.

[0056] Figure 2 The flowchart and method diagram for introducing a generative pre-trained model in the embodiments of the present invention are shown below;

[0057] Figure 3 This is a flowchart illustrating the combination of a generative pre-trained model and an adversarial generative network in an embodiment of the present invention.

[0058] Figure 4 These are some examples and index research results of the generative model generation effect in the embodiments of the present invention;

[0059] Figure 5 Comparison of the current model's generation effect with other models (left: original image, middle: embodiment of the present invention, right: DF-GAN). Detailed Implementation

[0060] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0061] In the description of this invention, "several" means one or more, "multiple" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0062] In the description of this invention, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0063] Example 1

[0064] An associative text-to-image generation method based on a generative pre-trained language model includes:

[0065] Step S1: Fine-tune the generative pre-trained model based on the dataset to enable the pre-trained model to obtain existing text information with good semantic retention, and obtain the fine-tuned pre-trained model.

[0066] Step S2: Take the ten sentences corresponding to each image in the original dataset as input to the fine-tuned pre-trained model obtained in Step S1 to obtain the generated dataset of the model output; perform constraint processing and semantic preservation evaluation on the generated dataset to obtain the associative text dataset.

[0067] Step S3: Based on the associative text dataset obtained in step S2, generate images that are consistent in cross-modal semantic features of the text images using a DF-GAN-based adversarial generative network model.

[0068] In some embodiments, step S1 includes:

[0069] Step S11: Obtain the dataset and arrange the ten sentences corresponding to each image in the dataset into a sentence string;

[0070] Step S12: Input the sentence strings of the dataset into the pre-trained model for training and fine-tuning to obtain the fine-tuned pre-trained model.

[0071] In some embodiments, step S11, arranging the ten sentences corresponding to each image in the dataset into a sentence string, includes: the dataset includes multiple images, each image corresponding to ten sentences; arranging the ten sentences corresponding to each image into a sentence string according to the following rules:

[0072] The sentence strings are arranged as follows: "$sentence a#sentence b#sentence c#...#sentence 9#sentence 10$";

[0073] The sentence string is divided into two parts: the first part is random initialization, where sentence a, sentence b, and sentence c are three sentences randomly initialized from ten sentences corresponding to an image;

[0074] The second part is the sequential concatenation of the remaining sentences, where "#" and "$" are the separator and start symbol, respectively. GPT-2 generates a structured sentence string. The separator facilitates the decomposition of the generated sentence string, and the start symbol is used to prevent the model from generating sentence strings that are too long or too short.

[0075] In some embodiments, step S12 includes:

[0076] The pre-trained model is a GPT-2 model; the training and fine-tuning methods for the GPT-2 model include:

[0077] Suppose that a given input string of sentences is represented as a sentence sequence [x1, x2, ..., x...]. m ], where m is the m-th sentence in the sentence string;

[0078] The loss functions for the GPT-2 model during pre-training and fine-tuning are L1(X) and L2(X), respectively, as shown in the following formulas:

[0079]

[0080]

[0081] The pre-training loss function L1(X) adopts the maximum likelihood function, P() represents the conditional probability, Θ is the neural network modeling parameter; i is the traversal value of 0, 1...k; k is less than m, and is the size of the sliding window;

[0082] The fine-tuning process uses supervised learning, and the training samples include sentence sequences [x1, x2, ..., x...]. m The first sentence, x1, is used as the class label; during the fine-tuning of the GPT-2 model, the class label is determined based on the sentence sequence [x1, x2, ..., x...]. m The predicted class label is L2(X);

[0083] The optimization function L3 is a weighted sum of L1 and L2:

[0084] L3=L2+λL1

[0085] Where λ is a hyperparameter, and L1 and L2 are the loss functions of the GPT-2 model during pre-training and fine-tuning, respectively.

[0086] In some embodiments, step S2 involves constraining the generated dataset, including:

[0087] The generated dataset is processed using the proximity principle, format regularization, and sentence selection.

[0088] In some embodiments, step S2, performing a semantic preservation evaluation on the generated dataset, includes:

[0089] The generated dataset is evaluated using the BLEU metric, which includes the BLEU values ​​of samples from the same category but with different poses and backgrounds. a Samples from different categories with significant differences b Samples with similar visual features but belonging to different categories, such as bleu c :

[0090]

[0091] Candidates represents the sentences used to generate the dataset, references are the sentences in the original dataset, and Count represents the count. clip ∑ represents the truncation count, n-gram indicates the number of consecutive words measured in the candidates appearing in the reference, n-gram' indicates the number of consecutive words measured in the candidates, and c, c' are the number of sentences measured simultaneously selected from the dataset; ∑ c∈candidates ,∑ c′∈candidates This indicates that all candidates are included; ∑ n-gram∈c ,∑ n-gram′∈c′ This indicates the number of matching sentences in the candidate variable and the number of specific variables in the reference; Count clip (n-gram) represents the number of sentences that match in the candidates in the reference; Count(n-gram′) represents the number of sentences that match n-gram′ in the candidates;

[0092] Calculate the three metrics bleu for the generated dataset and the original dataset respectively. a bleu b and bleu c ;

[0093] If the ratios of the three metrics of the generated dataset and the original dataset are consistent, it means that the three metrics are semantically consistent with the original dataset. The generated dataset that is semantically consistent with the original dataset is selected as the associative text dataset.

[0094] In some embodiments, in step S3, the DF-GAN-based adversarial generative network model includes: a pre-trained text encoder, a generator, and a discriminator;

[0095] Text Encoder: All text in the Lenovo Text Dataset is encoded by the text encoder, and the output sentence vectors are stored in the text encoding library;

[0096] The generator has two inputs: sentence vectors encoded by a text encoder and random noise from a normal distribution. The random noise is converted to a set size through a fully connected layer and then passes through a series of deep semantic fusion modules to generate image features. In each deep semantic fusion module, the method interacts multiple input sentences with the feature map of the current layer, calculates a cross-modal attention mechanism to distinguish the weight scores of sentences in different generator layers, and then converts the image features into an image through a convolutional layer. Each deep semantic fusion module includes an upsampling layer, a residual block, and a text-image feature fusion block.

[0097] The discriminator uses a series of downsampling layers to convert the image into image features, then concatenates the image features with the sentence vector, and calculates the adversarial loss through a one-step generation process to ensure visual realism and semantic consistency.

[0098] The loss functions for the generator and discriminator are as follows:

[0099]

[0100]

[0101] Among them, L D Let L be the loss function of the discriminator. G Let L be the loss function of the generator. D L G The hinge loss is calculated using the formula, where z is the noise vector sampled by a Gaussian distribution; D is the discriminator, G is the generator, G(z) represents the image generated by the generator, and e is the sentence vector; These represent the distributions of synthetic data, real data, and mismatched data, respectively. Let represent the hinge loss calculation function, x represent the real image, D(x,e) represent the discriminant value of the real image as input to the discriminator, and D(G(z),e) represent the discriminant value of the generated image as input to the discriminator.

[0102] In some embodiments, step S3, the generator's processing includes:

[0103] The generator uses an attention mechanism, let α n The attention mechanism weights corresponding to the nth sentence;

[0104]

[0105] Where X is the input sentence vector, z is the input random noise, s is the attention score function, W represents the number of linear layers that map the sentence vector to the vector in the latent space, and W(z) is the feature map of the image in the current layer of the generator. Given W(z) and X, α is calculated. n .

[0106] As mentioned earlier, text-to-image generation often results in poor image quality due to the lack of text information. To address this issue, this invention proposes a method that introduces a generative pre-trained model for text association and enrichment, which is then used for text-to-image generation.

[0107] In some embodiments, such as Figure 1 As shown, this invention explains the entire process from the generative pre-trained model to the final image generated by the adversarial generative network. It can be seen that this application first inputs the original text into the pre-trained model and then performs fine-tuning. Finally, the fine-tuned pre-trained model is used to compile the final dataset. This application pre-processes the text library to generate a sentence vector library through a text encoder, and finally obtains the text encoding directly to accelerate model training. After obtaining the text encoding, this application uses an adversarial generative network based on a cross-modal attention mechanism to generate the text-to-image mapping.

[0108] Therefore, the method of this invention comprises three steps: S1, fine-tuning the generative pre-trained model based on a small-scale dataset to allow the pre-trained model to obtain existing text information with good semantic retention; S2, based on the obtained pre-trained model, and since the generative pre-trained model can generate data not present in the original dataset, it can generate more generalized semantic information through association and inference, thus leveraging its rich associative capabilities to establish connections from the original dataset to richer information, associating a single text from existing data with richer text information; S3, using an adversarial generative network, with the help of an attention mechanism, selectively biasing the weights of different sentences in different levels of the model generator, thereby generating images consistent in cross-modal semantic features of text images. These will be described in detail below:

[0109] S1: Fine-tune the generative pre-trained model based on a small dataset to enable the pre-trained model to retain semantically accurate existing text information.

[0110] This application requires selectively searching for information related to the corresponding text from an external knowledge base, and training a generative pre-trained model using sentence prompts and completion-based fine-tuning methods. This enables the completion of missing text information and the generation of meaningful text information beyond the dataset.

[0111] Step S11: Candidate text selection and sentence string editing

[0112] The existing dataset contains ten sentences for each image, meaning one image corresponds to ten sentences. This step takes one of these texts as input, and the goal is for the model to generate a sentence string containing ten sentences. Model convergence is considered successful if the model retains the original ten sentence information.

[0113] Overall, the sentence string is divided into two parts. The first part is random initialization. Sentences a, b, and c (not marked) are three sentences randomly initialized from ten sentences corresponding to an image. Because models like GPT-2 can only generate subsequent sentences based on previous sentences and cannot recall previous sentences based on subsequent sentences, this application shuffles the order of the ten sentences. During each training session, the model attempts to reconstruct the original semantic information from three different sentences, which reduces the error caused by positional information. The second part is the sequential concatenation of the remaining sentences, where "#" and "$" are separators and start symbols, respectively. Since the GPT-2 model can generate structured text data, this application still needs to decompose the obtained sentence string after the model completes generation. Separators can better help this application decompose the generated sentence string, while start symbols prevent the model from generating sentence strings that are too long or too short.

[0114] Step S12: Generative pre-trained model training process

[0115] The process of fine-tuning the pre-trained model in this paper is the inference process of the GPT-2 model. The model is trained in two parts. First, the language model goes through an unsupervised pre-training process. The GPT-2 model used in this application was pre-trained on the Shakespeare play dataset.

[0116] This paper takes a sentence string as input and aims to reconstruct the original sentence string from the input sentences to ensure the model's generation performance. Given the input [x1,x2,...,xm], the loss functions for pre-training and fine-tuning are L1(X) and L2(X), respectively, as shown in the formulas below, where L1(X) is the maximum likelihood function, and [x1,x2,...,xm] is the input sentence. The fine-tuning process uses supervised learning, with training samples including the sentence sequence [x1,x2,...,xm] and a label using x1 as the class label. During GPT fine-tuning, the class label is predicted based on the sentence sequence [x1,x2,...,xm], which is L2(X).

[0117] Step S13: Generative pre-trained model inference process

[0118] After obtaining the pre-trained GPT-2 model, the model can reconstruct the remaining ten sentences in the original dataset from a given sentence. The GPT-2 model can generate structured text data, and the final generated result can retain delimiters and start characters.

[0119] The dataset used in this paper is the CUB dataset, which contains 11,788 images of birds from 200 different species, with each image corresponding to ten sentences. This application simulates the generation process of a typical generative adversarial network, randomly selecting one sentence from the ten sentences as the input sentence, requiring GPT-2 to restore a part of the original description that was not described in the original text, i.e., completion-based generation.

[0120] The input sentence will ultimately be output as a sentence string and stored for subsequent splitting and training of the adversarial generative network. It's important to note that this section only uses the training set portion of the CUB dataset (8855 images from 150 different bird species and their corresponding texts) to train the GPT-2 model. However, to ensure fairness when inferring and generating text, a test set (2933 images from 50 different bird species and their corresponding texts) is used.

[0121] S2. Based on the obtained pre-trained model, leverage its rich associative capabilities to establish connections from the original dataset to richer information, and associate a single text from the existing data with richer textual information.

[0122] The entire dataset is used as input, and the finely tuned pre-trained model is used to generate new text. Finally, the generated text is compiled into a new dataset. Due to the large increase in text information, pre-compiling into a dataset will greatly accelerate the training of the adversarial generative network, while also ensuring that the generated dataset retains good semantic information.

[0123] S21. Based on the fine-tuned pre-trained model and the original dataset, perform rich text inference and generation.

[0124] The original dataset contains ten sentences corresponding to one image. These ten sentences are then input into the GPT-2 model to generate a dataset with richer semantic information.

[0125] When selecting output results, this application adopts the following three principles to screen and organize the sentences generated by association:

[0126] 1. Proximity Principle: The GPT-2 model is a language model that predicts the next word with maximum posterior probability. As the length of the sentences generated by the model increases, the feasibility of sentences at the end of the word generation decreases. Therefore, this application performs different degrees of pruning on the results generated by the GPT-2 model each time, ensuring that the sentences at the front that meet the requirements can be included in the associative text in the dataset.

[0127] 2. Format regularization: Due to its more generalized knowledge base, the sentences generated by the GPT-2 model often result in a large error between the generated results and the expected results. At the most basic level, this application constrains the format of the sentences, requiring the generated sentences to retain the start character and delimiter defined in this application. If the condition is not met, the sentence is discarded and regenerated iteratively.

[0128] 3. Sentence selection: Since the birds in the dataset are often quite distinct, the GPT-2 model has difficulty ensuring a good fit for all bird species under the same number of training epochs during fine-tuning. Underfitting and overfitting are common. For birds with poor fitting, this application optimizes them separately by adding the L2 distance between them and the original sentence, thereby ensuring the credibility of the generated text in special cases.

[0129] S22. Evaluate the semantic preservation of the generated dataset.

[0130] Preserving sentence semantics is a difficult task to measure. In the method of this application, it is necessary to perform semantic analysis on the dataset generated by the generative pre-trained model, requiring it to retain the same level of semantic information as the original dataset.

[0131] The bleu-2 metric used in this section is expressed in the following formula. The bleu metric is a measure of the relevance of sentences. Given the special nature of the dataset text, this paper uses the bleu-2 metric to evaluate the generated results, that is, two adjacent words that are the same are considered to be the score.

[0132]

[0133] The following three scenarios were selected as references.

[0134] 1- Samples of the same category but with different poses and backgrounds (bleu) a

[0135] 2- Samples from different categories with significant differences b

[0136] 3- Samples with similar visual features but belonging to different categories (bleu) c

[0137] Intuitively, sentences describing the same bird species should be semantically close; however, in reality, even images of the same bird species can have significantly different visual characteristics due to factors such as posture, background, and environment. This application argues that bleu a It will be a more moderate value.

[0138] Birds that differ significantly in visual characteristics often exhibit substantial semantic differences in their texts. b This application considers it to be a relatively small value. This application searches for birds with similar appearances among different bird species. To verify that sentences corresponding to birds with similar appearances are semantically related, bleu... c This will result in a larger value. Using this as a benchmark, we determine whether the generated dataset retains consistent semantics (see Table 1). As shown in Table 1, all three metrics are semantically consistent with the original dataset.

[0139] Table 1

[0140] index CUB Multi-CUB <![CDATA[bleu a ]]> 0.230 0.317 <![CDATA[bleu b ]]> 0.212 0.298 <![CDATA[bleu c ]]> 0.317 0.378

[0141] S3. Using an adversarial generative network, with the help of an attention mechanism, the weights of different sentences in different levels of the model generator are selectively biased, thereby generating images that are consistent in cross-modal semantic features of text images.

[0142] This paper employs a DF-GAN-based adversarial generative network for image generation. This model is a single-stage text-to-image generation process that achieves good generation results with a relatively small number of parameters. The generated associative text dataset is input into the model, and a cross-modal attention mechanism based on text and images is added to optimize the image generation process.

[0143] Step S31: Text encoder pre-training and pre-storage

[0144] Since the Lenovo text dataset has more than ten times the capacity of the original dataset, encoding each input text line by line using the original strategy would consume a significant amount of training time. This method, however, pre-processes all text using a text encoder and stores the output text in an encoding book to facilitate subsequent model training and optimization.

[0145] Step S32: Text-to-image generative adversarial network based on multi-text association generated by generative pre-trained language models.

[0146] This paper adopts an adversarial generative network model based on DF-GAN, which abandons the previous stacked structure and uses only a generator, a discriminator, and a pre-trained text encoder.

[0147] The generator takes two inputs: a sentence vector encoded by a text encoder and random noise taken from a normal distribution. First, the noise is fed into a fully connected layer and reshaped to the required size. Then, it passes through a series of DF blocks to generate image features. Each DF block includes an upsampling layer, a residual block, and a text-image feature fusion block. Finally, a convolutional layer converts the image features into an image.

[0148] The discriminator uses a series of downsampling layers to convert the image into image features, then concatenates the image features with the sentence vector, and then performs a one-step generation to calculate the adversarial loss to ensure visual realism and semantic consistency.

[0149] Step S33: Text-to-Image Generator Network Based on Cross-Modal Attention Mechanism

[0150] Because GAN networks generate images by combining feature maps transformed from text and noise, when W(z) is the feature map of the current generation layer, the model will adaptively fit the current text plus noise vector with the corresponding image vector, so that the sentence vectors in each generator layer have different weights. For example, when the generator generates an image of a bird, fine-grained information such as the bird's feathers, beak, and color tends to have greater weight in the later layers of the generator, while coarse-grained information such as the background and contours have greater weight in the earlier layers.

[0151] This invention uses examples to demonstrate the effectiveness of the proposed method. The experiments are based on the CUB bird dataset, where each image corresponds to ten sentences. When generating the corresponding image, one sentence is randomly selected from these sentences for generation. See the attached image for the specific generation results. Figure 5 As shown.

[0152] Example 2

[0153] Secondly, this embodiment provides an associative text-to-image generation device based on a generative pre-trained language model, including a processor and a storage medium;

[0154] The storage medium is used to store instructions;

[0155] The processor is configured to operate according to the instructions to perform the steps of the method according to Embodiment 1.

[0156] Example 3

[0157] Thirdly, this embodiment provides a storage medium on which a computer program is stored, which, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0158] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0160] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0161] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0162] As is known from common technical knowledge, this invention can be implemented through other embodiments that do not depart from its spirit or essential characteristics. Therefore, the disclosed embodiments described above are merely illustrative in all respects and are not the only ones. All modifications within the scope of this invention or its equivalents are included in this invention.

Claims

1. A method for associative text-to-image generation based on a generative pre-trained language model, characterized in that, include: Step S1: Fine-tune the generative pre-trained model based on the dataset to enable the pre-trained model to obtain existing text information with good semantic retention, and obtain the fine-tuned pre-trained model. Step S2: Take the ten sentences corresponding to each image in the original dataset as input to the fine-tuned pre-trained model obtained in step S1, and obtain the generated dataset of the model output. Constraints and semantic preservation evaluations are applied to the generated dataset to obtain the associative text dataset; Step S3: Based on the associative text dataset obtained in step S2, generate images that are consistent in cross-modal semantic features of text images using a DF-GAN-based adversarial generative network model. Step S1 includes: Step S11: Obtain the dataset and arrange the ten sentences corresponding to each image in the dataset into a sentence string. This includes: the dataset contains multiple images, and each image corresponds to ten sentences; arrange the ten sentences corresponding to each image into a sentence string according to the following rules: the sentence string is arranged as follows: "$sentence a#sentence b#sentence c#...#sentence 9#sentence 10$"; the sentence string is divided into two parts: the first part is random initialization, where sentences a, b, and c are three sentences randomly initialized from the ten sentences corresponding to one image; the second part is the sequential concatenation of the remaining sentences, where "#" and "$" are the separator and start symbol, respectively. GPT-2 generates structured sentence strings. The separator facilitates the decomposition of the generated sentence string, and the start symbol is used to prevent the model from generating sentence strings that are too long or too short. Step S12: Input the sentence strings from the dataset into the pre-trained model for training and fine-tuning to obtain the fine-tuned pre-trained model, including: the pre-trained model is a GPT-2 model; the training and fine-tuning method of the GPT-2 model includes: let a given input sentence string be represented as a sentence sequence [x1, x2, ..., x...]. m ], where m is the m-th sentence in the sentence string; the loss functions of the GPT-2 model during pre-training and fine-tuning are L1(x) and L2(x), respectively, as shown in the following formulas: The pre-training loss function L1(X) adopts the maximum likelihood function, P() represents the conditional probability, Θ is the neural network modeling parameter; i is the traversal value of 0, 1...k; k is less than m, and is the size of the sliding window; The fine-tuning process uses supervised learning, and the training samples include sentence sequences [x1, x2, ..., x...]. m The first sentence, x1, is used as the class label; during the fine-tuning of the GPT-2 model, the class label is determined based on the sentence sequence [x1, x2, ..., x...]. m The predicted class label is L2(X); The optimization function L3 is a weighted sum of L1 and L2: L3 = L2 + λL1, where λ is a hyperparameter, and L1 and L2 are the loss functions of the GPT-2 model during pre-training and fine-tuning, respectively. In step S2, the generated dataset is subjected to constraints, including: processing the generated dataset using the proximity principle, format regularization, and sentence selection; and semantic preservation evaluation of the generated dataset is performed, including: The generated dataset is evaluated using the BLEU metric, which includes the BLEU values ​​of samples from the same category but with different poses and backgrounds. a Samples from different categories with significant differences b Samples with similar visual features but belonging to different categories, such as bleu c : Candidates represents the sentences used to generate the dataset, references are the sentences in the original dataset, and Count represents the count. clip ∑ represents the truncation count, n-gram represents the number of consecutive words measured in the candidates appearing in the reference, n-gram' represents the number of consecutive words measured in the candidates, and c, c' are the number of sentences measured simultaneously selected from the dataset; ∑ c∈candidates ,∑ c′∈candidates This indicates that all candidates are included; ∑ n-gram∈c , Σ n-gram′∈c′ This indicates the number of matching sentences in the candidate variable and the number of specific variables in the reference; Count clip (n-gram) represents the number of sentences that match in the candidates in the reference; Count(n-gram′) represents the number of sentences that match n-gram′ in the candidates; Calculate the three metrics bleu for the generated dataset and the original dataset respectively. a bleu b and bleu c ; If the ratios of the three metrics of the generated dataset and the original dataset are consistent, it means that the three metrics are semantically consistent with the original dataset. The generated dataset that is semantically consistent with the original dataset is selected as the associative text dataset.

2. The associative text-to-image generation method based on a generative pre-trained language model according to claim 1, characterized in that, The DF-GAN-based adversarial generative network model includes: a pre-trained text encoder, a generator, and a discriminator; Text Encoder: All text in the Lenovo Text Dataset is encoded by the text encoder, and the output sentence vectors are stored in the text encoding library; The generator has two inputs: sentence vectors encoded by a text encoder and random noise from a normal distribution. The random noise is converted to a set size through a fully connected layer and then passes through a series of deep semantic fusion modules to generate image features. In each deep semantic fusion module, the method interacts multiple input sentences with the feature map of the current layer, calculates a cross-modal attention mechanism to distinguish the weight scores of sentences in different generator layers, and then converts the image features into an image through a convolutional layer. Each deep semantic fusion module includes an upsampling layer, a residual block, and a text-image feature fusion block. The discriminator uses a series of downsampling layers to convert the image into image features, then concatenates the image features with the sentence vector, and calculates the adversarial loss through a one-step generation process to ensure visual realism and semantic consistency. The loss functions for the generator and discriminator are as follows: Among them, L D Let L be the loss function of the discriminator. G In the loss function of the generator, z is a noise vector sampled by a Gaussian distribution; D is the discriminator, G is the generator, G(z) represents the image generated by the generator, and e is the sentence vector; These represent the distributions of synthetic data, real data, and mismatched data, respectively. Let represent the hinge loss calculation function, x represent the real image, D(x,e) represent the discriminant value of the real image as input to the discriminator, and D(G(z),e) represent the discriminant value of the generated image as input to the discriminator.

3. The associative text-to-image generation method based on a generative pre-trained language model according to claim 2, characterized in that, The generator's processing includes: The generator uses an attention mechanism, let α n The attention mechanism weights corresponding to the nth sentence; Where X is the input sentence vector, z is the input random noise, s is the attention score function, W represents the number of linear layers that map the sentence vector to the vector in the latent space, and W(z) is the feature map of the image in the current layer of the generator. Given W(z) and X, α is calculated. n .

4. An associative text-to-image generation device based on a generative pre-trained language model, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 3.

5. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

Images