A method for optimizing prompts in large-scale text-based graph models based on scene graphs, electronic devices, and media.

Abstract

This invention discloses a method, electronic device, and medium for optimizing prompt words in a large text-to-image model based on scene graphs. The method includes: acquiring a first prompt word and parsing it to obtain a first scene graph; inputting the first prompt word into a large text-to-image model to generate a corresponding first image; semantically aligning the first scene graph with the first image to obtain a second scene graph; inputting the first prompt word and the second scene graph into a pre-trained prompt word optimization model to output a second prompt word; further including: encoding the first prompt word to obtain text features, encoding and convolving the second scene graph to obtain semantic features; concatenating the text features and semantic features and inputting them into a text editor / decoder to output actions including at least deletion, rearrangement, and insertion; and processing the first prompt word based on the output deletion, rearrangement, and insertion actions to obtain the second prompt word.

Description

Technical Field

[0001] This invention belongs to the field of generative artificial intelligence prompt word engineering, and specifically relates to a method for optimizing prompt words in a large text-to-image model based on scene graphs, as well as electronic devices and media. Background Technology

[0002] With the rapid development of artificial intelligence, text-to-image (TPI) models have become a cutting-edge technology in digital art and visual content generation. These models, such as Stable Diffusion, DALL-E 3, and Imagen, can automatically convert text descriptions (hereinafter referred to as cue words) into images and have been widely used in advertising, entertainment, and education. The performance of these models heavily relies on high-quality text cue input; therefore, numerous automated cue word optimization techniques have recently emerged. The specific workflow of these techniques involves inputting low-quality cue words, automatically converting them into high-quality cue words, and finally using them in the TPI process. While existing cue word optimization techniques can improve the aesthetic expression of the final image, they generally carry a high risk of semantic shift. Specifically, semantic shift occurs during the conversion of low-quality cue words into high-quality cue words, leading to a semantic mismatch between the final generated image and the original low-quality cue words.

[0003] In existing technologies, prompts are rewritten and optimized based on language models to improve the aesthetic expression of text-to-image models. However, the following drawbacks exist:

[0004] (1) Semantic Representation Problem: The cue word optimization model needs to fully perceive and analyze the semantics and generation intent of the original low-quality cue words in order to accurately preserve and express the original semantics in high-quality cue words. Current methods mostly directly use low-quality cue words as semantic representation input, lacking reasoning and abstraction of their expression in the image domain, as well as modeling of detailed semantics and their interrelationships.

[0005] (2) Training objective problem: Modeling the training objective of the optimization model relies on an accurate measure of the semantic consistency between low-quality cue words and final images, which is difficult in practice and is ignored or abandoned in most methods.

[0006] (3) Training cost problem: The prompt word optimization model is directly connected to the Wenshengtu large model in the workflow. The traditional gradient descent training method has a high training cost and there is a performance loss problem caused by sampling.

[0007] In existing technologies, reinforcement learning-based training paradigms employ existing image-text similarity assessment techniques in reward function modeling to measure the semantic consistency between low-quality cue words and the final image. These image-text similarity assessment methods have the following drawbacks:

[0008] (1) Poor accuracy: Existing image-text consistency assessment methods usually map the image and text as a whole to the same space and calculate their distance. This coarse-grained method is prone to ignoring the alignment of semantic details, especially when dealing with complex scenarios or performing combinatorial reasoning.

[0009] (2) Lack of interpretability: The coarse-grained methods mentioned above are based solely on the processing and output of neural networks, and their results are difficult to interpret and adjust, which limits the potential space for model optimization.

[0010] (3) Narrow scope of effectiveness: Some evaluation methods attempt fine-grained semantic decomposition and modeling, but often rely on specific datasets and preset semantic categories, which greatly limits their application in open-world scenarios. Summary of the Invention

[0011] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, electronic device, and medium for optimizing prompt words in a large text-to-image model based on scene graphs.

[0012] In a first aspect, embodiments of the present invention provide a method for optimizing prompt words in a large text-to-image model based on a scene graph, the method specifically including the following steps:

[0013] Obtain the first prompt word, parse the first prompt word to obtain the first scene image; input the first prompt word into the text-to-image large model to generate the corresponding first image; semantically align the first scene image with the first image to obtain the second scene image;

[0014] The first prompt word and the second scene image are input into a pre-trained prompt word optimization model, and the second prompt word is output. This includes: encoding the first prompt word to obtain text features, encoding and convolving the second scene image to obtain semantic features; concatenating the text features and semantic features and inputting them into a text editing decoder, which outputs actions including at least deletion, rearrangement, and insertion; and processing the first prompt word based on the output deletion, rearrangement, and insertion actions to obtain the second prompt word.

[0015] The training process of the prompt word optimization model includes:

[0016] Calculate the first image-text consistency evaluation index between the second scene image and the second image, calculate the second image-text consistency evaluation index between the first prompt word and the second image, and obtain the image aesthetic evaluation index based on the second image;

[0017] Reward functions are set based on the first image-text consistency evaluation index, the second image-text consistency evaluation index, and the image aesthetics evaluation index. The training of the prompt word optimization model is driven by the reward function through the reinforcement learning paradigm.

[0018] Secondly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described method for optimizing prompt words based on scene graph-based text-to-image large model.

[0019] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the above-described method for optimizing prompt words in a large text-to-image model based on a scene graph.

[0020] Fourthly, embodiments of the present invention provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-described method for optimizing prompt words based on scene graph-based text-to-image large model.

[0021] The beneficial effects of this invention are:

[0022] This invention provides a method for optimizing prompt words in a large text-to-image model based on scene graphs. It sets a reward function based on a first image-text consistency evaluation index, a second image-text consistency evaluation index, and an image aesthetic evaluation index. Using a reinforcement learning paradigm, the reward function drives the training of the prompt word optimization model. The first prompt word and a second scene graph are input into the pre-trained prompt word optimization model to obtain the second prompt word. This invention prevents semantic shift during the transformation of low-quality first prompt words into high-quality second prompt words, avoiding semantic mismatch between the final generated image and the original prompt word. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating the method for optimizing prompt words in a large text-generated image model based on a scene graph, provided in an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of the inference and training process of the prompt word optimization model provided in an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of the prompt word optimization model structure provided in an embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram of the decoder structure provided in an embodiment of the present invention;

[0028] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0029] The present invention will be further described below with reference to embodiments. The description of the embodiments below is only for the purpose of helping to understand the present invention. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principle of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

[0030] Furthermore, the technical solutions of the various embodiments can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0031] like Figure 1 As shown, this invention proposes a method for optimizing prompt words in a large text-to-image model based on scene graphs. The method specifically includes the following steps:

[0032] Step S1: Obtain the first prompt word, parse the first prompt word to obtain the first scene image; input the first prompt word into the text-to-image large model to generate the corresponding first image; semantically align the first scene image with the first image to obtain the second scene image.

[0033] Specifically, step S1 includes the following sub-steps:

[0034] Step S101: Obtain the first prompt word, and parse the first prompt word to obtain the first scene image;

[0035] Further, the original first prompt word is denoted as x = [w1, w2, ...]. The first scene graph is obtained by parsing the first prompt word using a text parser, denoted as S = {O, R}, where O = {o...} i} is the entity set in the first scene graph S, R = {(o i ,p i→j ,o j )} is the set of relations in the first scene graph S, o i Let o represent the i-th entity. j Let p represent the j-th entity. i→j This represents the relationship between the i-th entity and the j-th entity.

[0036] Step S102: Input the first prompt word x into the text-based image model to generate the corresponding first image I.

[0037] It should be noted that the first image I is the semantic representation of the first cue word x in the image domain.

[0038] Step S103: Semantically align the first scene graph S with the first image I to obtain the second scene graph S. * .

[0039] In this example, the entity set O in the first scene graph S and the first image I are input into the open word set object detection model GLIP (Grounded Language-Image Pre-training) to determine whether the entities in the entity set O in the first scene graph S exist in the first image I;

[0040] If an entity in entity set O in the first scene graph S does not exist in the first picture I, then the entity and its associated relationships are deleted from the first scene graph S.

[0041] When an entity in entity set O in the first scene graph S exists in the first image I, the relation set R = {(o...} in the first scene graph S... i ,p i→j ,o j The corresponding head entity o i The region and tail entity o in the first image I j Regions and relationships in the first image I i→j Input is fed into the image-text question answering model BLIP (Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation) to determine the head entity. i Tail entity o j The relationship between p i→j Whether it is true or false; if the head entity o i Tail entity o j The relationship between p i→j If this is not true, then relation p will be... i→j Delete from the first scene diagram S;

[0042] The first scene diagram S, after deleting all entities and relationships that do not exist in the first image I, is used as the second scene diagram S. * This completes the semantic alignment between the first scene image S and the first image I.

[0043] Step S2 involves inputting the first prompt word and the second scene image into a pre-trained prompt word optimization model and outputting the second prompt word. This includes: encoding the first prompt word to obtain text features, encoding and convolving the second scene image to obtain semantic features; concatenating the text features and semantic features and inputting them into a text editing decoder, which outputs actions including at least deletion, rearrangement, and insertion; and processing the first prompt word based on the output deletion, rearrangement, and insertion actions to obtain the second prompt word.

[0044] Specifically, such as Figure 2 As shown, step S2 includes the following sub-steps:

[0045] Step S201: A multimodal pre-trained neural network CLIP (Contrastive Language-Image Pre-training) is used to encode all nodes in the second scene graph into embedding vectors; a multi-layer graph convolutional network is then used to convolve the embedding vectors to extract semantic features. The textual features and semantic features are then concatenated to form multimodal features.

[0046] Step S202: Input the multimodal features obtained by concatenating text features and semantic features into the text editing decoder. The text editing decoder includes a first sub-decoder, a second sub-decoder, and a third sub-decoder.

[0047] Among them, such as Figure 3 As shown, the first sub-decoder takes multimodal features as input and deletion actions as output; the deletion actions are used to determine whether each token in the first prompt word has been deleted.

[0048] The second sub-decoder takes multimodal features as input and a rearrangement action as output; the rearrangement action is used to rearrange the order of the tokens retained in the first cue word.

[0049] The third sub-decoder takes multimodal features as input and an insertion action as output; the insertion action is used to insert a new token into the rearranged first prompt word.

[0050] For example, in this instance, the decoding layer of an instance of the Transformer architecture, BART (Bidirectional and Auto-Regressive Transformers), is used as the basic structure of the decoder, as shown in the following figure. Figure 4 As shown. The decoder consists of three sub-modules: a deletion action decoder, a rearrangement action decoder, and an insertion action decoder. These are all constructed by stacking the above basic structure and output the deletion action, rearrangement action, and insertion action for the first prompt word x in sequence.

[0051] It should be noted that in this example, the deletion and rearrangement actions are not generated autoregressively. Specifically, the corresponding actions for the entire first prompt word x are obtained all at once. However, the generation process for the insertion action is autoregressive. Specifically, the tokens that need to be inserted into the first prompt word x are generated one by one.

[0052] Step S203: Based on the deletion, rearrangement, and insertion actions in the output, process the first prompt word to obtain the second prompt word x. * .

[0053] The training process of the prompt word optimization model includes:

[0054] Specifically, in this example, given prompt words that have not been trained using reinforcement learning, the model weights Φ are optimized. sft The entire editing process is treated as a Markov decision process, and training is driven by a multi-objective joint reward function r.

[0055] Step S100: Calculate the first image-text consistency evaluation index between the second scene image and the second image; calculate the second image-text consistency evaluation index between the first prompt word and the second image; and obtain the image aesthetic evaluation index based on the second image; including:

[0056] The first image-text consistency evaluation index between the second scene image and the second picture is calculated as follows:

[0057]

[0058] L(s,I,γ s ) = clip(log(P(s|I,γ) s ))log(∈),log(1-∈))

[0059] In the formula, γ represents the second scene graph S * With the second picture I * The correspondence between them; γ o Represents the second scene diagram S * The entity o in the second image I * The region in; γ o,o′ Represents the second scene diagram S * In the second image I, entities o, o′, and the relationship p between entities o and o′ are shown. * In the region, O * R represents the entity set in the second scene graph. *represents the set of relations in the second scene graph; L represents the consistency score of a single entity or relation; clip(.) represents the clipping function; s is an entity or relation; ∈ represents a hyperparameter; log(∈) represents the minimum value in the clipping function; log(1-∈) represents the maximum value in the clipping function; in this example, ∈ is set to 0.2.

[0060] The second image is input into the aesthetics score predictor to obtain the image's aesthetic evaluation index;

[0061] Input the first prompt word and the second image into the CLIP model to obtain the second image-text consistency evaluation index.

[0062] Step S200: Based on the first image-text consistency evaluation index, the second image-text consistency evaluation index, and the image aesthetics evaluation index, a reward function is set, and the training of the prompt word optimization model is driven by the reward function through a reinforcement learning paradigm. This includes:

[0063] The expression for the reward function is as follows:

[0064] r(x,x * )=α1*r AES (I * )+α2*r SG (S * ,I * )+α3*r OV (x,I * )

[0065] In the formula, x represents the first prompt word, x * This indicates the second cue word, α1, α2, and α3 represent weight coefficients, and r AES (I * ) represents the image aesthetic evaluation index, r SG (S * ,I * ) represents the first consistency evaluation index between text and graphics, r OV (x,I * () indicates the second indicator for evaluating the consistency between text and graphics;

[0066] In this example, reinforcement learning training is performed using the Proximal Policy Optimization (PPO) algorithm, and the KL divergence penalty and entropy reward techniques are employed.

[0067] The optimized reward function is expressed as follows:

[0068]

[0069] In the formula, D represents the training dataset (in this example, the Lexica dataset, DiffusionDB dataset, and COCO dataset are used), Φ represents the weights of the prompt word optimization model in the current training round, and Φ sft KL(·) represents the weights of the untrained prompt word optimization model, KL(·) represents the relative entropy penalty, H(·) represents the entropy reward, and α and β represent the weight coefficients.

[0070] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-described scene graph-based text-to-image large model prompting optimization method. Figure 5 The diagram shown is a hardware structure diagram of any device with data processing capabilities for the scene graph-based text-to-image large model suggestion optimization method provided in this embodiment of the invention. (Except for...) Figure 5 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0071] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the above-described method for optimizing large-scale text-to-image prompts based on scene graphs. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0072] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A method for optimizing prompt words in a large text-to-image model based on scene graphs, characterized in that, The method specifically includes the following steps: Obtain the first prompt word, parse the first prompt word to obtain the first scene image; input the first prompt word into the text-to-image large model to generate the corresponding first image; semantically align the first scene image with the first image to obtain the second scene image; The first prompt word and the second scene image are input into a pre-trained prompt word optimization model, and the second prompt word is output. This includes: encoding the first prompt word to obtain text features, encoding and convolving the second scene image to obtain semantic features; concatenating the text features and semantic features and inputting them into a text editing decoder, which outputs actions including at least deletion, rearrangement, and insertion; and processing the first prompt word based on the output deletion, rearrangement, and insertion actions to obtain the second prompt word. The training process of the prompt word optimization model includes: Calculate the first image-text consistency evaluation index between the second scene image and the second image, calculate the second image-text consistency evaluation index between the first prompt word and the second image, and obtain the image aesthetic evaluation index based on the second image; Reward functions are set based on the first image-text consistency evaluation index, the second image-text consistency evaluation index, and the image aesthetics evaluation index. The training of the prompt word optimization model is driven by the reward function through the reinforcement learning paradigm.

2. The method for optimizing prompt words in a large text-based graph model based on scene graphs according to claim 1, characterized in that, The first scene image obtained by parsing the first prompt word includes: Parsing the first prompt word yields the first scene diagram, denoted as S = {O, R}, where O = {o i } is the entity set in the first scene graph S, R = {(o i p i→j o j )} is the set of relations in the first scene graph S, o i Let o represent the i-th entity. j Let p represent the j-th entity. i→j This represents the relationship between the i-th entity and the j-th entity.

3. The method for optimizing prompt words in a large text-based graph model based on a scene graph, as described in claim 1 or 2, is characterized in that... The process of semantically aligning the first scene image with the first picture includes: Input the entity set O in the first scene graph S and the first image I into the open word set object detection model GLIP to determine whether the entity in the entity set O in the first scene graph S exists in the first image I. If an entity in entity set O in the first scene graph S does not exist in the first picture I, then the entity and its associated relationships are deleted from the first scene graph S. When an entity in entity set O in the first scene graph S exists in the first image I, the relation set R = {(o...} in the first scene graph S... i p i→j o j The corresponding head entity o i The region and tail entity o in the first image I j Regions and relationships in the first image I i→j Input into the BLIP (Browser-Based Interaction) model to determine the entity head. i Tail entity o j The relationship between p i→j Whether it is true or false; if the head entity o i Tail entity o j The relationship between p i→j If this is not true, then relation p will be... i→j Delete from the first scene diagram S; The first scene graph S, after deleting all entities and relations that do not exist in the first image I, is used as the second scene graph S*, thus completing the semantic alignment between the first scene graph S and the first image I.

4. The method for optimizing prompt words in a large text-to-image model based on a scene graph according to claim 1, characterized in that, The process of encoding and convolving the second scene graph to obtain semantic features includes: Encode all nodes in the second scene graph into embedding vectors; Semantic features are extracted by convolving the embedded vectors.

5. The method for optimizing prompt words in a large text-to-image model based on a scene graph according to claim 1, characterized in that, The process of concatenating text features and semantic features and inputting them into a text editing decoder, outputting actions including at least deletion, rearrangement, and insertion, includes: The multimodal features, which are concatenated from text features and semantic features, are input into the text editing decoder, which includes a first sub-decoder, a second sub-decoder, and a third sub-decoder. The first sub-decoder takes multimodal features as input and deletion actions as output; the deletion actions are used to determine whether each token in the first prompt word has been deleted. The second sub-decoder takes multimodal features as input and a rearrangement action as output; the rearrangement action is used to rearrange the order of the tokens retained in the first cue word. The third sub-decoder takes multimodal features as input and an insertion action as output; the insertion action is used to insert a new token into the rearranged first prompt word.

6. The method for optimizing prompt words in a large text-to-image model based on a scene graph according to claim 1, characterized in that, The process of calculating the first image-text consistency evaluation index between the second scene image and the second picture, calculating the second image-text consistency evaluation index between the first prompt word and the second picture, and obtaining the image aesthetic evaluation index based on the second picture includes: The first image-text consistency evaluation index between the second scene image and the second picture is calculated as follows: L(s,I,γ s )=clip(log(P(s|I,γ s )),log(∈),log(1-∈)) In the formula, γ represents the second scene graph S * With the second picture I * The correspondence between them; γ o Represents the second scene diagram S * The entity o in the second image I * The region in; γ o，o′ Represents the second scene diagram S * In the second image I, entities o, o′, and the relationship p between entities o and o′ are shown. * In the region, O * R represents the entity set in the second scene graph. * represents the set of relations in the second scene graph; L represents the consistency score of a single entity or relation; clip(.) represents the clipping function; s is an entity or relation; ∈ represents a hyperparameter; log(∈) represents the minimum value in the clipping function; and log(1-∈) represents the maximum value in the clipping function. Input the second image into the aesthetic score predictor to obtain the image aesthetic evaluation index; Input the first prompt word and the second image into the CLIP model to obtain the second image-text consistency evaluation index.

7. A method for optimizing prompt words in a large text-based graph model based on a scene graph, as described in claim 1 or 6, characterized in that, Reward functions are set based on the first image-text consistency evaluation index, the second image-text consistency evaluation index, and the image aesthetics evaluation index. The training of the prompt word optimization model is driven by the reward function using a reinforcement learning paradigm, including: The expression for the reward function is as follows: r(x,x * )=α1*r AES (I*)+α2*r SG (S * ,I * )+α3*r OV (x,I * ) In the formula, x represents the first prompt word, x * This indicates the second cue word, α1, α2, and α3 represent weight coefficients, and r AES (I * ) represents the image aesthetic evaluation index, r SG (S * I * ) represents the first consistency evaluation index between text and graphics, r OV (x, I) * () indicates the second indicator for evaluating the consistency between text and graphics; The optimized reward function is expressed as follows: In the formula, D represents the training dataset, Φ represents the weights of the prompt word optimization model in the current training round, and Φ sft KL(·) represents the weights of the untrained prompt word optimization model, KL(·) represents the relative entropy penalty, H(·) represents the entropy reward, and α and β represent the weight coefficients.

8. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the text-based large model prompt word optimization method based on scene graph as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the text-based large model prompt word optimization method based on scene graph as described in any one of claims 1-7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the text-based large model prompt word optimization method based on scene graph as described in any of claims 1-7.

Images