Scene graph-based text graph large model cue word optimization method, electronic equipment and medium

Abstract

The invention discloses a text graph large model cue word optimization method based on a scene graph, electronic equipment and a medium, and the method comprises the steps: obtaining a first cue word, and carrying out the analysis to obtain a first scene graph; inputting the first prompt word into the text graph large model to generate a corresponding first picture; performing semantic alignment on the first scene graph and the first picture to obtain a second scene graph; inputting the first cue word and the second scene graph into a pre-trained cue word optimization model, and outputting a second cue word; comprising the following steps: encoding a first cue word to obtain a text feature, encoding a second scene graph, and performing convolution to obtain a semantic feature; the text features and the semantic features are spliced and then input into a text editing decoder, and actions at least including deletion, rearrangement and insertion are output; and processing the first cue word based on the output deletion, rearrangement and insertion actions to obtain a second cue word.

Description

Technical Field

[0001] The present invention belongs to the field of prompt word engineering of generative artificial intelligence, and in particular relates to a scene graph-based text graph large model prompt word optimization method, electronic equipment, and medium. Background Art

[0002] With the rapid development of artificial intelligence, large-scale text-based graph models have become cutting-edge technologies in the fields of digital art and visual content generation. These models, such as Stable Diffusion, DALL-E 3, and Imagen, can automatically convert text descriptions (hereinafter referred to as prompt words) into images and have been widely used in multiple industries such as advertising, entertainment, and education. The performance of these models depends greatly on high-quality text prompt input. Therefore, a large number of automated prompt word optimization technologies have emerged recently. The specific workflow of these technologies is to input low-quality prompt words, automatically convert them into high-quality prompt words, and finally use them in the text-based graph process. Although existing prompt word optimization technologies can improve the aesthetic expression of the final image, they generally have a high risk of semantic deviation. Specifically, semantic deviation occurs in the process of converting low-quality prompt words into high-quality prompt words, resulting in a semantic mismatch between the final generated image and the original low-quality prompt words.

[0003] In the prior art, the prompt words are rewritten and optimized based on the language model to improve the aesthetic expression of the text-generated graph model. However, there are the following defects:

[0004] (1) Semantic representation problem: The cue word optimization model needs to fully perceive and analyze the semantics and generative intent of the original low-quality cue words in order to accurately preserve and express the original semantics in the high-quality cue words. Current methods often directly use low-quality cue words as semantic representation input, lacking reasoning and abstraction of their expression in the image domain, and modeling of detailed semantics and their interrelationships.

[0005] (2) Training objective problem: Modeling the training objective of the optimization model relies on accurately measuring the semantic consistency between low-quality cue words and final images, which is difficult in practice and is ignored or abandoned in most methods.

[0006] (3) Training cost problem: The prompt word optimization model is directly connected to the large text graph model in the workflow. The traditional gradient descent training method has high training cost and suffers from performance loss caused by sampling.

[0007] In the prior art, based on the training paradigm of reinforcement learning, the existing image-text consistency evaluation technology is used in the reward function modeling to measure the semantic consistency of low-quality prompt words and final images. The shortcomings of these image-text similarity evaluation methods include:

[0008] (1) Poor accuracy: Existing image-text consistency assessment methods usually map the image and text as a whole into the same space and calculate their distance. This coarse-grained method easily ignores the alignment of semantic details, especially when dealing with complex scenes or performing combinatorial reasoning.

[0009] (2) Lack of interpretability: The above coarse-grained methods are only based on the processing and output of neural networks, and their results are difficult to interpret and adjust, which limits the potential space for model optimization.

[0010] (3) Narrow effective range: Some evaluation methods attempt fine-grained semantic analysis and modeling, but they often rely on specific datasets and preset semantic categories, which greatly limits their application in open-world scenarios. Summary of the invention

[0011] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a scene graph-based text graph large model prompt word optimization method, electronic device, and medium.

[0012] In a first aspect, an embodiment of the present invention provides a method for optimizing prompt words of a large model of a text graph based on a scene graph, and the method specifically comprises the following steps:

[0013] Obtain a first prompt word, parse the first prompt word to obtain a first scene graph; input the first prompt word into the Wensheng graph model to generate a corresponding first picture; semantically align the first scene graph with the first picture to obtain a second scene graph;

[0014] Inputting the first prompt word and the second scene graph into a pre-trained prompt word optimization model, and outputting the second prompt word; including: encoding the first prompt word to obtain text features, encoding and convolving the second scene graph to obtain semantic features; splicing the text features and the semantic features and inputting them into a text editing decoder, and outputting at least actions including deletion, rearrangement, and insertion; processing the first prompt word based on the output deletion, rearrangement, and insertion actions to obtain the second prompt word;

[0015] The training process of the prompt word optimization model includes:

[0016] Calculating a first image-text consistency evaluation index between the second scene graph and the second image, calculating a second image-text consistency evaluation index between the first prompt word and the second image, and obtaining an image aesthetic evaluation index based on the second image;

[0017] A reward function is set based on the first image-text consistency evaluation index, the second image-text consistency evaluation index, and the image aesthetic evaluation index, and the reward function is used to drive the training of the prompt word optimization model through the reinforcement learning paradigm.

[0018] In a second aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned scene graph-based text graph large model prompt word optimization method.

[0019] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned scene graph-based text graph large model prompt word optimization method.

[0020] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned scene graph-based text graph large model prompt word optimization method.

[0021] The beneficial effects of the present invention are:

[0022] The present invention provides a scene graph-based text-graph large model prompt word optimization method, which sets a reward function based on a first text-graph consistency evaluation index, a second text-graph consistency evaluation index, and a picture aesthetic evaluation index, and uses a reinforcement learning paradigm to drive the training of a prompt word optimization model using the reward function. The first prompt word and the second scene graph are input into a pre-trained prompt word optimization model to obtain a second prompt word. The present invention prevents semantic deviation in the process of converting a low-quality first prompt word into a high-quality second prompt word, thereby avoiding semantic mismatch between a finally generated picture and the original prompt word. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0024] Figure 1 A flow chart of a method for optimizing prompt words of a large model of a text graph based on a scene graph provided by an embodiment of the present invention;

[0025] Figure 2 A schematic diagram of the reasoning and training process of the prompt word optimization model provided in an embodiment of the present invention;

[0026] Figure 3 A schematic diagram of the structure of a prompt word optimization model provided by an embodiment of the present invention;

[0027] Figure 4 A schematic diagram of the decoder structure provided by an embodiment of the present invention;

[0028] Figure 5 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principles of the present invention, several improvements and modifications may be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0030] In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0031] like Figure 1 As shown, the present invention proposes a method for optimizing prompt words of a large model of a text graph based on a scene graph, and the method specifically comprises the following steps:

[0032] Step S1, obtain a first prompt word, parse the first prompt word to obtain a first scene graph; input the first prompt word into the Wensheng graph model to generate a corresponding first image; semantically align the first scene graph with the first image to obtain a second scene graph.

[0033] Specifically, step S1 includes the following sub-steps:

[0034] Step S101, obtaining a first prompt word, parsing the first prompt word to obtain a first scene graph;

[0035] Furthermore, the original first prompt word is recorded as x = [w1, w2, ...], and the first prompt word is parsed by the text parser to obtain the first scene graph, which is recorded as S = {O, R}, where O = {o i} is the entity set in the first scene graph S, R = {(o i ,p i→j ,o j )} is the set of relations in the first scene graph S, o i represents the i-th entity, o j represents the jth entity, p i→j Represents the relationship between the i-th entity and the j-th entity.

[0036] Step S102: input the first prompt word x into the text graph model to generate a corresponding first image I.

[0037] It should be noted that the first image I is the expression of the semantics of the first prompt word x in the image domain.

[0038] Step S103: semantically align the first scene graph S with the first image I to obtain a second scene graph S * .

[0039] In this example, the entity set O in the first scene graph S and the first image I are input into the open word set object detection model GLIP (Grounded Language-Image Pre-training) to determine whether the entity in the entity set O in the first scene graph S exists in the first image I;

[0040] When an entity in the entity set O in the first scene graph S does not exist in the first image I, the entity and its associated relationships are deleted from the first scene graph S;

[0041] When an entity in the entity set O in the first scene graph S exists in the first image I, the relationship set R in the first scene graph S is i ,p i→j ,o j )} corresponding to the head entity o i The area in the first picture I, the tail entity o j The area in the first picture I, the relationship p i→j Input to the image-text question answering model BLIP (Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation) to determine the head entity o i , tail entity o j The relationship between i→j Is it true? If the head entity o i , tail entity o j The relationship between i→j If it does not hold, then the relationship p i→j Delete from the first scene graph S;

[0042] The first scene graph S after deleting all entities and relationships that do not exist in the first image I is used as the second scene graph S * , thereby completing the semantic alignment of the first scene graph S and the first image I.

[0043] Step S2, inputting the first prompt word and the second scene graph into a pre-trained prompt word optimization model, and outputting the second prompt word; including: encoding the first prompt word to obtain text features, encoding the second scene graph and convolving to obtain semantic features; splicing the text features and the semantic features and inputting them into a text editing decoder, outputting at least actions including deletion, rearrangement, and insertion; processing the first prompt word based on the output deletion, rearrangement, and insertion actions to obtain the second prompt word.

[0044] Specifically, Figure 2 As shown, step S2 includes the following sub-steps:

[0045] Step S201: All nodes in the second scene graph are encoded into embedded vectors using a multimodal pre-training neural network CLIP (Contrastive Language-Image Pre-training); the embedded vectors are convolved by a multi-layer graph convolutional network to extract semantic features. The text features and semantic features are concatenated to form multimodal features.

[0046] Step S202, inputting the multimodal features obtained by splicing the text features and the semantic features into a text editing decoder, wherein the text editing decoder includes a first sub-decoder, a second sub-decoder, and a third sub-decoder;

[0047] Among them, Figure 3 As shown, the first sub-decoder takes the multimodal feature as input and takes the deletion action as output; the deletion action is used to determine whether each token in the first prompt word is deleted;

[0048] The second sub-decoder takes the multimodal feature as input and takes a rearrangement action as output; the rearrangement action is used to rearrange the order of the tokens retained in the first prompt word;

[0049] The third sub-decoder takes the multimodal feature as input and takes an insertion action as output; the insertion action is used to insert a new token into the rearranged first prompt word.

[0050] For example, in this example, the decoding layer of BART (Bidirectional and Auto-Regressive Transformers), an example of the Transformer architecture, is used as the basic structure of the decoder. The specific structure is as follows: Figure 4 The decoder includes three submodules: a deletion action decoder, a rearrangement action decoder, and an insertion action decoder, all of which are stacked by the above basic structures, and output the deletion action, rearrangement action, and insertion action for the first prompt word x in sequence.

[0051] It should be noted that, in this example, the deletion action and the rearrangement action are generated non-autoregressively. Specifically, the corresponding actions of the entire first prompt word x will be obtained at one time, but the generation process of the insertion action is autoregressive. Specifically, the tokens that need to be inserted into the first prompt word x will be generated one by one.

[0052] Step S203: Process the first prompt word based on the output deletion, rearrangement, and insertion actions to obtain the second prompt word x * .

[0053] The training process of the prompt word optimization model includes:

[0054] Specifically, in this example, given the prompt word optimization model weight Φ that has not been trained with reinforcement learning sft , the entire editing process is regarded as a Markov decision process, and the training is driven by a multi-objective joint reward function r.

[0055] Step S100, calculating a first image-text consistency evaluation index between the second scene graph and the second image, calculating a second image-text consistency evaluation index between the first prompt word and the second image, and obtaining an image aesthetic evaluation index based on the second image; including:

[0056] Calculate the first image-text consistency evaluation index between the second scene graph and the second image. The expression is as follows:

[0057]

[0058] L(s,I,γ s )=clip(log(P(s|I,γ s ))log(∈),log(1-∈))

[0059] Where γ represents the second scene graph S * With the second picture I * The corresponding relationship between γ o Represents the second scene graph S * The entity o in the second image I * The area in γ o,o′ Represents the second scene graph S * The entity o, entity o' and the relationship p between entity o and entity o' in the second image I * The area in O * Represents the entity set in the second scene graph, R *Represents the set of relations in the second scene graph; L represents the consistency score of a single entity or relation; clip(.) represents the clipping function, s is an entity or relation, ∈ represents a hyperparameter, log(∈) represents the minimum value in the clipping function, and log(1-∈) represents the maximum value in the clipping function; in this example, ∈ is set to 0.2.

[0060] Inputting the second image into an aesthetics score predictor to obtain an aesthetics evaluation index of the image;

[0061] The first prompt word and the second image are input into the CLIP model to obtain the second image-text consistency evaluation index.

[0062] Step S200, a reward function is set based on the first image-text consistency evaluation index, the second image-text consistency evaluation index, and the image aesthetic evaluation index, and the reward function is used to drive the training of the prompt word optimization model through the reinforcement learning paradigm. It includes:

[0063] The expression of the reward function is as follows:

[0064] r(x,x * )=α1*r AES (I * )+α2*r SG (S * ,I * )+α3*r OV (x,I * )

[0065] In the formula, x represents the first prompt word, x * represents the second prompt word, α1, α2, α3 represent weight coefficients, r AES (I * ) represents the image aesthetic evaluation index, r SG (S * ,I * ) represents the first image-text consistency evaluation index, r OV (x,I * ) represents the second image-text consistency evaluation index;

[0066] In this example, the Proximal Policy Optimization (PPO) algorithm is used for reinforcement learning training, and the KL divergence penalty and entropy reward techniques are adopted.

[0067] Optimize the reward function, the expression is as follows:

[0068]

[0069] In the formula, D represents the training dataset (in this example, the Lexica dataset, DiffusionDB dataset, and COCO dataset are used), Φ represents the weight of the prompt word optimization model in the current training round, and Φ sft represents the weight of the untrained prompt word optimization model, KL(·) represents the relative entropy penalty, H(·) represents the entropy reward, and α and β represent the weight coefficients.

[0070] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the scene graph-based large model prompt optimization method as described above. Figure 5 As shown, a hardware structure diagram of any device with data processing capability where the scene graph-based Wensheng graph large model prompt optimization method provided by an embodiment of the present invention is located, except Figure 5 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0071] Correspondingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, the above-mentioned scene graph-based Wensheng graph large model prompt optimization method is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0072] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A method for optimizing prompt words of a large model of a scene graph, characterized in that: The method specifically comprises the following steps: Obtain a first prompt word, parse the first prompt word to obtain a first scene graph; input the first prompt word into the Wensheng graph model to generate a corresponding first image; semantically align the first scene graph with the first image to obtain a second scene graph; Inputting the first prompt word and the second scene graph into a pre-trained prompt word optimization model, and outputting the second prompt word; including: encoding the first prompt word to obtain text features, encoding and convolving the second scene graph to obtain semantic features; splicing the text features and the semantic features and inputting them into a text editing decoder, and outputting at least actions including deletion, rearrangement, and insertion; processing the first prompt word based on the output deletion, rearrangement, and insertion actions to obtain the second prompt word; The training process of the prompt word optimization model includes: Calculating a first image-text consistency evaluation index between the second scene graph and the second image, calculating a second image-text consistency evaluation index between the first prompt word and the second image, and obtaining an image aesthetic evaluation index based on the second image; A reward function is set based on the first image-text consistency evaluation index, the second image-text consistency evaluation index, and the image aesthetic evaluation index, and the reward function is used to drive the training of the prompt word optimization model through the reinforcement learning paradigm.

2. According to the scene graph-based text graph large model prompt word optimization method of claim 1, it is characterized in that: The first scene graph obtained by parsing the first prompt word includes: The first prompt word is parsed to obtain the first scene graph, which is recorded as S = {O, R}, where O = {o i } is the entity set in the first scene graph S, R = {(o i , p i→j , o j )) is the set of relations in the first scene graph S, o i represents the i-th entity, o j represents the jth entity, p i→j Represents the relationship between the i-th entity and the j-th entity.

3. The method for optimizing the prompt words of a large model of a text graph based on a scene graph according to claim 1 or 2, characterized in that: The process of semantically aligning the first scene graph with the first image includes: Input the entity set O in the first scene graph S and the first picture into the open word set object detection model GLIP to determine whether the entity in the entity set O in the first scene graph S exists in the first picture; When an entity in the entity set O in the first scene graph S does not exist in the first picture, then the entity and its associated relationships are deleted from the first scene graph S; When an entity in the entity set O in the first scene graph S exists in the first picture, the relationship set R in the first scene graph S is set to i , p i→j , o j ))The corresponding header entity o i In the first picture, the area in the middle, the tail entity o j In the first picture, the region, the relationship p i→j Input to the BLIP judgment head entity o of the picture and text question answering model i , tail entity o j The relationship between i→j Is it true? If the head entity o i , tail entity o j The relationship between i→j If it does not hold, then the relationship p i→j Delete from the first scene graph S; The first scene graph S after deleting all entities and relationships that do not exist in the first picture, is used as the second scene graph S*, thereby completing the semantic alignment of the first scene graph S and the first picture,.

4. The method for optimizing the prompt words of a large model of a scene graph based on a text graph according to claim 1, characterized in that: The process of encoding and convolving the second scene graph to obtain semantic features includes: Encode all nodes in the second scene graph into embedding vectors; Convolution is performed on the embedded vector to extract semantic features.

5. The method for optimizing the prompt words of a large model of a scene graph based on a text graph according to claim 1, characterized in that: The process of splicing text features and semantic features and inputting them into a text editing decoder and outputting at least actions including deletion, rearrangement, and insertion includes: Inputting the multimodal features obtained by concatenating the text features and the semantic features into a text editing decoder, wherein the text editing decoder includes a first sub-decoder, a second sub-decoder, and a third sub-decoder; The first sub-decoder takes the multimodal feature as input and takes the deletion action as output; the deletion action is used to determine whether each token in the first prompt word is deleted; The second sub-decoder takes the multimodal feature as input and takes a rearrangement action as output; the rearrangement action is used to rearrange the order of the tokens retained in the first prompt word; The third sub-decoder takes the multimodal feature as input and takes an insertion action as output; the insertion action is used to insert a new token into the rearranged first prompt word.

6. The method for optimizing the prompt words of a large model of a scene graph based on a text graph according to claim 1, characterized in that: The process of calculating the first image-text consistency evaluation index between the second scene graph and the second image, calculating the second image-text consistency evaluation index between the first prompt word and the second image, and obtaining the image aesthetic evaluation index based on the second image includes: Calculate the first image-text consistency evaluation index between the second scene graph and the second image. The expression is as follows: L(s,I,γ s )=clip(log(P(s|I,γ s )),log(∈),log(1-∈)) Where γ represents the second scene graph S * With the second picture I * The corresponding relationship between γ o Represents the second scene graph S * The entity o in the second image I * The area in γ o，o′ Represents the second scene graph S * The entity o, entity o' and the relationship p between entity o and entity o' in the second image I * The area in O * Represents the entity set in the second scene graph, R * Represents the set of relations in the second scene graph; L represents the consistency score of a single entity or relation; clip(.) represents the clipping function, s is an entity or relation, ∈ represents a hyperparameter, log(∈) represents the minimum value in the clipping function, and log(1-∈) represents the maximum value in the clipping function; Inputting the second image into the aesthetic score predictor to obtain an aesthetic evaluation index of the image; The first prompt word and the second image are input into the CLIP model to obtain the second image-text consistency evaluation index.

7. The method for optimizing the prompt words of a large model of a text graph based on a scene graph according to claim 1 or 6, characterized in that: A reward function is set based on the first image-text consistency evaluation index, the second image-text consistency evaluation index, and the image aesthetic evaluation index. The reward function is used to drive the training of the prompt word optimization model through the reinforcement learning paradigm, including: The expression of the reward function is as follows: r(x,x * )=α1*r AES (I*)+α2*r SG (S * ,I * )+α3*r OV (x,I * ) In the formula, x represents the first prompt word, x * represents the second prompt word, α1, α2, α3 represent weight coefficients, r AES (I * ) represents the image aesthetic evaluation index, r SG (S * , I * ) represents the first image-text consistency evaluation index, r OV (x,I * ) represents the second image-text consistency evaluation index; Optimize the reward function, the expression is as follows: In the formula, D represents the training data set, Φ represents the weight of the prompt word optimization model in the current training round, and Φ sft represents the weight of the untrained prompt word optimization model, KL(·) represents the relative entropy penalty, H(·) represents the entropy reward, and α and β represent the weight coefficients.

8. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the scene graph-based text graph large model prompt word optimization method described in any one of claims 1-7 above.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for optimizing prompt words of a large model of a text graph based on a scene graph is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for optimizing the prompt words of the large model of the text graph based on the scene graph described in any one of claims 1-7 is implemented.

Images