Training-free text graph semantic binding method and system based on token merging

Abstract

The invention belongs to the technical field of text graphs, and provides a training-free text graph semantic binding method and system based on token merging, and the technical scheme is that the method comprises the steps: carrying out the marking processing of an input prompt text, and coding the text to obtain text embedding; analyzing the text embedding to obtain the coupling property and semantic additive property of the text embedding; constructing a composite token based on the coupling of text embedding and semantic additive, and meanwhile, replacing an original ending token with an ending token which does not contain attribute information; iteratively updating the composite token based on semantic binding loss and entropy loss, and obtaining final text embedding based on the updated composite token and the replaced ending token; and generating a corresponding image based on the final text embedding. The method disclosed by the invention is excellent in performance in a complex scene involving a plurality of objects and attributes, which is difficult to process by an existing method.

Description

Technical Field

[0001] The present invention belongs to the technical field of text graphs, and in particular to a training-free text graph semantic binding method and system based on token merging. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] The emergence of diffusion models has led to rapid development in the field of text-to-image generation. These models are able to generate high-quality images using text prompts as control conditions, but aligning the generated images with the text prompts remains a significant challenge; one of the most common problems in existing text-to-image (T2I) is the lack of proper semantic binding, that is, a given object is not properly bound to its attributes or related objects.

[0004] In order to address the challenge of aligning the T2I diffusion model with the complex semantics in the textual cues, a variety of enhancement strategies have been proposed, including optimizing the latent representation, guiding the generation process through layout priors, or fine-tuning the T2I model. Despite some progress, these methods still have limitations in generating high-quality images of complex scenes, such as multiple objects and multiple attributes.

[0005] Optimization-based methods, such as strengthening attention maps by adjusting text embeddings or optimizing noise signals, are mainly inspired by text-based image editing methods, showing that the layout of objects is determined by self-attention and cross-attention maps in the T2I diffusion model UNet. However, these methods often fail when dealing with complex scenes that generate multiple objects or multiple attributes.

[0006] Layout-generated image methods widely use layouts, especially in the form of bounding boxes or segmentation maps, as a popular intermediate means to bridge the gap between text input and generated images; although these methods have certain potential, they significantly prolong the training time. More importantly, the application of layout priors becomes challenging when global context descriptions or abstract elements are involved. This limitation constrains the generality of these techniques and makes them difficult to effectively deploy in real-world scenarios that require non-specific spatial layouts.

[0007] The large language model-based method first generates a layout using a large language model (LLMs), and then forces the text-to-image (T2I) generation process to follow these guidelines, similar to the layout-guided method; the fine-tuning-based method enhances the semantic alignment capability by updating the model parameters on massive datasets. There are also some improved T2I models trained from scratch on large-scale datasets. These methods implicitly improve the semantic alignment capability through better architecture design and a large amount of training data. However, these methods require extremely large computing resources to achieve their goals. Summary of the invention

[0008] In order to solve at least one of the technical problems in the above-mentioned background technology, the present invention provides a training-free text-graph semantic binding method and system based on token merging, which solves the semantic binding problem in a training-free manner, does not require the use of a large language model, and does not require training on additional data sets. In addition, it performs better when dealing with complex T2I generation scenarios, especially when users require binding of multiple objects or multiple attributes of an object, the effect is particularly significant.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] The first aspect of the present invention provides a training-free text-graph semantic binding method based on token merging, comprising the following steps:

[0011] Tokenizing the input prompt text and encoding it to obtain text embedding; wherein tokenizing the input prompt text includes filling a start token and a plurality of end tokens at the beginning and the end of the text respectively;

[0012] Analyze text embedding to obtain the coupling and semantic additivity of text embedding;

[0013] A composite token is constructed based on the coupling and semantic additivity of text embeddings, and the original end token is replaced by an end token that does not contain attribute information.

[0014] Iteratively update the composite token based on semantic binding loss and entropy loss, and obtain the final text embedding based on the updated composite token and the replaced end token;

[0015] Generate the corresponding image based on the final text embedding.

[0016] Furthermore, based on the semantic additivity of token embedding, entities and attributes are expressed together by explicitly binding tokens, and the semantic fusion of tokens is performed by element-by-element addition to obtain a composite token.

[0017] Furthermore, the prompt text contains K entities represented by nouns and their corresponding tokens. For a prompt word containing K entities, the calculation formula for the composite token is:

[0018]

[0019] in, Represents a composite token, where each entity is associated with a token of a related object or attribute, represented as (n k ,a k ).

[0020] Furthermore, the calculation formula of semantic binding loss is:

[0021]

[0022] Where t is the time step, ∈ θ represents the diffusion model, z t represents the latent variable, represents a composite token, is text embedding, and K represents the number of entities.

[0023] Furthermore, the calculation formula of entropy regularization loss is:

[0024]

[0025] Among them, A k represents the cross attention map corresponding to the kth token, Represents the entropy of each token.

[0026] Furthermore, the CLIP text encoder is used to encode the input text prompt to obtain text embeddings, and each embedding corresponds to a token.

[0027] Furthermore, the SDXL model and text embedding are used to generate corresponding images, wherein the SDXL model includes an autoencoder and a diffusion model.

[0028] A second aspect of the present invention provides a system for generating images from training-free text based on token merging, comprising:

[0029] A text embedding extraction module is used to tokenize the input prompt text and encode it to obtain a text embedding; wherein the tokenization of the input prompt text includes filling a start token and a plurality of end tokens at the beginning and the end of the text respectively;

[0030] Text embedding analysis module, which is used to analyze text embedding and obtain the coupling and semantic additivity of text embedding;

[0031] The text-graph semantic binding module is used to construct a composite token based on the coupling and semantic additivity of text embeddings, and replace the original end token with an end token that does not contain attribute information; it iteratively updates the composite token based on semantic binding loss and entropy loss, and obtains the final text embedding based on the updated composite token and the replaced end token; and generates the corresponding image based on the final text embedding.

[0032] A third aspect of the present invention provides a computer-readable storage medium.

[0033] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the above-mentioned method for generating images from training-free text based on token merging.

[0034] A fourth aspect of the present invention provides a computer device.

[0035] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the method for generating images from training-free text based on token merging as described above are implemented.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. The present invention proposes a training-free method, called token merging, as a more efficient and robust method to solve the semantic binding problem. The effect of this method is further improved by the proposed end token replacement and iterative composite token update techniques.

[0038] 2. The present invention analyzes the semantic binding problem. Based on the analysis of text embedding, the present invention obtains the coupling and semantic additivity of text embedding, and obtains a diffusion model that can explain the composite token generated by adding multiple tokens and integrate the semantic properties of the combined tokens.

[0039] 3. This paper defines semantic binding as the task of associating an object with its attributes (called attribute binding) or associating it with other related sub-objects (called object binding), and proposes a new method called token merging. Semantic binding is enhanced by aggregating related tokens into a composite token. This method ensures that objects, their attributes and sub-objects share the same cross-attention graph.

[0040] 4. In order to further optimize T2I, the present invention also introduces two auxiliary losses: entropy loss and semantic binding loss. These losses iteratively update the composite token to improve the completeness of the generation.

[0041] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0043] Figure 1 It is a flow chart of a training-free text-graph semantic binding method based on token merging provided by an embodiment of the present invention;

[0044] Figure 2 are multiple images generated based on different input prompt words provided by an embodiment of the present invention, where (a) uses a single token to embed [dog] and an end token [EOT]; (b) is the probability that the generated image contains "sunglasses";

[0045] Figure 3 is the token additivity result provided by the embodiment of the present invention;

[0046] Figure 4 The ToMe provided by the embodiment of the present invention consists of two parts, wherein (a) is token merging and end token replacement, and (b) is iterative composite token update through two auxiliary losses;

[0047] Figure 5 It is a qualitative comparison of different text-to-image (T2I) methods provided by the embodiments of the present invention when processing complex prompt words;

[0048] Figure 6 It is a display of image generation results based on texts with different configurations provided by an embodiment of the present invention;

[0049] Figure 7 It is a visualization result display of the cross attention graph under different configurations provided by the embodiments of the present invention;

[0050] Figure 8 This is an additional application of semantic additivity in text embedding provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0052] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0053] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0054] Text-to-image (T2I) models have excellent generation capabilities, but they often fail to accurately combine the semantic objects in the input prompt words with their attributes, a problem known as semantic binding.

[0055] Previous approaches require extensive fine-tuning of the entire T2I model, or require users or large language models to specify the generation layout, which increases complexity. We define semantic binding as the task of associating an object with its attributes (called attribute binding) or associating it with other related sub-objects (called object binding). We propose a new method, called TokenMerging (ToMe), to enhance semantic binding by aggregating related tokens into a composite token. This method ensures that objects, their attributes, and sub-objects share the same cross-attention map.

[0056] In addition, in order to deal with potential confusion between main objects in complex text prompts, the present invention proposes a complementary strategy called end token replacement. In order to further optimize the initial stage of T2I generation (i.e., the layout determination stage), two auxiliary losses are introduced: entropy loss and semantic binding loss. These losses iteratively update the composite tokens to improve the completeness of the generation. The present invention conducted a large number of experiments to verify the effectiveness of ToMe, and compared it with various existing methods on the T2I-CompBench and the GPT-4o object binding benchmark proposed by the present invention. The method of the present invention performs better in complex scenarios involving multiple objects and attributes that are difficult for existing methods to handle.

[0057] Embodiment 1

[0058] like Figure 1 As shown, this embodiment provides a method for generating images from training-free text based on token merging, comprising the following steps:

[0059] Step 1: Mark and encode the input text prompt to obtain text embedding;

[0060] For a given text prompt word The CLIP text model is extended to its length M by padding the start token [SOT] and multiple end tokens [EOT] before and after it. After the CLIP text encoder Tx After processing, the condition is formalized as Each row in represents the corresponding token embedding after passing through the CLIP text encoder. For example, the text embedding of sentence P = "acat wearing sunglasses and a dog wearing a hat" is represented as:

[0061] Step 2: Generate initial image based on text embedding;

[0062] In this example, a new method for semantic alignment is constructed based on the standard SDXL model. The model consists of two main parts: an autoencoder (including an encoder ε and a decoder ) and the diffusion model (i.e., the model with parameter θ∈ θ ). Model∈ θ Updated by the following loss function:

[0063]

[0064] Model∈ θ is a UNet with the conditional input as the latent variable z t , Text Embedding And the time step t~Uniform(1,T).

[0065] More specifically, the text-guided diffusion model aims to generate T and condition input prompt words Generate an image. In order to distinguish it from the general LDMs condition, the text condition is expressed as Among them, τ ξ is the CLIP text encoder, the cross attention map is passed Get, set For model ∈ θ The output feature map is passed through the projection network l Q Calculate the query matrix

[0066] Similarly, given a text embedding Through the projection network Calculate the bond matrix

[0067] Next, the attention map is calculated according to the following formula:

[0068] Where d is the latent dimension, unit Defines the weight of the i-th token to the j-th token.

[0069] Step 3: Based on the constructed semantic misalignment adjustment model, the misalignment between the object and its attribute in the initial image is adjusted to obtain an adjusted image; specifically, the steps include:

[0070] To address the semantic binding problem, we focus on the text embeddings used in the generation process of the diffusion model, as they primarily determine the content of the generated images.

[0071] Still taking the text prompt given in step 1 as an example, the text embedding is analyzed, including:

[0072] Step 301: Analyze text embedding to obtain coupling and semantic additivity of text embedding;

[0073] The coupling analysis process includes:

[0074] In generating text-based embeddings After the initial image is obtained, there is a semantic misalignment problem between the objects in the initial image and their attributes. Figure 2 As shown in the first two columns at the bottom of (a), it is observed that there is a misalignment between the attributes of dogs and cats. Subsequently, token embeddings are extracted from the text embeddings And input it into the UNet model ∈ θ (Right now like Figure 2 As shown in the middle column of (a), dog objects often wear glasses, further highlighting the semantic leakage problem. When used as input, the generated image is similar to the entire text embedding The images obtained are very similar. Since [EOT] interacts with all tokens, it usually contains the entire semantic information.

[0075] The DetScore report further generates the probability of the corresponding object (such as "sunglasses") in the 100 images, as shown in Figure 2 As shown in (b), for these three different cases, the DetScore is 22.6%, 69.6% and 75.0%, respectively. These results are consistent with the above observations.

[0076] The analysis process of semantic additivity includes:

[0077] Inspired by the semantic additivity of text embeddings, we experimented with the additivity property of CLIP text embeddings. The text embedding corresponding to the prompt word "a photo of a dog" is represented as The text embedding representation of the prompt word "a photo of a hat" is

[0078] Next, the object token (i.e. ) and the corresponding [EOT] tokens are added element-wise. Specifically, the generated new embedding is represented as

[0079] Then embed the text In the input diffusion model UNet, it is observed that this additivity property can add objects, remove objects, and even perform complex semantic calculations. To explore the mechanism behind this phenomenon, the PCA dimensionality reduction visualization is performed on the token representation of each prompt word, as shown in the figure below: Figure 3 As shown in Figure 2, the direction vector obtained from “queenking” is almost identical to the direction vector of “woman-man”, with a cosine similarity of 0.998.

[0080] Summary: According to the coupling and additivity analysis, the semantic contents of text tokens are coupled and intertwined, leading to attribute confusion between different subjects. In addition, in the diffusion model, text embeddings show semantic additivity. This shows that the diffusion model is able to explain the composite tokens generated by the addition of multiple tokens and integrate the semantic properties of the combined tokens.

[0081] Step 302: construct a composite token based on the coupling and semantic additivity of text embedding, and replace the original end token with an end token that does not contain attribute information;

[0082] Semantic binding in the text-to-image (T2I) process refers to accurately establishing the association between an object and its related attributes or sub-objects. This process can avoid semantic misalignment in the generated image and ensure that each visual element accurately corresponds to the description in the text. Figure 4 A schematic diagram of ToMe is shown;

[0083] Inspired by the semantic additivity of object token combination token embedding, the entity and attribute are expressed together by explicitly binding tokens, and the semantic fusion of tokens is performed by element-by-element addition.

[0084] Assume initial prompt word Contains K entities represented by nouns and their corresponding tokens, denoted as {n 1 ,…,n k …,n K Each entity is associated with a token for the relevant object or attribute, represented as (n k ,a k ). For example, in the sentence “a cat wearing sunglasses and a dog wearing a hat”, n 1 = <cat> ,a 1 ＝{ <wearing> , <glasses>},n 2 ＝ <dog> ,a 2 ＝{ <with>, , <hat>}.

[0085] For a prompt word containing K entities For each entity-attribute pair (n k ,a k ) is merged into We call this compound token. This innovative approach compresses long sequences of cue words using a single compound token, generating a unified cross-attention map that avoids semantic misalignment. We further demonstrate these observations in ablation experiments.

[0086] At the same time, since the semantic information contained in [EOT] may interfere with the expression of attributes, in this embodiment, [EOT] is replaced to reduce such interference and retain the semantic information of each subject. For example, when the prompt word is "a cat wearing hat and a dog wearing sunglasses", the original [EOT] is replaced with [EOT] from the prompt word "a cat and a dog".

[0087] As< / hat> Figure 4 As shown in (a), the final text embedding after subject-attribute enhancement and EOT replacement is: Among them, dog* and EOT* represent the tokens after token merging and end token replacement respectively.

[0088] Step 303: Iteratively update the composite token using the semantic binding loss and the entropy regularization loss to obtain an updated composite token.

[0089] As described in step 301, the semantic information embedded in each token is intrinsically related. After strengthening the relationship between the subject and its attributes, it becomes crucial to eliminate any irrelevant semantic information in the composite token to prevent the misrepresentation of the attributes. Figure 4 As shown in (b), in order to ensure that the semantics of the composite tokens accurately correspond to the noun phrases they represent, in this embodiment, a clean prompt word is used as a supervisory signal.

[0090] Specifically, for composite token embedding It corresponds to the noun phrase "a dog wearing hat", and the goal in this example is to keep the diffusion model's noise predictions for this compound token consistent with the full phrase. Mathematically, this goal can be expressed as ensuring

[0091] This actually aligns At time step t, a semantic binding loss is used to align the semantics of the tokens, which is defined as:

[0092]

[0093] Next, the information carried by each token embedding is calculated through entropy statistics.

[0094] like Figure 7 As shown, the cross-attention map corresponding to the kth token is extracted Normalize the cross attention map to After that, the entropy of each token is calculated as:

[0095] Reducing the entropy of the cross-attention map helps ensure that the tokens focus on their designated regions, thus preventing the cross-attention map from being too distracting. Figure 7 As shown, we observe instances of attribute confusion where different tokens incorrectly affect the same image region. At time step t, the entropy regularized loss is defined as:

[0096]

[0097] Finally, the comprehensive loss is: These two new losses are <T opt The period is used to update the composite token, where λ is a balancing hyperparameter.

[0098] Experimental setup

[0099] Evaluation Benchmarks and Metrics The effectiveness of ToMe is evaluated using T2I-CompBench, a comprehensive open-world compositional text-to-image (T2I) benchmark covering attribute binding and object relations. By focusing on the semantic binding problem, T2I-CompBench is evaluated mainly through three attribute subsets, namely color, shape, and texture. Following the evaluation protocol, 300 validation cue words from each subset are used for evaluation, and the BLIP-VQA score is used as the evaluation metric. Then, the ImageReward model is used to evaluate the human preference score, which comprehensively measures the image quality and cue word alignment.

[0100] In order to comprehensively evaluate the object binding performance, a new GPT-4o Benchmark is introduced, which uses 50 prompt words for testing and the template is "a[objectA]with a[itemA]and a[objectB]with a[itemB].". For example, objectA and objectB can be "cat" and "dog", while itemA and itemB are related items such as "hat" and "glasses". Afterwards, the multimodal model GPT-4o is used to calculate the consistency score between the generated image and the prompt word for objective evaluation.

[0101] Implementation details SDXL is used as the base model. For automated image generation evaluation, SpaCy is used to parse the prompt words and identify each object and its corresponding attributes for token merging. Iterative updates of composite tokens are performed in the first 20% of the denoising step, with Topt=0.2T and hyperparameter λ=0.2.

[0102] Comparison Methods To evaluate the effectiveness of our method, we compared it with the most advanced methods. These methods mainly include: (1) the most advanced T2I diffusion models, such as SDXL and Playground-v2; (2) fine-tuning-based methods, such as CoMat and ELLA; (3) optimization-based methods SynGen; (4) Ranni, a fine-tuning method that enhances large language models (LLMs).

[0103] Experimental Results

[0104] Quantitative comparison As shown in Table 1, ToMe consistently outperforms or is comparable to existing methods in BLIPVQA scores in the color, texture, and shape attribute binding subsets, indicating its effectiveness in avoiding attribute confusion. Human preference scores evaluated by the ImageReward model (note that the model scores are logits and may be negative) show that images generated by ToMe are better aligned with the prompt words. In particular, although ELLA uses LLama or T5-XL to replace the CLIP text encoder to enhance text embedding, the method of the present invention still outperforms ELLA in BLIP-VQA scores. The significant improvement in GPT-4o scores also demonstrates the effectiveness of ToMe in textit object binding.

[0105] Qualitative comparison According to the SynGen method, the failure of attribute binding is divided into three categories: (i) semantic leakage in the cue word, the attribute ak is not matched with its corresponding entity nk; (ii) semantic leakage outside the cue word, the attribute ak describes the background or the entity not mentioned in the cue word P; (iii) attribute neglect, the attribute ak is completely ignored during the image generation process. Figure 5 The qualitative comparison results of the present invention and other methods are shown. The first three rows show more complex object binding results, and the last two rows show attribute binding results. The semantic binding errors in the images generated by SDXL are mainly attributed to (i) semantic leakage in the prompt words, as shown in the first and second rows. Playground-v2 encounters similar semantic binding problems as SDXL. ELLA occasionally succeeds in handling simple attribute binding (as shown in the fifth row), but frequently encounters (i) semantic leakage in the prompt words and (iii) attribute neglect errors when handling the first three prompt words. Ranni generates images based on layouts generated by large language models, which can partially solve the problem in some complex object binding situations (such as the second row). However, layout-driven methods may be limited in achieving correct image layouts, such as the performance when handling complex descriptions in the first row. SynGen focuses on the attribute binding problem and achieves good results in color and shape binding, but performs poorly in object binding, with varying degrees of (i) and (iii) failures. Compared with these methods, ToMe of the present invention performs well in object and attribute binding scenarios, which is consistent with the quantitative indicators reflected in Table 1.

[0106] Table 1 shows the quantitative results of semantic binding evaluation on different benchmark subsets.

[0107]

[0108] The quantitative results of each component of the ablation study are shown in Table 2.

[0109] Table 2 shows the results of ablation studies on the T2I-CompBench benchmark

[0110]

[0111] It can be observed that using configuration B containing only token merging techniques (i.e., ToMe and ETS) brings a slight performance improvement, which is consistent with Figure 6 However, token merging is fundamental to subsequent optimizations. When they are combined with the entropy loss Lent (configuration C), performance improves significantly. This is presumably partly due to the more regularized cross-attention maps, e.g. Figure 7 However, configuration C still leads to poor generation performance without semantic binding loss, as shown in Figure 6 As shown, the dog on the right still exhibits cat-like characteristics. Adding the semantic alignment loss Lsem (as the default configuration for this example) ensures that the two subjects are correctly bound to their respective attributes without appearance confusion, achieving the best results both quantitatively and qualitatively. If token merging is ignored and only optimization is applied (configuration D and configuration E), the performance is only comparable to the baseline. Removing Lent (configuration F) can also improve the baseline, but there will be obvious artifacts in the generated results, which is mainly due to the lack of sufficient regularization in the cross-attention map. In summary, each element of these three new techniques in ToMe contributes to achieving state-of-the-art performance.

[0112] Additional applications such as Figure 8 As shown, ToMe not only successfully solves the semantic binding problem, but can also be applied to other widely existing problems in text-to-image (T2I), including adding objects, removing objects, and even for tasks such as bias mitigation.

[0113] In this paper, a key problem in text-to-image (T2I) models, namely semantic binding, is studied. This phenomenon refers to the difficulty encountered by T2I models in accurately understanding and correctly mapping relevant semantics to images. It is found that previous methods often require extensive fine-tuning of the entire T2I model or explicitly specifying the generation layout through a large language model. To address these problems, this paper proposes a new training-free method called Token Merging, or ToMe, to solve the semantic binding problem in T2I generation. ToMe introduces an innovative mechanism by stacking object tokens with their related tokens into a composite token. This mechanism eliminates semantic misalignment by unifying the cross-attention map. In addition, end token replacement and iterative composite token update techniques are combined to further enhance semantic binding. In extensive experiments, ToMe is compared with various existing methods, and the evaluation benchmarks include T2I-CompBench and the proposed GPT-4o benchmark. The results show that compared with existing methods, ToMe performs better in handling complex and demanding generation tasks, especially in the object binding problem that has been neglected in previous studies, and has achieved significant improvements.

[0114] Embodiment 2

[0115] This embodiment provides a training-free text-graph semantic binding system based on token merging, including:

[0116] A text embedding extraction module is used to tokenize the input prompt text and encode it to obtain a text embedding; wherein the tokenization of the input prompt text includes filling a start token and a plurality of end tokens at the beginning and the end of the text respectively;

[0117] Text embedding analysis module, which is used to analyze text embedding and obtain the coupling and semantic additivity of text embedding;

[0118] The text-graph semantic binding module is used to construct a composite token based on the coupling and semantic additivity of text embeddings, and replace the original end token with an end token that does not contain attribute information; it iteratively updates the composite token based on semantic binding loss and entropy loss, and obtains the final text embedding based on the updated composite token and the replaced end token; and generates the corresponding image based on the final text embedding.

[0119] Embodiment 3

[0120] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the training-free text-graph semantic binding method based on token merging are implemented as described above.

[0121] Embodiment 4

[0122] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the training-free text-graph semantic binding method based on token merging as described above are implemented.

[0123] Embodiment 5

[0124] This embodiment provides a program product, which is a computer program product, including a computer program. When the computer program is executed by a processor, the steps in the training-free text-graph semantic binding method based on token merging as described above are implemented.

[0125] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0126] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0127] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0128] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0129] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0130] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.< / with> < / dog> < / glasses> < / wearing> < / cat>

Claims

1. A training-free text-graph semantic binding method based on token merging, characterized in that: The steps include: Tokenizing the input prompt text and encoding it to obtain text embedding; wherein tokenizing the input prompt text includes filling a start token and a plurality of end tokens at the beginning and the end of the text respectively; Analyze text embedding to obtain the coupling and semantic additivity of text embedding; A composite token is constructed based on the coupling and semantic additivity of text embeddings, and the original end token is replaced by an end token that does not contain attribute information. Iteratively update the composite token based on semantic binding loss and entropy loss, and obtain the final text embedding based on the updated composite token and the replaced end token; Generate the corresponding image based on the final text embedding.

2. The token-merging-based training-free text-graph semantic binding method according to claim 1, characterized in that: Based on the semantic additivity of token embedding, entities and attributes are expressed together by explicitly binding tokens, and the semantic fusion of tokens is performed by element-by-element addition to obtain a composite token.

3. The token-merging-based training-free text-graph semantic binding method according to claim 1, characterized in that: The prompt text contains K entities represented by nouns and their corresponding tokens. For a prompt word containing K entities, the calculation formula for the composite token is: in, Represents a composite token, where each entity is associated with a token of a related object or attribute, represented as (n k ,a k ).

4. The token-merging-based training-free text-graph semantic binding method according to claim 1, characterized in that: The calculation formula of semantic binding loss is: Where t is the time step, ∈ θ represents the diffusion model, z t represents the latent variable, represents a composite token, is text embedding, and K represents the number of entities.

5. The token-merging-based training-free text-graph semantic binding method according to claim 1, characterized in that: The formula for calculating the entropy regularization loss is: Among them, A k represents the cross attention map corresponding to the kth token, Represents the entropy of each token.

6. The token-merging-based training-free text-graph semantic binding method according to claim 1, characterized in that: The CLIP text encoder is used to encode the input text prompt to obtain text embeddings, each of which corresponds to a token.

7. The token-merging-based training-free text-graph semantic binding method according to claim 1, characterized in that: The SDXL model and text embedding are used to generate corresponding images, where the SDXL model includes an autoencoder and a diffusion model.

8. A training-free text-graph semantic binding system based on token merging, characterized in that: include: A text embedding extraction module is used to tokenize the input prompt text and encode it to obtain a text embedding; wherein the tokenization of the input prompt text includes filling a start token and a plurality of end tokens at the beginning and the end of the text respectively; Text embedding analysis module, which is used to analyze text embedding and obtain the coupling and semantic additivity of text embedding; The text-graph semantic binding module is used to construct a composite token based on the coupling and semantic additivity of text embeddings, and replace the original end token with an end token that does not contain attribute information; it iteratively updates the composite token based on semantic binding loss and entropy loss, and obtains the final text embedding based on the updated composite token and the replaced end token; and generates the corresponding image based on the final text embedding.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the token merging-based training-free text-graph semantic binding method as described in any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the token merging-based training-free text-graph semantic binding method according to any one of claims 1 to 7 are implemented.

Images