Token merging based training-free text-to-graph semantic binding method and system

Abstract

The application belongs to the technical field of text-to-image, and provides a token merging-based training-free text-to-image semantic binding method and system, which has the technical scheme that: the input prompt text is tokenized and encoded to obtain text embedding; the text embedding is analyzed to obtain the coupling and semantic additivity of the text embedding; based on the coupling and semantic additivity of the text embedding, a composite token is constructed, and the original end token is replaced with an end token that does not contain attribute information; the composite token is iteratively updated based on semantic binding loss and entropy loss, and the final text embedding is obtained based on the updated composite token and the replaced end token; and the corresponding image is generated based on the final text embedding. The method performs very well in complex scenes involving multiple objects and attributes which are difficult to handle by existing methods.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of text-to-image, and particularly relates to a token merging based training-free text-to-image semantic binding method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] The emergence of diffusion models has rapidly developed the field of text-to-image, and these models can generate high-quality images using text prompts as control conditions, but aligning the generated images with the text prompts is still a significant challenge; one of the most common problems in existing text-to-image (T2I) is the lack of proper semantic binding, i.e., given objects fail to correctly bind with their attributes or related objects.

[0004] In order to solve the problem of aligning T2I diffusion models with complex semantics in text prompts, various enhancement strategies are proposed, including optimizing latent representations, guiding the generation process through layout priors, or fine-tuning T2I models, although some progress has been made, but these methods still have limitations in generating high-quality images of complex scenes, such as multiple objects and multiple attributes.

[0005] Based on optimization methods, such as adjusting text embeddings or optimizing noise signals to strengthen attention maps, these methods are mainly inspired by text-based image editing methods, which show that the layout of objects is determined by self-attention and cross-attention maps in the UNet of T2I diffusion models, however, these methods often fail when dealing with complex scenes that generate multiple objects or multiple attributes.

[0006] Layout generation image methods widely use layouts, especially in the form of bounding boxes or segmentation maps, as a popular intermediate means to bridge the gap between text input and generated images; although these methods have certain potential, they significantly prolong the training time. More importantly, when it comes to global background descriptions or abstract elements, the application of layout priors becomes challenging. This limitation constrains the universality of these technologies, making them difficult to effectively deploy in handling real-world scenarios that require non-specific spatial layouts.

[0007] Based on the large language model method, first use large language model (Large Language Model, LLM) to generate layout, and then force the text to generate image (T2I) generation process to follow these guidelines, similar to the layout guide method; based on the fine-tuning method, by updating the model parameters on a large amount of data set, the semantic alignment ability is enhanced. There are also some improved T2I models that train from scratch on large-scale data sets, and these methods implicitly improve the semantic alignment ability through better architecture design and a large amount of training data. However, these methods require extremely large computing resources to achieve the goal. SUMMARY

[0008] In order to solve at least one technical problem in the above background art, the present application provides a token merging based training-free text-to-image semantic binding method and system, which solves the semantic binding problem in a training-free manner, neither using large language models nor training on additional data sets. In addition, it performs better in complex T2I generation scenarios, especially when users require multiple object or multiple attribute binding of one object, the effect is particularly significant.

[0009] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0010] The first aspect of the present application provides a token merging based training-free text-to-image semantic binding method, comprising the following steps:

[0011] The input prompt text is tokenized and encoded to obtain a text embedding; wherein the tokenization of the input prompt text includes padding with a start token at the beginning of the text and a plurality of end tokens at the end of the text;

[0012] The text embedding is analyzed to obtain the coupling and semantic additivity of the text embedding;

[0013] Based on the coupling and semantic additivity of the text embedding, a composite token is constructed, and the original end token is replaced with an end token that does not contain attribute information;

[0014] The composite token is iteratively updated based on the semantic binding loss and the entropy loss, and the final text embedding is obtained based on the updated composite token and the replaced end token;

[0015] Based on the final text embedding, a corresponding image is generated.

[0016] Further, based on the semantic additivity of the token embedding, the entity and the attribute are jointly expressed by explicitly binding the token, and the semantic fusion of the token is performed in a element-by-element addition manner to obtain a composite token.

[0017] Further, the prompt text contains K entities represented by nouns and corresponding tokens, and the calculation formula of the composite token for the prompt text containing K entities is:

[0018]

[0019] wherein, represents the composite token, each entity is associated with the token of the related object or attribute, and is represented as (n k ,a k ).

[0020] Further, the calculation formula of the semantic binding loss is:

[0021]

[0022] wherein, t is a time step, and ∈ θ represents the diffusion model, z t represents a latent variable, represents the composite token, is a text embedding, and K represents the number of entities.

[0023] Further, the calculation formula of the entropy regularization loss is:

[0024]

[0025] wherein, A k represents the cross-attention graph corresponding to the kth token, represents the entropy of each token.

[0026] Further, the input text prompt is encoded by using a CLIP text encoder to obtain a text embedding, and each embedding corresponds to a token.

[0027] Further, an SDXL model and a text embedding are used to generate a corresponding image, wherein the SDXL model includes an autoencoder and a diffusion model.

[0028] The second aspect of the present application provides a system for training-free text-to-image generation based on token merging, comprising:

[0029] A text embedding extraction module is configured to tokenize and encode the input prompt text to obtain a text embedding, wherein the tokenization of the input prompt text includes padding a start token and a plurality of end tokens at the beginning and end of the text, respectively.

[0030] A text embedding analysis module is configured to analyze the text embedding to obtain the coupling and semantic additivity of the text embedding.

[0031] A text-to-image semantic binding module is used to construct a composite token based on the coupling and semantic additivity of the text embedding, and to replace the original end token with an end token that does not contain attribute information; the composite token is iteratively updated based on the semantic binding loss and the entropy loss, and the final text embedding is obtained based on the updated composite token and the replaced end token; and the corresponding image is generated based on the final text embedding.

[0032] The third aspect of the present application provides a computer readable storage medium.

[0033] A computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method for training-free text-to-image generation based on token merging.

[0034] The fourth aspect of the present application provides a computer device.

[0035] A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the above-described method for training-free text-to-image generation based on token merging when executing the program.

[0036] Compared with the prior art, the present application has the following advantages:

[0037] 1. The present application proposes a training-free method called token merging as a more efficient and robust method to solve the semantic binding problem, which further improves the effectiveness of the method through the proposed end token replacement and iterative composite token update technology.

[0038] 2. The present application analyzes the semantic binding problem, and based on the analysis of the text embedding, the coupling and semantic additivity of the text embedding are obtained, and the diffusion model can explain the composite token generated by adding multiple tokens, and the semantic attributes of the combined tokens are integrated.

[0039] 3. The present application defines semantic binding as the task of associating a certain object with its attributes (called attribute binding) or associating it with other related sub-objects (called object binding), and proposes a new method called token merging. The related tokens are aggregated into a composite token to enhance semantic binding. This method can ensure that the object, its attributes and sub-objects share the same cross-attention graph.

[0040] 4. In order to further optimize T2I, the present application also introduces two auxiliary losses: entropy loss and semantic binding loss. These losses iteratively update the composite token to improve the completeness of the generated token.

[0041] Advantages of the additional aspects of the application will become apparent in the following description. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate an implementation of the application and, together with the description, serve to explain the application.

[0043] Figure 1 is a flow chart of a token merging based training-free text-to-image semantic binding method provided by an embodiment of the application;

[0044] Figure 2 is a plurality of images generated based on different input prompt words provided by an embodiment of the application, wherein (a) is a single token embedding [dog] and an end token [EOT] used; (b) is the probability of generating an image containing "sunglasses";

[0045] Figure 3 is a token additivity result provided by an embodiment of the application;

[0046] Figure 4 is a ToMe provided by an embodiment of the application, which consists of two parts, wherein (a) is token merging and end token replacement, and (b) is iterative compound token update through two auxiliary losses;

[0047] Figure 5 is a qualitative comparison of different text-to-image (T2I) methods when processing complex prompt words provided by an embodiment of the application;

[0048] Figure 6 is a text-to-image result display based on different configurations provided by an embodiment of the application;

[0049] Figure 7 is a cross-attention map visualization result display under different configurations provided by an embodiment of the application;

[0050] Figure 8 is an additional application of semantic additivity in text embedding provided by an embodiment of the application. DETAILED DESCRIPTION

[0051] The application will be further described below with reference to the drawings and embodiments.

[0052] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs.

[0053] It is to be understood that the terms used herein are for the purpose of describing particular embodiments only and are not intended to limit exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0054] Text-to-image (T2I) models have shown remarkable generation capabilities, but they often fail to accurately combine semantic objects in the input prompt with their attributes, a problem referred to as semantic binding.

[0055] Previous methods require extensive fine-tuning of the entire T2I model or rely on user or large language model specifications for the generation layout, increasing complexity. The present application defines semantic binding as the task of associating an object with its attributes (referred to as attribute binding) or with other relevant sub-objects (referred to as object binding). The present application proposes a new method, called TokenMerging (ToMe), which enhances semantic binding by aggregating relevant tokens into a composite token. This method ensures that the object, its attributes, and sub-objects share the same cross-attention map.

[0056] In addition, to address potential confusion between main objects in complex text prompts, the present application proposes a complementary strategy called end token replacement. To further optimize the initial stage of T2I generation (i.e., the layout determination stage), two auxiliary losses are also introduced: entropy loss and semantic binding loss. These losses iteratively update the composite token to improve the completeness of the generation. Extensive experiments were conducted to verify the effectiveness of ToMe, and comparisons were made with various existing methods on T2I-CompBench and the GPT-4o object binding benchmark proposed by the present application. The present application performs better in complex scenarios involving multiple objects and attributes, which existing methods struggle to handle.

[0057] Embodiment One

[0058] As shown in Figure 1 The present embodiment provides a token-merging-based training-free text-to-image method, comprising the following steps:

[0059] Step 1: Tokenize and encode the input text prompt to obtain text embeddings;

[0060] For a given text prompt The CLIP text model is padded with a start token [SOT] and multiple end tokens [EOT] before and after it to expand its length to M. After passing through the CLIP text encoder Tx After processing, the conditional is formalized as Each row in the above table represents the corresponding token embedding after passing through the CLIP text encoder. For example, the text embedding for the sentence P = "a cat wearing sunglasses and a dog wearing a hat" is represented as:

[0061] Step 2: Generate an initial image based on the text embedding;

[0062] In this embodiment, a new method for semantic alignment is constructed based on the standard SDXL model. The model consists of two main parts: an autoencoder (including an encoder ε and a decoder ) and a diffusion model (i.e., a model ∈ θ with parameters θ). The model ∈ θ is updated by the following loss function:

[0063]

[0064] The model ∈ θ is a UNet, with conditional inputs being latent variables z t , text embedding , and a time step t ~ Uniform(1, T).

[0065] More specifically, the text-guided diffusion model aims to generate images from random noise z T and conditional input prompts . To distinguish from general LDMs conditions, the text condition is represented as where τ ξ is the CLIP text encoder, and the cross-attention map is obtained by , where is the feature map output by the model ∈ θ , and the query matrix Q is calculated by a projection network l

[0066] Similarly, given the text embedding , the key matrix is calculated by a projection network

[0067] Next, the attention map is calculated according to the following formula:

[0068] where d is the latent dimension, and the unit defines the weight of the i-th token on the j-th token.

[0069] Step 3: Based on the constructed semantic misalignment adjustment model, adjust the misalignment between objects and their attributes in the initial image to obtain the adjusted image; specifically, this includes the following steps:

[0070] To address the semantic binding problem, we need to focus on the text embeddings used in the diffusion model generation process, as they primarily determine the content of the generated image.

[0071] Taking the text prompt given in step 1 as an example, the text embedding is analyzed, including:

[0072] Step 301: Analyze the text embeddings to obtain the coupling and semantic additivity of the text embeddings;

[0073] The coupling analysis process includes:

[0074] In generating text-based embeddings After generating the initial image, there is a semantic misalignment between the objects in the initial image and their attributes. For example... Figure 2 As shown in the bottom two columns of (a), a misalignment of attributes between dogs and cats is observed. Subsequently, token embeddings are extracted from the text embeddings. Input it into the UNet model ∈ θ (Right now like Figure 2 As shown in the middle column of (a), the dog object frequently wears glasses, further highlighting the semantic leakage problem. Furthermore, when... When used as input, the generated image is compared with the image using the entire text embedding. The resulting images are very similar. Since [EOT] interacts with all tokens, it typically contains the entire semantic information.

[0075] Further, the probability of the corresponding object (e.g., "sunglasses") being present in the 100 images generated using the DetScore report was analyzed. Figure 2 As shown in (b), the DetScores for these three different cases were 22.6%, 69.6%, and 75.0%, respectively. These results are consistent with the observations above.

[0076] The analysis process for semantic additivity includes:

[0077] Inspired by previous studies on the semantic additivity of text embeddings, we experimented with the additivity properties of CLIP text embeddings. The text embedding corresponding to the prompt "a photo of a dog" is represented as follows: The text embedding of the prompt "a photo of a hat" is represented as follows:

[0078] Next, the object token (i.e. ) and the corresponding [EOT] token are element-wise added. Specifically, the new embedding is generated as

[0079] The text embedding In the input diffusion model UNet, this additive property is observed to add objects, remove objects, and even perform complex semantic calculations. To explore the mechanism behind this phenomenon, PCA dimensionality reduction visualization is performed on the token representation of each prompt word, as shown in Figure 3 The direction vector obtained from "queenking" is almost the same as that of "woman-man", with a cosine similarity of 0.998.

[0080] Summary: According to the coupling and additivity analysis, the semantic content of text tokens is coupled and intertwined, leading to attribute confusion between different subjects. In addition, in the diffusion model, the text embedding exhibits semantic additivity. This indicates that the diffusion model can interpret the composite token generated by the addition of multiple tokens, integrating the semantic attributes of the combined tokens.

[0081] Step 302, based on the coupling and semantic additivity of the text embedding, a composite token is constructed, and the original end token is replaced with an end token that does not contain attribute information;

[0082] Semantic binding in the process of generating images from text (T2I) refers to accurately establishing the association between objects and their related attributes or sub-objects. This process can avoid semantic misplacement in generated images, ensuring that each visual element accurately corresponds to the description in the text. Figure 4 The schematic diagram of ToMe is shown;

[0083] Based on the semantic additivity of object token combined token embedding, the entity and attribute are jointly expressed by explicitly binding tokens, and the element-wise addition method is used for semantic fusion of tokens.

[0084] Assume the initial prompt word contains K entities represented by nouns and their corresponding tokens, denoted as {n 1 ,…,n k …,n K}. Each entity is associated with the token of the related object or attribute, represented as (n k ,a k ). For example, in the sentence "a cat wearing sunglasses and a dog wearing a hat", n 1 = <cat> ,a 1 ＝{ <wearing> , <glasses>},n 2 ＝ <dog> ,a 2 ＝{ <with>, , <hat>}.

[0085] For a prompt containing K entities Each entity-attribute pair (n k ,a k ) is fused into a composite token. This innovative approach compresses the longer prompt sequence by using a single composite token, generates a unified cross-attention map, and thus avoids semantic misalignment. These observations are further demonstrated in ablation experiments.

[0086] At the same time, since the semantic information contained in [EOT] may interfere with the expression of attributes, in this embodiment, the interference is alleviated by replacing [EOT], and the semantic information of each subject is retained. For example, when the prompt is "a cat wearing hat and a dog wearing sunglasses", the original [EOT] is replaced with [EOT] from the prompt "a cat and a dog".

[0087] As< / hat> Figure 4 As shown in (a), the final text embedding after subject-attribute enhancement and EOT replacement is: Wherein, dog* and EOT* represent the tokens after token merging and end token replacement, respectively.

[0088] Step 303: Iteratively update the composite token using semantic binding loss and entropy regularization loss to obtain the updated composite token.

[0089] As described in step 301, the semantic information embedded in each token is intrinsically related. After strengthening the relationship between the subject and its attributes, it becomes crucial to eliminate any irrelevant semantic information in the composite token to prevent misrepresentation of attributes. As... Figure 4 As shown in (b), in order to ensure that the semantics of the composite tokens accurately correspond to the noun phrases they represent, a clean cue word is used as a monitoring signal in this embodiment.

[0090] Specifically, for composite token embedding It corresponds to the noun phrase "a dog wearing a hat." The goal in this embodiment is for the diffusion model to maintain consistent noise predictions for the composite token and the complete phrase. Mathematically, this goal can be expressed as ensuring...

[0091] This is actually aligned At time step t, the semantic binding loss is used to align the semantics of the tokens, which is defined as:

[0092]

[0093] Next, the information carried by each token embedding is calculated using entropy statistics.

[0094] like Figure 7 As shown, extract the cross-attention graph corresponding to the k-th token. Normalize the cross-attention graph to Then, the entropy of each token is calculated as follows:

[0095] Reducing the entropy of the cross-attention graph helps ensure that tokens focus on their designated regions, thus preventing the cross-attention graph from becoming too scattered. For example... Figure 7 As shown, instances of attribute confusion where different tokens incorrectly affect the same image region are observed. At time step t, the entropy regularization loss is defined as:

[0096]

[0097] Ultimately, the total loss is: These two new losses are used to update the composite token at each time step t < T opt where λ is a balancing hyperparameter.

[0098] Experimental setup

[0099] Evaluation benchmarks and metrics T2I-CompBench is used to evaluate the effectiveness of ToMe, which is a comprehensive open-world compositional text-to-image (T2I) benchmark that covers attribute bindings and object relationships. By focusing on the semantic binding problem, T2I-CompBench is mainly evaluated through three attribute subsets, namely color, shape, and texture. Following the evaluation protocol, 300 validation prompt words are used for each subset, and the BLIP-VQA score is used as the evaluation metric. Next, the ImageReward model is used to evaluate the human preference score, which comprehensively measures the image quality and prompt alignment.

[0100] To comprehensively evaluate the object binding performance, a new GPT-4o Benchmark is introduced, which uses 50 prompts for testing, with the template being "a [objectA] with a [itemA] and a [objectB] with a [itemB]." For example, objectA and objectB can be "cat" and "dog," while itemA and itemB are associated items such as "hat" and "glasses." Then, the multimodal model GPT-4o is used to calculate the consistency score between the generated image and the prompt for objective evaluation.

[0101] Implementation details SDXL is used as the base model. To automate image generation evaluation, SpaCy is used to perform syntactic parsing on the prompt, identifying each object and its corresponding attributes for token merging. Iterative updating of the composite token is performed in the first 20% of the denoising step, Topt = 0.2T, and the hyperparameter λ = 0.2.

[0102] Comparative methods To evaluate the effectiveness of the method of the present invention, the present invention is compared with the current most advanced methods. These methods mainly include: (1) the most advanced T2I diffusion model, such as SDXL, Playground-v2; (2) fine-tuning-based methods, such as CoMat, ELLA; (3) optimization-based method SynGen; (4) fine-tuning method Ranni enhanced by large language model (LLM).

[0103] Experimental results

[0104] Quantitative comparisons are shown in Table 1, where ToMe consistently outperforms or is comparable to existing methods in BLIP-VQA scores on color, texture, and shape attribute binding subsets, indicating its effectiveness in avoiding attribute confusions. Human preference scores evaluated by the ImageReward model (note that the model scores are logits, which can be negative) show that images generated by ToMe are better aligned with the prompt words. In particular, despite ELLA using LLama or T5-XL to replace the CLIP text encoder to enhance text embeddings, the method of the present invention still outperforms ELLA in BLIP-VQA scores. The significant improvement in GPT-4o scores also demonstrates the effectiveness of ToMe in textit object binding.

[0105] Qualitative comparisons follow the method of SynGen, which categorizes failures in attribute binding into three categories: (i) semantic leakage in the prompt, where attribute ak does not match its corresponding entity nk; (ii) semantic leakage outside the prompt, where attribute ak describes an entity not mentioned in the prompt P; and (iii) attribute omission, where attribute ak is completely ignored in the image generation process. Figure 5 Qualitative comparison results of the present invention with other methods are shown. The first three rows show more complex object binding results, and the last two rows show attribute binding results. Semantic binding errors in images generated by SDXL are mainly due to (i) semantic leakage in the prompt, as shown in the first and second rows. Playground-v2 encounters similar semantic binding problems as SDXL. ELLA occasionally succeeds in handling simple attribute bindings (as shown in the fifth row), but frequently encounters (i) semantic leakage in the prompt and (iii) attribute omission errors when handling the first three prompts. Ranni generates images based on layouts generated by large language models, which can partially solve problems in some complex object binding cases (such as the second row). However, layout-driven methods can be limited in achieving correct image layouts, for example, in the first row when handling complex descriptions. SynGen focuses on attribute binding problems and achieves good results in color and shape binding, but performs poorly in object binding, with varying degrees of (i) and (iii) failures. Compared to these methods, ToMe of the present invention performs well in both object and attribute binding scenarios, which is consistent with the quantitative indicators reflected in Table 1.

[0106] Table 1 Quantitative results show semantic binding evaluations on different benchmark subsets

[0107]

[0108] Quantitative results of ablation studies for each component are shown in Table 2.

[0109] Table 2 shows the results of ablation studies on the T2I-CompBench benchmark

[0110]

[0111] It can be observed that using only the token merging technique (i.e., configuration B of ToMe and ETS) brings a slight performance improvement, which is consistent with the qualitative results in Figure 6 However, token merging is the basis of subsequent optimizations. When they are used in combination with the entropy loss Lent (configuration C), the performance is significantly improved. It is conjectured that this is due to the cross-attention map being more normalized, as shown in Figure 7 However, configuration C still leads to poor generation performance without the semantic alignment loss Lsem, as shown in Figure 6 The dog on the right still exhibits cat-like features. Adding the semantic alignment loss Lsem (as the default configuration of the present embodiment) can ensure that the two subjects are correctly bound to their respective attributes without appearance confusion, thus achieving the best results both quantitatively and qualitatively. If token merging is ignored and only optimizations are applied (configurations D and E), their performance is only comparable to the baseline. Removing Lent (configuration F) can also improve the baseline, but there will be obvious artifacts in the generated results, mainly due to the lack of sufficient regularization of the cross-attention map. In summary, each element of the three new techniques in ToMe contributes to achieving the most advanced performance.

[0112] As shown in Figure 8 ToMe can not only successfully solve the semantic binding problem, but also be applied to other problems widely existing in text-to-image (T2I) generation, including adding objects, removing objects, and even used for bias mitigation and other tasks.

[0113] In the present application, a key problem in text-to-image (T2I) models, namely semantic binding, is studied. This phenomenon refers to the difficulty that T2I models encounter in accurately understanding and correctly mapping relevant semantics to images. It is found that previous methods often require extensive fine-tuning of the entire T2I model or explicitly specify the generated layout through a large language model. To solve these problems, the present application proposes a new method without training, called Token Merging, namely ToMe, to solve the semantic binding problem in T2I generation. ToMe introduces an innovative mechanism by superimposing object tokens with their related tokens into a composite token. This mechanism eliminates semantic misplacement through a unified cross-attention graph. In addition, end token replacement and iterative composite token updating techniques are combined to further enhance semantic binding. In extensive experiments, ToMe and various existing methods are compared, and the evaluation benchmarks include T2I-CompBench and the GPT-4o benchmark proposed in the present application. The results show that ToMe performs better in handling complex and demanding generation tasks compared to existing methods, and particularly in the object binding problem that has been overlooked in previous studies, significant improvements have been achieved.

[0114] Embodiment Two

[0115] The present embodiment provides a token merging based training-free text-to-image semantic binding system, comprising:

[0116] a text embedding extraction module for tokenizing and encoding the input prompt text to obtain text embedding; wherein the tokenizing of the input prompt text includes padding a start token and multiple end tokens at the beginning and end of the text, respectively;

[0117] a text embedding analysis module for analyzing the text embedding to obtain the coupling and semantic additivity of the text embedding;

[0118] a text-to-image semantic binding module for constructing a composite token based on the coupling and semantic additivity of the text embedding, and replacing the original end token with an end token that does not contain attribute information; iteratively updating the composite token based on semantic binding loss and entropy loss, obtaining the final text embedding based on the updated composite token and the replaced end token; and generating a corresponding image based on the final text embedding.

[0119] Embodiment Three

[0120] The present embodiment provides a computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the steps of the token merging based training-free text-to-image semantic binding method as described above.

[0121] Embodiment Four

[0122] The embodiment provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the training-free text-to-graph semantic binding method based on token merging when executing the program.

[0123] Embodiment five

[0124] The embodiment provides a program product, which is a computer program product, comprising a computer program, and the computer program implements the steps in the training-free text-to-graph semantic binding method based on token merging when executed by a processor.

[0125] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage, etc.) containing computer-usable program code.

[0126] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device implemented in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in the flow or flows and / or block or blocks.

[0127] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in the flow or flows and / or block or blocks.

[0128] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the flowcharts and / or block diagrams. Figure 1 one or more processes and / or blocks Figure 1 steps of the functions specified in the one or more blocks.

[0129] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The program can be stored in a computer readable storage medium, and when the program is executed, the processes of the above-mentioned embodiment methods can be included. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), a random access memory (RAM), or the like.

[0130] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Those skilled in the art can make various modifications and changes to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.< / with> < / dog> < / glasses> < / wearing> < / cat>

Claims

1. A method for token-merge based unsupervised text-to-graph semantic binding, characterized in that, The method comprises the following steps: Tokenization processing is performed on the input prompt text, and text embedding is obtained by encoding; wherein, the tokenization processing performed on the input prompt text comprises filling a start token and a plurality of end tokens at the beginning and end of the text, respectively; The text embedding is analyzed to obtain the coupling and semantic additivity of the text embedding; Based on the coupling and semantic additivity of the text embedding, a composite token is constructed, and the original end token is replaced with an end token that does not contain attribute information; The composite token is iteratively updated based on the semantic binding loss and the entropy loss, and the final text embedding is obtained based on the updated composite token and the replaced end token; Based on the final text embedding, a corresponding image is generated; Based on the semantic additivity of the token embedding, the entity and the attribute are jointly expressed by explicitly binding the token, and the semantic fusion of the token is performed in a element-by-element addition manner to obtain the composite token. The prompt text contains K entities represented by nouns and their corresponding tokens. For a prompt text containing K entities, the calculation formula of the composite token is: ， wherein, represents a composite token, each entity being associated with a token of a related object or property, represented as ; The calculation formula of the semantic binding loss is: ， wherein, is a time step, denotes a diffusion model, denotes a latent variable, denotes a composite token, is a text embedding, K denotes a number of entities.

2. The token merge based unsupervised text-to-diagram semantic binding method of claim 1, wherein, The calculation formula of the entropy regularization loss is: ， wherein, represents a cross-attention map corresponding to the k token, represents the entropy of each token.

3. The token merge based unsupervised text-to-diagram semantic binding method of claim 1, wherein, The input text prompt is encoded by using a CLIP text encoder to obtain a text embedding. Each embedding corresponds to a token.

4. The token merge based unsupervised text-to-diagram semantic binding method of claim 1, wherein, An SDXL model and the text embedding are used to generate a corresponding image. The SDXL model comprises an autoencoder and a diffusion model.

5. A training-free text-to-graph semantic binding system based on token merge, characterized in that, The token merging based training-free text-to-image semantic binding method of any one of claims 1-4 comprises: A text embedding extraction module is configured to perform tokenization processing on the input prompt text and encode the text embedding. The tokenization processing performed on the input prompt text comprises filling a start token and a plurality of end tokens at the beginning and end of the text, respectively. A text embedding analysis module is configured to analyze the text embedding to obtain the coupling and semantic additivity of the text embedding. A text-to-image semantic binding module is configured to construct a composite token based on the coupling and semantic additivity of the text embedding, and replace the original end token with an end token that does not contain attribute information. The composite token is iteratively updated based on the semantic binding loss and the entropy loss, and the final text embedding is obtained based on the updated composite token and the replaced end token. Based on the final text embedding, a corresponding image is generated.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps in the token merging based training-free text-to-image semantic binding method of any one of claims 1-4.

7. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps in the token merging based training-free text-to-image semantic binding method of any one of claims 1-4.

Images