Entity and interaction constraint oriented image generation method and device, equipment and medium

Abstract

The application discloses an entity and interaction constraint-oriented image generation method and device, equipment and medium, and belongs to the technical field of image processing. The method comprises the following steps: generating explicit triple interaction information according to the explicit triples contained in a target text prompt; extracting implicit triples contained in the target text prompt through a large language model, and generating implicit triple interaction information according to the implicit triples; the implicit triples are triple relationships that are not explicitly shown in the target text prompt; in the denoising process of a diffusion model, intermediate image features are obtained according to the explicit triple interaction information and the implicit triple interaction information and current image features in the current step denoising process; current image features in the next step denoising process are obtained according to entity semantic features and the intermediate image features through an entity control network; the above process is repeated until a target image is obtained, thereby improving the quality of the image generated by the diffusion model.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to an image generation method, apparatus, device, and medium oriented towards entities and interaction constraints. Background Technology

[0002] In the field of artificial intelligence, text-to-image (T2I) technology has made rapid progress. This technology empowers models to transform text descriptions into visual images, opening up entirely new creative avenues for numerous industries. While diffusion models, as the mainstream T2I model, can generate diverse, high-quality images and reconstruct the distribution of the original data, precisely controlling the generated content remains a key challenge in current research. Researchers have proposed various methods to control generated content through classes, text, images (including edges, lines, doodles, and skeletons), and layout. However, these methods struggle to generate high-quality images that meet the needs of practical applications when dealing with scenarios involving multiple entities and their complex interactions.

[0003] Specifically, these methods have the following main problems that limit their application in generating complex scenes: First, due to the lack of mining and understanding of implicit interaction relationships in the text, diffusion models have difficulty generating interactive scenes that conform to real-world logic. Second, models that rely on bounding box layouts are not effective in controlling the generation of individual entities, resulting in the loss of entity details or abnormal shapes.

[0004] In conclusion, how to simultaneously and precisely control the generation of each entity in the text as well as the interaction relationships between them remains a challenging problem. Summary of the Invention

[0005] The purpose of this application is to provide an image generation method, apparatus, device, and medium oriented towards entities and interaction constraints. By introducing Large Language Models (LLMs) to mine implicit interaction relationships and combining them with explicit interaction relationships as guiding conditions, and constructing an entity control network, high-quality image generation with reasonable interaction relationships and clear entity construction can be achieved, thereby solving the above-mentioned problems.

[0006] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide an image generation method oriented towards entities and interaction constraints, the method comprising: Based on the explicit triples contained in the target text prompt, generate explicit triple interaction information. The explicit triples are the triples explicitly shown in the target text prompt, and the triples include subject, action, and object. Using a large language model, the implicit triples contained in the target text prompt are extracted, and implicit triple interaction information is generated based on the implicit triples. The implicit triples are triple relationships that are not explicitly shown in the target text prompt. In the denoising process of the diffusion model, intermediate image features are obtained based on the explicit triplet interaction information and the implicit triplet interaction information and the current image features in the current denoising step. The entity control network obtains the current image features for the next denoising process based on the entity semantic features and the intermediate image features. The above process is repeated until the target image is obtained.

[0007] Optionally, obtaining the current image features in the next denoising process based on the entity semantic features and the intermediate image features includes: Model the entity semantic features and the intermediate image features to generate an entity soft attention mask. The entity semantic features are the semantic features corresponding to the subject and the object contained in the explicit triple. The entity localization image features are obtained by locating the subject and object contained in the explicit triples in the intermediate image features using the entity soft attention mask. The entity localization image is an intermediate image feature carrying the localization information corresponding to the subject and object contained in the explicit triples. The entity localization image features are enhanced by a multi-scale convolutional network to obtain multi-scale entity detail features; The multi-scale entity detail features and the intermediate image features are input into the cross-attention layer of the diffusion model, and the current image features in the next denoising process are obtained based on the output residual of the cross-attention layer and the intermediate image features.

[0008] Optionally, the method further includes: Extract multiple actions from the explicit triples contained in the target text prompt, and semantically encode the multiple actions to obtain a set of action semantic features; The set of action semantic features is clustered to obtain multiple semantic clusters, and each semantic cluster contains multiple action semantic features with similar semantics; In the global offset direction, each action semantic feature is offset along the direction of the cluster center of the semantic cluster to which the action semantic feature belongs, to obtain the first offset action semantic feature; In the local offset direction, each action semantic feature is offset according to the unit vector direction of the cluster center of the semantic cluster to which the action semantic feature belongs, to obtain the second offset action semantic feature; Based on multiple first offset action semantic features and multiple second offset action semantic features corresponding to multiple action semantic features, interactive action clustering offset information is generated. In the denoising process of the diffusion model, intermediate image features are obtained based on the explicit triplet interaction information and the implicit triplet interaction information, along with the current image features in the current denoising step. These features include: In the denoising process of the diffusion model, intermediate image features are obtained based on the explicit triplet interaction information, the implicit triplet interaction information, the interaction action clustering offset information, and the current image features in the current step of the denoising process.

[0009] Optionally, in the denoising process of the diffusion model, intermediate image features are obtained based on the explicit triplet interaction information, the implicit triplet interaction information, the interaction action clustering offset information, and the current image features in the current denoising step, including: The explicit triplet interaction information, the implicit triplet interaction information, and the interaction action clustering offset information are concatenated with the current image features in the current step of the denoising process to obtain a joint input representation; The joint input representation is mapped to Q vector, K vector, and V vector through a linear transformation; By using an interactive scaling factor, the calculation results of the Q vector, the K vector, and the V vector are magnified during the attention calculation process to obtain intermediate image features.

[0010] Optionally, the method further includes: In the first denoising stage of the diffusion model, when obtaining the current image features in the next denoising process based on the current image features in the current denoising process, the explicit triplet interaction information, the implicit triplet interaction information, or the interaction action clustering offset information are not introduced. In the denoising process of the diffusion model, intermediate image features are obtained based on the explicit triplet interaction information, the implicit triplet interaction information, the interaction action clustering offset information, and the current image features in the current denoising step, including: In the second denoising stage of the diffusion model, intermediate image features are obtained based on the explicit triplet interaction information and the implicit triplet interaction information and the current image features in the current denoising process. In the third denoising stage of the diffusion model, intermediate image features are obtained based on the implicit triplet interaction information, the interaction action clustering offset information, and the current image features in the current denoising process. The first denoising stage precedes the second denoising stage, and the second denoising stage precedes the third denoising stage.

[0011] Optionally, generating implicit triplet interaction information based on the implicit triplet includes: Semantic encoding is performed on each implicit triple to obtain the semantic features of each implicit triple; Differentiated feature mapping is performed on the subject, relation, and object in the implicit triple through three independent linear transformation layers to obtain the differentiated features of each implicit triple; By using a self-attention mechanism, the relationships between the subject, relation, and object in the differential features of each implicit triple are calculated to obtain the structured semantic association information of the implicit triple; Based on the structured semantic association information and the semantic features of the implicit triples, the implicit triple interaction information of each implicit triple is obtained.

[0012] Optionally, generating explicit triplet interaction information based on the explicit triplets contained in the target text prompt includes: Semantic encoding is performed on each explicit triple to obtain the semantic features of each explicit triple; Position encoding is performed on the bounding box information corresponding to the subject, action, and object in each explicit triple to obtain the position features of each explicit triple; For each explicit triple, the semantic features and positional features of the explicit triple are concatenated to obtain the concatenated features of each explicit triple. For each explicit triple, the splicing features related to the subject and object are input into the first multilayer perceptron, and the splicing features related to the action are input into the second multilayer perceptron to obtain the explicit triple interaction information for each explicit triple.

[0013] Secondly, embodiments of this application provide an image generation apparatus oriented towards entities and interaction constraints, the apparatus comprising: The explicit triplet interaction information generation module is used to generate explicit triplet interaction information based on the explicit triplets contained in the target text prompt. The explicit triplet is the triplet explicitly shown in the target text prompt, and the triplet includes a subject, an action, and an object. The implicit triple interaction information generation module is used to extract the implicit triples contained in the target text prompt through a large language model, and generate implicit triple interaction information based on the implicit triples, wherein the implicit triples are triple relationships not explicitly shown in the target text prompt. The intermediate image feature generation module is used to obtain intermediate image features in the denoising process of the diffusion model based on the explicit triplet interaction information and the implicit triplet interaction information and the current image features in the current step of the denoising process. The target image generation module is used to obtain the current image features in the next denoising process through the entity control network, based on the entity semantic features and the intermediate image features, and repeat the above process until the target image is obtained.

[0014] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps in the entity-oriented and interaction-constrained image generation method described in the first aspect.

[0015] Fourthly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps in the entity-oriented and interaction-constrained image generation method as described in the first aspect.

[0016] The image generation method proposed in this application, oriented towards entities and interaction constraints, includes: generating explicit triplet interaction information based on explicit triplets contained in a target text prompt, wherein the explicit triplets are those explicitly shown in the target text prompt, and the triplets include a subject, an action, and an object; extracting implicit triplets contained in the target text prompt using a large language model, and generating implicit triplet interaction information based on the implicit triplets, wherein the implicit triplets are triplet relationships not explicitly shown in the target text prompt; obtaining intermediate image features based on the explicit triplet interaction information, the implicit triplet interaction information, and the current image features in the current step of the denoising process using a diffusion model; obtaining the current image features in the next step of the denoising process using entity control network based on entity semantic features and the intermediate image features, and repeating the above process until the target image is obtained.

[0017] The technical solution of this application introduces a large language model to mine implicit triples in the target text prompt and performs deep semantic encoding to obtain implicit triple interaction information. Then, through attention, the implicit triple interaction information and explicit triple interaction information are injected into the diffusion model as additional guiding conditions to control the image generation process. At the same time, in conjunction with an entity control network, the generation control of each entity in the image and the interaction between entities is realized, which effectively improves the quality of the target image generated by the diffusion model. Attached Figure Description

[0018] Figure 1 This is a system diagram of an image generation method based on entities and interaction constraints proposed in this application; Figure 2 This is a flowchart illustrating an image generation method oriented towards entities and interaction constraints provided in an embodiment of this application; Figure 3 This is a complete framework diagram of an image generation method oriented towards entities and interaction constraints provided in an embodiment of this application; Figure 4 This is a schematic diagram of the framework of an image generation device oriented towards entities and interaction constraints provided in an embodiment of this application; Figure 5 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0021] The core idea of ​​the image generation method oriented towards entities and interaction constraints proposed in this application is as follows: 1) Obtain explicit interaction relationships in the target text prompt, and mine implicit interaction relationships in the input target text prompt using Large Language Models (LLMs). 2) Deeply semantically encode the relationships through a self-attention mechanism, using them as guiding conditions for the diffusion model to enhance the model's understanding of explicit and implicit interaction relationships. 3) Cluster and shift the interactive action features in the target text prompt, constructing a global-local bidirectional shift to achieve semantic enhancement and detail supplementation of the original actions, also using them as additional guiding conditions for the diffusion model, allowing the diffusion model to more accurately understand the concept of interactive "actions". Finally, constrain the various entities contained in the target text prompt through an entity control network.

[0022] Figure 1 The system diagram for the image generation method based on entities and interaction constraints proposed in this application is shown below: This application provides an image generation method oriented towards entities and interaction constraints, which is applied to an improved diffusion model, such as... Figure 1 As shown, in this improved diffusion model, among the n U-Net networks, In addition to the original Self Attention (SA) and Cross Attention (CA) layers, an Interaction Enhance Attention (IEA) layer and an Entity Control Network (ECN) layer are added between the Self Attention and Cross Attention layers.

[0023] Each U-Net network, in top-to-bottom order, includes at least: a self-attention layer, an interactive information-enhanced attention layer, an entity control network, and a cross-attention layer.

[0024] The interaction information enhancement attention layer is a gated self-attention layer inserted into the original Transformer structure of the diffusion model. This interaction information enhancement attention layer is used to fuse explicit triple interaction information, implicit triple interaction information, and interaction action clustering offset information. First, the current image features are concatenated with various types of interaction information to form a joint input before self-attention calculation. At the same time, a learnable interaction scaling coefficient δ is introduced into the self-attention calculation, and the diffusion model's ability to focus on important interaction information is enhanced by amplifying the key components related to interaction in the attention weights. In addition, the interaction information enhancement attention layer also designs a phased interaction information injection strategy to achieve precise control and generation guidance of complex interaction information.

[0025] Entity control networks (ECNs) are semantically guided entity control architectures built after interactive information enhancement attention layers to achieve fine-grained control over various entities (subjects and objects) in generated images. The ECN first uses a multilayer perceptron (MLP) to interact with current image features based on the semantic features of the subject and object, generating soft attention masks to dynamically locate the regions where each entity is located in the feature map. Then, a multi-scale convolutional network is used to extract and enhance the features of each masked entity in parallel, capturing detailed features of entities at different scales through convolutional kernels with different receptive fields. Finally, a dynamic fusion network is integrated, adaptively adjusting the convolutional kernel weights according to the input features to achieve semantic fusion of subject and object features. This fusion is then added to the output of the interactive information enhancement attention layer as input to the next layer (the cross-attention layer in the Transformer diffusion model). Simultaneously, the output residual of the cross-attention layer is connected to the output of the interactive information enhancement attention layer to maintain the coherence of the image content. Thus, without relying on bounding box annotations, the ECN achieves accurate localization and detail enhancement of multiple entities, significantly improving the semantic consistency and visual quality of entities in the generated image.

[0026] The following will, in conjunction with the accompanying drawings, describe in detail a method for generating images based on entities and interaction constraints, and the application of this improved diffusion model in this method, through specific embodiments and application scenarios.

[0027] Figure 2 This is a flowchart illustrating an image generation method based on entities and interaction constraints, according to an embodiment of this application. Figure 3 This is a complete framework diagram of an image generation method oriented towards entities and interaction constraints provided in an embodiment of this application.

[0028] Existing text-to-image (T2I) methods based on diffusion models suffer from the following significant problems in generating complex interactive scenes: The first problem is insufficient understanding of interaction relationships. Existing methods struggle to accurately understand the complex interaction relationships between entities described in text, leading to unreasonable interaction relationships in the generated images. For example, when generating a scene of "a person feeding a dog," problems may arise such as the "person" or "dog" being suspended in mid-air, the "person" not holding "food," the "food" pointing to the wrong entity position, and missing interaction relationships.

[0029] The second problem is the weak control over entities. Existing methods lack fine-grained control over individual entities during the generation process, resulting in poor entity generation quality, such as distorted entity shapes, blurred details, or abnormal limbs.

[0030] Based on the two problems mentioned above, this embodiment provides an image generation method oriented towards entities and interaction constraints, such as... Figure 2 As shown, the method includes steps S11 to S14: Step S11: Generate explicit triplet interaction information based on the explicit triplet contained in the target text prompt. The explicit triplet is the triplet explicitly shown in the target text prompt, and the triplet includes subject, action, and object.

[0031] In this embodiment, the target text prompt refers to a given text used to generate the target image. The target text prompt contains the content of the target image to be generated, such as explicit triples. An explicit triple is a triple relationship explicitly shown in the target text prompt that includes a subject, an action, and an object. For example, suppose the target text prompt is: "A person is blowing out a candle." Then the explicit triple it contains is [person-blowing-candle].

[0032] After extracting the explicit triples, explicit triple interaction information (i.e., ...) is generated based on the explicit triples. Figure 1In the context of EII (Explicit Interactive Information), explicit triplet interactive information refers to the interactive information between triples that are rich in semantic information and spatial awareness, obtained after encoding and feature fusion of each explicit triplet.

[0033] Optionally, in another embodiment, the step S11 of "generating explicit triplet interaction information based on the explicit triplets contained in the target text prompt" specifically includes steps S11-1 to S11-4: Step S11-1: Semantic encoding is performed on each explicit triple to obtain the semantic features of each explicit triple.

[0034] In this embodiment, reference Figure 1 In Figure a, the pre-trained CLIP model is used to semantically encode the subject, action, and object phrases in each explicit triplet in the text, thereby obtaining the semantic features of the subject, action, and object phrases in each explicit triplet.

[0035] Step S11-2: Encode the bounding box information corresponding to the subject, action, and object in each explicit triple to obtain the position features of each explicit triple.

[0036] In this embodiment, Fourier position encoding is performed on the bounding box information corresponding to the subject, action, and object phrases in each explicit triple contained in the text to obtain the position features of the subject, action, and object phrases in each explicit triple.

[0037] Step S11-3: For each explicit triple, the semantic features and positional features of the explicit triple are concatenated to obtain the concatenated features of each explicit triple.

[0038] In this embodiment, for the semantic and positional codes corresponding to the phrases "subject," "action," and "object" in each explicit triple, the semantic and positional codes of each phrase are concatenated at the feature level to obtain the concatenated features of each phrase contained in each explicit triple.

[0039] Step S11-4: For each explicit triplet, the splicing features related to the subject and object are input into the first multilayer perceptron, and the splicing features related to the action are input into the second multilayer perceptron to obtain the explicit triplet interaction information for each explicit triplet.

[0040] In this embodiment, the concatenated features of each phrase contained in each explicit triple are input into a multilayer perceptron (MLP) for deep feature fusion. Specifically, the concatenated features of phrases belonging to subjects and objects are input into a first multilayer perceptron (Object MLP), while the concatenated features of phrases belonging to actions are input into a second multilayer perceptron (Action MLP). These two multilayer perceptron networks are specially designed with different parameter structures and nonlinear activation functions, enabling differentiated feature extraction and mapping based on the different characteristics of subjects, actions, and objects. Finally, the explicit triplet interaction information of each explicit triplet is obtained, which can be formalized as follows: ,in These represent the final encoding results for the subject, action, and object, respectively.

[0041] Step S12: Extract the implicit triples contained in the target text prompt using a large language model, and generate implicit triple interaction information based on the implicit triples. The implicit triples are triple relationships not explicitly shown in the target text prompt.

[0042] In this embodiment, the target text prompt is input into a large language model. The implicit interaction relationships in the target text prompt are mined through the thought chain reasoning of the large language model, and these implicit interaction relationships are organized into the form of triples, i.e., implicit triples.

[0043] Specifically, a specific thought chain prompt template is first constructed, and the carefully designed prompt template guides the large language model to perform deep reasoning on the target text prompt.

[0044] For example, given a simple description like "a person is blowing out candles," the large language model can infer a series of implicit interactions that align with real-world logic, such as "the person is approaching the cake," "there are candles on the cake," and "the person's mouth is pointing at the flame." These inference results are systematically organized into standard triplet forms, resulting in multiple implicit triples.

[0045] Then, after extracting the implicit triples, a self-attention mechanism is used to perform deep semantic encoding on the implicit triples, generating semantically rich implicit triple interaction information (i.e. Figure 1 (III, Implicit interactive information).

[0046] Optionally, in another embodiment, in order to enhance the diffusion model's semantic understanding of complex interaction scenarios and to provide rich implicit relational priors for the subsequent generation of images that conform to real-world logic, step S12, "generating implicit triplet interaction information based on the implicit triplets," specifically includes steps S12-1 to S12-4: Step S12-1: Semantically encode each implicit triple to obtain the semantic features of each implicit triple.

[0047] In this embodiment, reference Figure 1 In Figure b, the pre-trained CLIP model is used to semantically encode the subject, action, and object phrases in each implicit triple, thereby obtaining the semantic features of the subject, action, and object phrases in each implicit triple.

[0048] Step S12-2: Differentiated feature mapping is performed on the subject, relation, and object in the implicit triple through three independent linear transformation layers to obtain the differentiated features of each implicit triple.

[0049] In this embodiment, the semantic features corresponding to the subject, relation, and object in each implicit triple are then subjected to differential feature mapping using three independent linear transformation layers to obtain the differential features of each implicit triple. The design of three independent linear transformation layers enables the subject, relation, and object in each implicit triple to be processed in a targeted manner according to their different roles in the interaction relationship.

[0050] Step S12-3: Through the self-attention mechanism, calculate the relationship between the subject, relation, and object in the differential features of each implicit triple to obtain the structured semantic association information of the implicit triple.

[0051] In this embodiment, after obtaining the differentiated features of each implicit triple, the structured semantic associations within the implicit triples are deeply mined through a self-attention mechanism. This mechanism can dynamically calculate the attention weights between the subject, relation, and object in each implicit triple, capture the complex dependency relationships between the subject, relation, and object, and thus obtain the structured semantic association information of each implicit triple.

[0052] Step S12-4: Based on the structured semantic association information and the semantic features of the implicit triples, obtain the implicit triple interaction information of each implicit triple.

[0053] In this embodiment, the structured semantic association information extracted from each implicit triple is injected into the semantic features obtained in step S12-1 through residual connections, forming the implicit triple interaction information corresponding to that implicit triple, which can be formalized as follows: ,in These represent the final encoding results of the subject, action, and object in the implicit triple, respectively, enhancing the understanding of complex interaction logic in target text prompts and providing important, implicit semantic supplements for generating images that conform to real-world logic.

[0054] The output of the self-attention layer contains the global semantic relationships within a triple. This output is then injected into the original semantic features of the roles within each implicit triple via residual connections, resulting in the final deep semantic encoding result: the implicit triple interaction information of each implicit triple. This implicit triple interaction information enriches the diffusion model's understanding of the interaction scenario and improves the plausibility of the generated interaction relationships in the image.

[0055] Step S13: In the denoising process of the diffusion model, intermediate image features are obtained based on the explicit triplet interaction information and the implicit triplet interaction information and the current image features in the current denoising process.

[0056] In this embodiment, as Figure 3 As shown, in the denoising process of the diffusion model, the explicit triplet interaction information and implicit triplet interaction information obtained in the above steps are fused with the current image features output by the self-attention layer in the current denoising process through the interaction information enhancement attention layer. This allows the diffusion model to enhance its understanding of the target text prompts by combining the explicit triplet interaction information and implicit triplet interaction information during the denoising process, thereby obtaining the intermediate image features in the current denoising process. Here, the current image features refer to the image visual features in the current denoising process, and the intermediate image features refer to the output results of the interaction information enhancement self-attention layer.

[0057] Step S14: Using the entity control network, the current image features for the next denoising process are obtained based on the entity semantic features and the intermediate image features. The above process is repeated until the target image is obtained.

[0058] In this embodiment, as Figure 3 As shown, after obtaining the intermediate image features in the current denoising process, entity semantic features in the target text prompt are extracted. Entity semantic features refer to the semantic features in the target text prompt related to the subject and object contained in the explicit triple. The entity semantic features and the intermediate image features output by the interaction information enhancement attention layer are fused through the entity control network and used as the input of the cross attention layer. The current image features in the next denoising process are obtained through the output of the cross attention layer.

[0059] In the subsequent denoising process, steps S13 and S14 are repeated until the image features generated in the final denoising step are used as the target image generated by the diffusion model according to the target prompt text.

[0060] The technical solution described above involves introducing a large language model to mine implicit triples in the target text prompt and performing deep semantic encoding to obtain implicit triple interaction information. Then, through attention, the implicit triple interaction information and explicit triple interaction information are injected into the diffusion model as additional guiding conditions to control the image generation process. Simultaneously, in conjunction with an entity control network, the generation control of various entities in the image and the interactions between entities is realized, effectively improving the generation quality of the target image.

[0061] In conjunction with the technical solutions of the above embodiments, an embodiment of this application also provides another image generation method oriented towards entities and interaction constraints. In this method, step S14, "obtaining the current image features in the next denoising process based on the entity semantic features and the intermediate image features," specifically includes steps S14-1 to S14-4: Step S14-1: Model the entity semantic features and the intermediate image features to generate an entity soft attention mask. The entity semantic features are the semantic features corresponding to the subject and object contained in the explicit triple.

[0062] In this embodiment, reference Figure 1 The e-graph in the diagram uses a multilayer perceptron to extract the semantic features (i.e., entity semantic features) corresponding to the subject and object in the explicit triples. Figure 1 The entity semantic features in the current step are used to perform deep interaction modeling with the intermediate image features obtained in step S13 during the current step diffusion process to generate soft attention masks corresponding to the subject and object in the explicit triples. Figure 1 In the context of Entity Masks, soft attention masks are not simple binarization segments, but attention maps with continuous weights that can accurately indicate the spatial distribution of the subject and object in the explicit triple on the feature map. In the attention map containing soft attention masks, the mask value of features related to interaction information is not 0, while the mask value of irrelevant features is 0.

[0063] Step S14-2: Locate the subject and object contained in the explicit triples in the intermediate image features using the entity soft attention mask to obtain entity localization image features. Figure 1 The entity positioning image (which contains the Subject and Object in the explicit triple) is an intermediate image feature carrying the positioning information corresponding to the subject and object contained in the explicit triple.

[0064] In this embodiment, for each subject or object included in the explicit triple, a CNN feature enhancement network is used ( Figure 1 The Feature Enhanced CNN processes the intermediate image features according to the soft attention mask corresponding to each subject or object, and obtains the entity localization image features corresponding to each entity (subject or object in an explicit triple). The entity localization image features are the masked entity features, which contain the image features related to the entity in the intermediate image features.

[0065] Step S14-3: Enhance the entity localization image features using a multi-scale convolutional network to obtain multi-scale entity detail features.

[0066] In this embodiment, a multi-scale convolutional network containing convolutional kernels of different scales is used. Figure 1 The FeatureFusion Network in the model combines the features of the localized image with the features of the current image for parallel processing and enhancement. It captures the local detail features (such as edges and textures) and global structural features (such as shape and pose) of each entity through convolutional kernels of different scales.

[0067] Small-scale convolutional kernels are used to extract local detail features, while large-scale convolutional kernels are used to extract global structural features. This multi-scale convolutional network design ensures adaptive processing of entities of different sizes.

[0068] For example, small-scale convolutional kernels can be used to capture local detail features, such as facial texture and clothing folds, while large-scale convolutional kernels can be used to capture global structural features, such as the overall posture and position of a person. The scale of the convolutional kernel can be determined based on the size of the feature region to be extracted in the feature map.

[0069] After enhancing the entity localization image features through a multi-scale convolutional network, the obtained entity enhancement features (subject enhancement features and object enhancement features) are input into the entity feature dynamic fusion network. The entity feature dynamic fusion network dynamically adjusts the weights of the convolution kernels based on the input entity enhancement features, enabling the diffusion model to adaptively optimize the feature fusion effect under different inputs.

[0070] Specifically, the entity feature dynamic fusion network dynamically generates fusion weights based on the input entity enhancement features. During the feature fusion stage, it dynamically adjusts the convolution kernel weights, adaptively enhancing or suppressing specific region features based on the distribution characteristics of the input entity enhancement features. This improves the adaptability of the diffusion model to different input features and the feature fusion effect. When the mean of the input entity enhancement features is high (e.g., in salient regions), the convolution kernel weights are increased to strengthen the response; when the input entity enhancement features are low (e.g., in background noise), the weights are decreased to suppress interference from irrelevant backgrounds. In summary, the dynamic enhancement of the entity feature dynamic fusion network enables the diffusion model to adaptively optimize the feature fusion effect under different input conditions (e.g., time step, noise level), improving the semantic coherence and detail quality of the subject and object mask fused image. It achieves precise control over the generation of each entity part, effectively avoiding confusion between entities, improving the overall quality of the generated image, and ultimately obtaining multi-scale entity detail features.

[0071] Step S14-4: Input the multi-scale entity detail features and the intermediate image features into the cross-attention layer of the diffusion model, and obtain the current image features in the next denoising process based on the output residual of the cross-attention layer and the intermediate image features.

[0072] In this embodiment, the multi-scale entity detail features are added to the intermediate image features output by the interactive information enhancement attention layer and used as the input to the next layer (i.e., the cross-attention layer) in the diffusion model. At the same time, the output residual of the cross-attention layer is connected to the intermediate image features output by the interactive information enhancement attention layer to maintain the coherence and naturalness of the image background, and finally the current image features in the next denoising process are obtained.

[0073] in, Figure 1 In this context, "prompt" refers to the original target text prompt. We need to input this prompt into the cross-attention layer. Since the entity control network only focuses on the features of the subject and object regions, we need to add the output of the entity control network and the interaction information enhancement attention layer as the visual feature input for the subsequent cross-attention layer. Our designed interaction information enhancement attention layer introduces a large amount of additional interaction information, including many action information that is strictly different from the target text prompt. This may lead to biases in the generated results. To prevent these information from producing suboptimal effects on the diffusion model, the subsequent cross-attention layer also needs to be guided by the target text prompt (i.e., the prompt), mainly for action correction to mitigate the biases generated by the diffusion model. Furthermore, we require the output residual of this cross-attention layer to be connected to the output of the previous interaction attention layer to ensure the coherence of the generated image.

[0074] By introducing an entity control network, the technical solution of this embodiment can achieve the localization and detail enhancement of multiple entities contained in the target text prompt without relying on entity bounding box annotations, thus significantly improving the visual quality of each entity contained in the target image to be generated.

[0075] In conjunction with the technical solutions of the above embodiments, this application also provides another image generation method oriented towards entities and interaction constraints. To address the problem of the diffusion model's ambiguous semantic understanding of the concept of "action," leading to action generation deviations and severely affecting the semantic accuracy of images (e.g., generating "feeding" as "grabbing food" or "throwing food"), this method, based on the above issues, extracts interaction action clustering offset information from the target text prompts, in addition to combining implicit and explicit interaction information, and uses this information as a guiding condition for the diffusion model.

[0076] The process of extracting the clustering offset information of interactive actions is specifically shown in steps S21 to S25: Step S21: Extract multiple actions from the explicit triples contained in the target text prompt, and semantically encode the multiple actions to obtain a set of action semantic features.

[0077] In this embodiment, reference Figure 1 In Figure c, the pre-trained CLIP is first used to semantically encode all action phrases appearing in the explicit triples contained in the target text prompt, thus constructing an action semantic feature set. The action semantic feature set contains the action semantic features of each action appearing in the explicit triples contained in the target text prompt.

[0078] Step S22: Cluster the set of action semantic features to obtain multiple semantic clusters, each semantic cluster containing multiple action semantic features with similar semantics.

[0079] In this embodiment, considering that semantic encoding of actions alone may result in ambiguous expression or unclear boundaries, the K-means clustering algorithm is introduced in step S22 to perform unsupervised clustering analysis on all action semantic features in the action semantic feature set, thereby discovering the inherent relationship between action semantic features and classifying actions with similar action semantic features into the same semantic cluster, resulting in multiple semantic clusters. For example, "racing" and "riding" are classified as dynamic motion and assigned to the same semantic cluster.

[0080] Step S23: In the global offset direction, each action semantic feature is offset according to the direction of the cluster center of the semantic cluster to which the action semantic feature belongs, to obtain the first offset action semantic feature.

[0081] In this embodiment, in the global offset direction, the action semantic features of each action are moved along the direction of the cluster center of the semantic cluster to which the action semantic features belong, to obtain the first offset action semantic features. The first offset action semantic features refer to the action semantic features obtained after the action semantic features of each action are offset along the direction of the cluster center in the global offset direction. In this way, the common features of multiple actions contained in each semantic cluster can be strengthened, ensuring that the multiple actions contained in each semantic cluster maintain category consistency.

[0082] Step S24: In the local offset direction, each action semantic feature is offset according to the unit vector direction of the cluster center of the semantic cluster to which the action semantic feature belongs, to obtain the second offset action semantic feature.

[0083] In this embodiment, in the local offset direction, each action semantic feature is offset along the unit vector direction of the cluster center of the semantic cluster to which the action semantic feature belongs, to obtain the second offset action semantic feature. This adjustment in the local offset direction enriches the detailed representation of the action semantic feature and can generate action variants with subtle differences (such as generating "hand-carrying", "shoulder-carrying", etc. for the "carrying" action).

[0084] Specifically, offsets are set in the local offset direction and the global offset direction, such as [-0.1, -0.05, 0.05, 0.1, 0.15, 0.2]. Positive offsets are used to enhance the original action, while negative offsets are used to reduce the influence of certain features.

[0085] At each offset, each action feature is offset along two directions: (a) the direction of the cluster center of each action feature as the global offset direction; and (b) the direction of the normalized vector of each action feature to its cluster center as the local offset direction.

[0086] The global offset direction aligns each action feature with its corresponding semantic category, reinforcing the commonalities of basic action classes. Meanwhile, the local offset direction adds local details such as pose, intensity, or angular changes for each action. This method enables the model to learn a richer distribution of action semantics, avoiding overfitting to a single feature.

[0087] Step S25: Generate interactive action clustering offset information based on multiple first offset action semantic features and multiple second offset action semantic features corresponding to multiple action semantic features.

[0088] In this embodiment, by systematically combining first and second offset action semantic features with different offsets and directions, multiple sets of semantically enhanced action semantic features are generated for a single action semantic feature, namely, interactive action clustering offset information (i.e., Figure 1 (AOI, Action offset information).

[0089] The step S13, "in the denoising process of the diffusion model, obtaining intermediate image features based on the explicit triplet interaction information and the implicit triplet interaction information and the current image features in the current denoising process," specifically includes: Step S13-1: In the denoising process of the diffusion model, intermediate image features are obtained based on the explicit triplet interaction information, the implicit triplet interaction information, the interaction action clustering offset information, and the current image features in the current denoising process.

[0090] In this embodiment, during the denoising process of the diffusion model, the explicit triplet interaction information, implicit triplet interaction information obtained in steps S11 and S12, and the interaction action clustering offset information obtained in steps S21 to S25 are fused with the current image features in the current step of the denoising process through the interaction information enhancement attention layer. This allows the diffusion model to enhance its understanding of the target text prompts by combining the explicit triplet interaction information, implicit triplet interaction information, and interaction action clustering offset information during the denoising process, thereby obtaining intermediate image features in the current step of the denoising process. Here, the current image features refer to the image visual features in the current step of the denoising process.

[0091] Through the above embodiments, in the process of denoising the diffusion model, not only are explicit triple interaction information and implicit triple interaction information used as guiding conditions in the current step of denoising, but also interactive action clustering offset information is generated to maintain the semantic core of the basic actions contained in the target text prompt, while also covering rich action detail changes. The generated interactive action clustering offset information is also used as a guiding condition for the diffusion model, providing sufficient learning material for the diffusion model to understand complex action semantics, and indicating the diffusion model's ability to distinguish and generate complex action semantics.

[0092] In conjunction with the technical solutions of the above embodiments, an embodiment of this application also provides another image generation method oriented towards entities and interaction constraints. In this method, the step S13-1, "in the denoising process of the diffusion model, obtaining intermediate image features based on the explicit triplet interaction information, the implicit triplet interaction information, and the interaction action clustering offset information and the current image features in the current step of the denoising process," specifically includes steps S13-1-1-1 to S13-1-1-3: Step S13-1-1-1: The explicit triplet interaction information, the implicit triplet interaction information, and the interaction action clustering offset information are concatenated with the current image features in the current step of denoising to obtain a joint input representation.

[0093] In this embodiment, during the denoising process of the current step of the diffusion model, the current image features obtained in the current step of the denoising process are concatenated with the three types of interaction information (explicit triplet interaction information, implicit triplet interaction information, and interaction action clustering offset information) obtained above to form a rich joint input representation. This concatenation is not a simple feature stacking, but rather ensures that information from different sources can interact effectively through specific dimension alignment and normalization processing.

[0094] Step S13-1-1-2: Through linear transformation, the joint input representation is mapped to Q vector, K vector and V vector.

[0095] In this embodiment, the joint input representation obtained in the above steps is mapped to self-attention Query, Key and Value vectors, namely Q vector, K vector and V vector, through linear transformation.

[0096] Step S13-1-1-3: During the attention calculation process, the calculation results of the Q vector, the K vector, and the V vector are magnified by the interactive scaling factor to obtain intermediate image features.

[0097] In this embodiment, a learnable interaction scaling factor is introduced during the attention calculation process. The interaction scaling factor significantly enhances the diffusion model's ability to focus on key interaction information—explicit triplet interaction information, implicit triplet interaction information, and interaction action clustering offset information—by amplifying the weights of these interaction-related attention vectors (Q vector, K vector, and V vector). This self-attention mechanism, which enhances the three types of interaction information, can be formalized as follows: .

[0098] By introducing an interaction scaling factor, the diffusion model tends to focus more on interaction-related regions (such as the contact point between a "hand" and an "object," or the association between a "person" and an "action") when calculating feature interactions. This enhances its ability to capture important interaction relationships, strengthens its modeling ability for complex interactions (such as person-object relationships and action-target associations), and improves generation stability. By adding this interaction-enhanced attention, the rich interaction information obtained above can be effectively input into the diffusion model, strengthening the control and guidance of interactions during denoising, thereby improving the rationality and accuracy of interactions and enhancing the overall quality of generated images.

[0099] In conjunction with the technical solutions of the above embodiments, this application also provides another image generation method oriented towards entities and interaction constraints. In this embodiment, a phased interaction information injection strategy is introduced. To implement this strategy, this embodiment divides the entire denoising process of the diffusion model into three denoising stages, namely, a first denoising stage, a second denoising stage, and a third denoising stage. The first denoising stage precedes the second denoising stage, and the second denoising stage precedes the third denoising stage. Different interaction information is introduced in different denoising stages to ensure the accurate guidance of the diffusion model by various complex interaction information, thereby improving the accuracy and stability of the diffusion model in the process of generating the target image.

[0100] Specifically, in this embodiment, the method further includes: Step S31: In the first denoising stage of the diffusion model, in the process of obtaining the current image features in the next denoising process based on the current image features in the current denoising process, the explicit triplet interaction information, the implicit triplet interaction information, or the interaction action clustering offset information are not introduced.

[0101] In this embodiment, reference Figure 1 The strategy of injecting three types of interaction information (explicit interaction, implicit interaction, and interaction action offset information) in stages into the d-graph is a carefully designed method aimed at optimizing the performance of the diffusion model in the image generation process. The core advantage of this strategy is that it can dynamically adjust the model's attention to different interaction information according to the changes in noise level during denoising, thereby improving the model's stability and efficiency while ensuring the quality of the generated image.

[0102] in, Figure 1 The "Freeze" option indicates that the parameters of the layer are frozen and do not require training, as the method in this application is training-free.

[0103] The diffusion process of the diffusion model is a reverse denoising process. T represents the total number of denoising steps, that is, it will go through T U-Net denoising networks. Therefore, the first denoising step refers to step T, the second denoising step refers to step T-1, and so on. The last denoising step refers to step 1 (i.e., T=1).

[0104] In the early stages of the denoising process (i.e., the first denoising stage, t ≥ s1 × T, where t represents the current denoising progress, T represents the total number of denoising steps, and s1 is a hyperparameter with a value of [0,1], used to balance the level of text description and interactive control, and can control the number of steps in the first denoising stage, i.e., how many denoising steps the first stage will undergo), the noise level in the image is high. At this time, if complex interactive information is directly introduced, the diffusion model may find it difficult to construct a reasonable overall structure due to over-focusing on the detailed features in the interactive information. Therefore, to avoid explicit triplet interactive information, implicit triplet interactive information, and interactive action clustering offset information, etc., The introduction of interactive information has a suboptimal impact on the basic diffusion model. In the early stages of the denoising process, the three types of interactive information—explicit triplet interactive information, implicit triplet interactive information, and interactive action clustering offset information—are not introduced. Instead, the diffusion model directly generates a basic image with a reasonable structure but lacking detailed features based on the target text prompts. In other words, in the U-Net network of the diffusion model in the first denoising stage, interactive information is not introduced to enhance the attention layer and entity control network. Instead, the output of the self-attention layer in the U-Net network is directly input into the cross-attention layer to obtain the current image features in the next denoising process.

[0105] The step S13-1, "In the denoising process of the diffusion model, intermediate image features are obtained based on the explicit triplet interaction information, the implicit triplet interaction information, the interaction action clustering offset information, and the current image features in the current denoising process," specifically includes steps S13-1-2-1 to S13-1-2-2: Step S13-1-2-1: In the second denoising stage of the diffusion model, intermediate image features are obtained based on the explicit triplet interaction information and the implicit triplet interaction information and the current image features in the current denoising process.

[0106] In this embodiment, as the intermediate denoising process (i.e., the second denoising stage, s2×T<t<s1×T, where t represents the current denoising progress, T represents the total number of denoising steps, and s2 is a hyperparameter with a value of [0,1], used to control the sampling time for injecting more detailed and richer interactive information to supplement the details of interactive actions, and also greatly affects the generation process. s2 and s1 jointly control the number of steps in the second denoising stage, i.e. how many denoising steps the second stage will undergo) progresses, considering that if a large amount of complex interactive information is used directly, the diffusion model may overemphasize local details in the high noise of the intermediate denoising stage, thus leading to structural distortion of the target image finally generated by the diffusion model, this step proposes a technical concept of injecting explicit triplet interactive information and implicit triplet interactive information in the second denoising stage to construct a reasonable scene layout and basic structure. Therefore, in the second denoising stage of the diffusion model, the explicit triplet interactive information and implicit triplet interactive information are respectively injected with the current image features in the current denoising process ( Figure 1 The visual features in the current denoising process are fused to obtain the fusion result of explicit triplet interaction information and current image features in the current denoising process, as well as the fusion result of implicit triplet interaction information and current image features in the current denoising process. The two fusions in this step are combined and weighted and summed. For example, the weight of the fusion result of explicit triplet interaction information and current image features in the current denoising process is w, and the weight of the fusion result of implicit triplet interaction information and current image features in the current denoising process is 1-w, so as to obtain the intermediate image features in each step of the denoising process in the second denoising stage.

[0107] For example, in the second denoising stage, by introducing explicit and implicit triplet interaction information, the diffusion model is guided to construct a reasonable scene layout and basic structure for the image. Explicit interaction information (such as "Xiaoming is handing an apple to Xiaohong") provides the diffusion model with clear semantic feature guidance from the target text prompt, while implicit interaction information (such as "Xiaoming and Xiaohong are face-to-face", "Xiaoming is holding an apple", "Xiaoming's hand is facing Xiaohong", etc.) further provides the diffusion model with implicit semantic feature guidance from the target text prompt.

[0108] Step S13-1-2-2: In the third denoising stage of the diffusion model, intermediate image features are obtained based on the implicit triplet interaction information, the interaction action clustering offset information, and the current image features in the current denoising process.

[0109] In this embodiment, in the later stage of the denoising phase of the diffusion model (i.e., the third denoising phase, t≤s2×T, where s2 controls the number of steps in the third denoising phase, i.e., how many denoising iterations the third phase will undergo), a technical concept is proposed to fuse implicit triplet interaction information and interaction action clustering offset information with the current image features in the current denoising step, in order to refine the interaction relationships and interaction actions in the implicit triplets. Therefore, in the third denoising phase of the diffusion model, the interaction action clustering offset information and implicit triplet interaction information are respectively fused with the current image features in the current denoising step. The two fusions are then combined to obtain the fusion results of the interaction action clustering offset information and the current image features in the current step of denoising, as well as the fusion results of the implicit triplet interaction information and the current image features in the current step of denoising. The two fusions in this step are then weighted and summed. For example, the weight of the fusion result of the interaction action clustering offset information and the current image features in the current step of denoising is w, and the weight of the fusion result of the implicit triplet interaction information and the current image features in the current step of denoising is 1-w. This yields the intermediate image features in each step of the denoising process in the third denoising stage.

[0110] For example, in the third denoising stage, since the structure of the image in the previous denoising process is already relatively clear, introducing implicit triplet interaction information and interaction action offset information (such as Xiaoming's arm movement details) can further refine the implicit interaction information and the expression of interaction actions, making the generated image more natural and realistic.

[0111] The phased interactive information injection strategy proposed in this embodiment can dynamically adjust the degree of attention of the diffusion model to different interactive information according to the changes in noise level during the diffusion model denoising process, thereby improving the stability and efficiency of the diffusion model while ensuring the quality of the generated target image.

[0112] To verify the effectiveness and robustness of the proposed entity- and interaction-constrained image generation method in real-world image generation, experiments were conducted using the standardized public dataset HICO-DET, which includes complex interaction scenarios. The evaluation metrics used in this application are Frèchet Inception Distance (FID), Kernel Inception Distance (KID), and HOI Detection Score. The experimental system was deployed on an Autodl server equipped with an NVIDIA A100-SXM4-40GB GPU. The evaluation test environment was conducted in a real-world image generation task, using the InteractDiffusion checkpoint model as the base diffusion model, with the PLMS sampler performing 50-step diffusion sampling. CLIP was used for text prompt semantic encoding, and the large language model qwen-turbo was used for implicit interaction relationship mining. This application conducted quantitative comparative tests on the aforementioned datasets, and the quantitative results are shown in Table 1 below. This application outperforms existing state-of-the-art methods such as Stable Diffusion, GLIGEN, and InteractDiffusion on the HICO-DET dataset, exhibiting lower FID and KID and higher HOI detection scores, demonstrating excellent robustness.

[0113] Table 1

[0114] In summary, this application proposes a diffusion-based image generation method oriented towards dual constraints of entities and interactions. It utilizes a large language model to mine implicit interaction relationships in the input target text prompts and performs deep semantic encoding, effectively improving the rationality of interaction relationships between entities in the generated images. Furthermore, by combining a clustering and offset mechanism based on interactive action features, a global-local bidirectional offset is constructed to enhance action semantic understanding and detail representation, achieving accurate generation of interactive actions. Simultaneously, an entity control network is designed, and semantically guided mask generation enables refined generation control of each entity, ultimately improving the quality of images generated by the diffusion model.

[0115] Experimental results show that this application significantly improves the rationality of interaction relationships, the accuracy of actions, and the overall quality of images in complex human-object interaction scenarios. It overcomes the limitations of existing methods in understanding entities and their interaction control, and has broad application value and promotion potential in various application scenarios such as intelligent content creation, virtual scene construction, and human-computer interaction simulation.

[0116] The key point of this application is to propose a diffusion-based image generation method oriented towards dual constraints of entities and interactions. By deeply integrating the semantic understanding capabilities of a large language model, the semantic enhancement mechanism of interactive actions, and entity-level visual feature control, it achieves precise joint control over the entities themselves and their complex interactive relationships in the generated image.

[0117] First, this application utilizes Large Language Models (LLMs) to mine the implicit interaction relationships described in the input target text prompts, and performs deep semantic encoding through a self-attention mechanism, injecting it as a guiding condition into the diffusion model, which significantly improves the rationality of the interaction relationships between entities in the generated image.

[0118] Secondly, this application proposes a clustering offset method for interactive action features. By constructing a global-local bidirectional offset, the original action is semantically enhanced and details are supplemented. This strategy makes the diffusion model's understanding of the "action" concept more accurate and improves the semantic consistency and richness of the generated actions.

[0119] Furthermore, this application designs an entity control network, which first guides the generation of soft attention masks through entity semantics, and then integrates multi-scale convolutional feature enhancement and dynamic feature fusion, thus solving the problem of poor quality of entity detail generation in existing text-based image tasks.

[0120] This application significantly improves the interactivity, motion accuracy, and overall quality of generated images in complex human-object interaction scenarios, and has good scalability and application potential.

[0121] It should be noted that the entity- and interaction-constrained image generation method provided in this application can be executed by an entity- and interaction-constrained image generation device, or a control module within that device for executing the entity- and interaction-constrained image generation method. This application uses the execution of the entity- and interaction-constrained image generation method by an entity- and interaction-constrained image generation device as an example to illustrate the entity- and interaction-constrained image generation method provided in this application.

[0122] Figure 4 This is a schematic diagram of the framework of an image generation device oriented towards entities and interaction constraints according to an embodiment of this application, with reference to... Figure 4 .

[0123] One embodiment of this application provides an image generation apparatus oriented towards entities and interaction constraints, the apparatus comprising: The explicit triplet interaction information generation module 11 is used to generate explicit triplet interaction information based on the explicit triplet contained in the target text prompt. The explicit triplet is the triplet explicitly shown in the target text prompt, and the triplet includes a subject, an action, and an object. The implicit triplet interaction information generation module 12 is used to extract the implicit triplets contained in the target text prompt through a large language model, and generate implicit triplet interaction information based on the implicit triplets, wherein the implicit triplets are triplet relationships not explicitly shown in the target text prompt. The intermediate image feature generation module 13 is used to obtain intermediate image features in the denoising process of the diffusion model based on the explicit triplet interaction information and the implicit triplet interaction information and the current image features in the current step of the denoising process. The target image generation module 14 is used to obtain the current image features in the next denoising process through the entity control network based on the entity semantic features and the intermediate image features, and repeat the above process until the target image is obtained.

[0124] Optionally, the target image generation module 14 includes: An entity semantic feature generation unit is used to model entity semantic features and intermediate image features to generate entity soft attention masks. The entity semantic features are the semantic features corresponding to the subject and object contained in the explicit triples. An entity localization image feature generation unit is used to locate the subject and object contained in the explicit triples in the intermediate image features through the entity soft attention mask to obtain entity localization image features. The entity localization image is an intermediate image feature carrying localization information corresponding to the subject and object contained in the explicit triples. A multi-scale entity detail feature generation unit is used to enhance the entity localization image features through a multi-scale convolutional network to obtain multi-scale entity detail features. The current image feature generation unit is used to input the multi-scale entity detail features and the intermediate image features into the cross-attention layer of the diffusion model, and obtain the current image features in the next denoising process based on the output residual of the cross-attention layer and the intermediate image features.

[0125] Optionally, the device further includes: The action semantic feature set acquisition module is used to extract multiple actions from the explicit triples contained in the target text prompt, and to semantically encode the multiple actions to obtain the action semantic feature set. The clustering module is used to cluster the set of action semantic features to obtain multiple semantic clusters, and each semantic cluster contains multiple action semantic features with similar semantics; The global offset module is used to offset each action semantic feature in the global offset direction according to the direction of the cluster center of the semantic cluster to which the action semantic feature belongs, so as to obtain the first offset action semantic feature. The local offset module is used to offset each action semantic feature in the local offset direction according to the unit vector direction of the cluster center of the semantic cluster to which the action semantic feature belongs, so as to obtain the second offset action semantic feature. The interactive action clustering offset information acquisition module is used to generate interactive action clustering offset information based on multiple first offset action semantic features and multiple second offset action semantic features corresponding to multiple action semantic features. The intermediate image feature generation module 13 includes: The intermediate image feature generation unit is used to obtain intermediate image features during the denoising process of the diffusion model based on the explicit triplet interaction information, the implicit triplet interaction information, the interaction action clustering offset information, and the current image features in the current step of the denoising process.

[0126] Optionally, the intermediate image feature generation unit includes: The joint input representation acquisition unit is used to concatenate the explicit triplet interaction information, the implicit triplet interaction information, and the interaction action clustering offset information with the current image features in the current step of denoising to obtain the joint input representation; Attention vector mapping unit, used to map the joint input representation into Q vector, K vector and V vector through linear transformation; The attention amplification unit is used to amplify the calculation results of the Q vector, the K vector, and the V vector during the attention calculation process by using an interactive scaling factor, so as to obtain intermediate image features.

[0127] Optionally, the device further includes: The first denoising module is used in the first denoising stage of the diffusion model, in the process of obtaining the current image features in the next denoising process based on the current image features in the current denoising process, without introducing the explicit triplet interaction information, the implicit triplet interaction information, or the interaction action clustering offset information. The intermediate image feature generation unit includes: The first intermediate image feature generation unit is used to obtain intermediate image features in the second denoising stage of the diffusion model based on the explicit triplet interaction information and the implicit triplet interaction information and the current image features in the current denoising process. The second intermediate image feature generation unit is used to obtain intermediate image features in the third denoising stage of the diffusion model based on the implicit triplet interaction information, the interaction action clustering offset information, and the current image features in the current step denoising process. The first denoising stage precedes the second denoising stage, and the second denoising stage precedes the third denoising stage.

[0128] Optionally, the implicit triplet interaction information generation module 12 includes: The implicit semantic feature acquisition unit is used to semantically encode each implicit triple to obtain the semantic features of each implicit triple; The differential feature acquisition unit is used to perform differential feature mapping on the subject, relation and object in the implicit triple through three independent linear transformation layers to obtain the differential features of each implicit triple; The structured semantic association information acquisition unit is used to calculate the relationship between the subject, relation and object in the differential features of each implicit triple through a self-attention mechanism, so as to obtain the structured semantic association information of the implicit triple. The implicit triple interaction information acquisition unit is used to obtain the implicit triple interaction information of each implicit triple based on the structured semantic association information and the semantic features of the implicit triple.

[0129] Optionally, the explicit triplet interaction information generation module 11 includes: The explicit semantic feature acquisition unit is used to semantically encode each explicit triple to obtain the semantic features of each explicit triple; The explicit position feature acquisition unit is used to encode the position of the bounding box information corresponding to the subject, action and object in each explicit triple to obtain the position feature of each explicit triple. The splicing feature acquisition unit is used to splice the semantic features and positional features of each explicit triple to obtain the splicing features of each explicit triple. The explicit triplet interaction information acquisition unit is used to input the splicing features related to the subject and object into the first multilayer perceptron and the splicing features related to the action into the second multilayer perceptron for each explicit triplet, so as to obtain the explicit triplet interaction information for each explicit triplet.

[0130] The image generation device oriented towards entities and interaction constraints in this application embodiment can be a device, or it can be a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0131] The entity-oriented and interaction-constrained image generation device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0132] The entity-oriented and interaction-constraint-based image generation apparatus provided in this application can achieve... Figures 1 to 3 The various processes implemented by the image generation device oriented towards entities and interaction constraints in the method embodiment will not be described again here to avoid repetition.

[0133] Optionally, Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. This application also provides an electronic device; it should be noted that the electronic device in this application includes the mobile electronic device and non-mobile electronic device described above.

[0134] The electronic device includes, but is not limited to, components such as: radio frequency unit, network module, audio output unit, input unit, sensor, display unit, user input unit, interface unit, memory, and processor.

[0135] Those skilled in the art will understand that electronic devices may also include power supplies (such as batteries) that supply power to various components. The power supply may be connected to the processor logic through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0136] As an example, such as Figure 5 As shown, the electronic device 600 includes a memory 610 and a processor 620. The memory 610 and the processor 620 are connected via a bus for communication. The memory 610 stores a computer program that can run on the processor 620 to implement the steps in the entity-oriented and interaction-constrained image generation method disclosed in the above embodiments of this application.

[0137] As the apparatus is basically similar to the method embodiment, it is described in a relatively simple way. For relevant details, please refer to the description of the method embodiment.

[0138] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0139] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects.

[0140] Furthermore, this application embodiment also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described entity-oriented and interaction-constrained image generation method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0141] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0142] This application also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described entity-oriented and interaction-constrained image generation method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0143] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0144] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0146] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image generation method oriented towards entities and interaction constraints, characterized in that, The method includes: Based on the explicit triples contained in the target text prompt, generate explicit triple interaction information. The explicit triples are the triples explicitly shown in the target text prompt, and the triples include subject, action, and object. Using a large language model, the implicit triples contained in the target text prompt are extracted, and implicit triple interaction information is generated based on the implicit triples. The implicit triples are triple relationships that are not explicitly shown in the target text prompt. In the denoising process of the diffusion model, intermediate image features are obtained based on the explicit triplet interaction information and the implicit triplet interaction information and the current image features in the current denoising step. The entity control network obtains the current image features for the next denoising process based on the entity semantic features and the intermediate image features. The above process is repeated until the target image is obtained.

2. The image generation method based on entities and interaction constraints according to claim 1, characterized in that, The step of obtaining the current image features in the next denoising process based on the entity semantic features and the intermediate image features includes: Model the entity semantic features and the intermediate image features to generate an entity soft attention mask. The entity semantic features are the semantic features corresponding to the subject and the object contained in the explicit triple. The entity localization image features are obtained by locating the subject and object contained in the explicit triples in the intermediate image features using the entity soft attention mask. The entity localization image is an intermediate image feature carrying the localization information corresponding to the subject and object contained in the explicit triples. The entity localization image features are enhanced by a multi-scale convolutional network to obtain multi-scale entity detail features; The multi-scale entity detail features and the intermediate image features are input into the cross-attention layer of the diffusion model, and the current image features in the next denoising process are obtained based on the output residual of the cross-attention layer and the intermediate image features.

3. The image generation method based on entities and interaction constraints according to claim 1, characterized in that, The method further includes: Extract multiple actions from the explicit triples contained in the target text prompt, and semantically encode the multiple actions to obtain a set of action semantic features; The set of action semantic features is clustered to obtain multiple semantic clusters, and each semantic cluster contains multiple action semantic features with similar semantics; In the global offset direction, each action semantic feature is offset along the direction of the cluster center of the semantic cluster to which the action semantic feature belongs, to obtain the first offset action semantic feature; In the local offset direction, each action semantic feature is offset according to the unit vector direction of the cluster center of the semantic cluster to which the action semantic feature belongs, to obtain the second offset action semantic feature; Based on multiple first offset action semantic features and multiple second offset action semantic features corresponding to multiple action semantic features, interactive action clustering offset information is generated. In the denoising process of the diffusion model, intermediate image features are obtained based on the explicit triplet interaction information and the implicit triplet interaction information, along with the current image features in the current denoising step. These features include: In the denoising process of the diffusion model, intermediate image features are obtained based on the explicit triplet interaction information, the implicit triplet interaction information, the interaction action clustering offset information, and the current image features in the current step of the denoising process.

4. The image generation method based on entities and interaction constraints according to claim 3, characterized in that, In the denoising process of the diffusion model, intermediate image features are obtained based on the explicit triplet interaction information, the implicit triplet interaction information, the interaction action clustering offset information, and the current image features in the current denoising step, including: The explicit triplet interaction information, the implicit triplet interaction information, and the interaction action clustering offset information are concatenated with the current image features in the current step of the denoising process to obtain a joint input representation; The joint input representation is mapped to Q vector, K vector, and V vector through a linear transformation; By using an interactive scaling factor, the calculation results of the Q vector, the K vector, and the V vector are magnified during the attention calculation process to obtain intermediate image features.

5. The image generation method based on entities and interaction constraints according to claim 3, characterized in that, The method further includes: In the first denoising stage of the diffusion model, when obtaining the current image features in the next denoising process based on the current image features in the current denoising process, the explicit triplet interaction information, the implicit triplet interaction information, or the interaction action clustering offset information are not introduced. In the denoising process of the diffusion model, intermediate image features are obtained based on the explicit triplet interaction information, the implicit triplet interaction information, the interaction action clustering offset information, and the current image features in the current denoising step, including: In the second denoising stage of the diffusion model, intermediate image features are obtained based on the explicit triplet interaction information and the implicit triplet interaction information and the current image features in the current denoising process. In the third denoising stage of the diffusion model, intermediate image features are obtained based on the implicit triplet interaction information, the interaction action clustering offset information, and the current image features in the current denoising process. The first denoising stage precedes the second denoising stage, and the second denoising stage precedes the third denoising stage.

6. The image generation method based on entities and interaction constraints according to claim 1, characterized in that, The step of generating implicit triplet interaction information based on the implicit triplet includes: Semantic encoding is performed on each implicit triple to obtain the semantic features of each implicit triple; Differentiated feature mapping is performed on the subject, relation, and object in the implicit triple through three independent linear transformation layers to obtain the differentiated features of each implicit triple; By using a self-attention mechanism, the relationships between the subject, relation, and object in the differential features of each implicit triple are calculated to obtain the structured semantic association information of the implicit triple; Based on the structured semantic association information and the semantic features of the implicit triples, the implicit triple interaction information of each implicit triple is obtained.

7. The image generation method based on entities and interaction constraints according to claim 1, characterized in that, The step of generating explicit triplet interaction information based on the explicit triplets contained in the target text prompt includes: Semantic encoding is performed on each explicit triple to obtain the semantic features of each explicit triple; Position encoding is performed on the bounding box information corresponding to the subject, action, and object in each explicit triple to obtain the position features of each explicit triple; For each explicit triple, the semantic features and positional features of the explicit triple are concatenated to obtain the concatenated features of each explicit triple. For each explicit triple, the splicing features related to the subject and object are input into the first multilayer perceptron, and the splicing features related to the action are input into the second multilayer perceptron to obtain the explicit triple interaction information for each explicit triple.

8. An image generation apparatus oriented towards entities and interaction constraints, characterized in that, The device includes: The explicit triplet interaction information generation module is used to generate explicit triplet interaction information based on the explicit triplets contained in the target text prompt. The explicit triplet is the triplet explicitly shown in the target text prompt, and the triplet includes a subject, an action, and an object. The implicit triple interaction information generation module is used to extract the implicit triples contained in the target text prompt through a large language model, and generate implicit triple interaction information based on the implicit triples, wherein the implicit triples are triple relationships not explicitly shown in the target text prompt. The intermediate image feature generation module is used to obtain intermediate image features in the denoising process of the diffusion model based on the explicit triplet interaction information and the implicit triplet interaction information and the current image features in the current step of the denoising process. The target image generation module is used to obtain the current image features in the next denoising process through the entity control network, based on the entity semantic features and the intermediate image features, and repeat the above process until the target image is obtained.

9. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the entity-oriented and interaction-constrained image generation method as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the entity-oriented and interaction-constrained image generation method as described in any one of claims 1-7.

Images