Method and device for generating human-object interaction image based on structured semantic enhancement

Abstract

The application provides a human-object interaction image generation method and device based on structured semantic enhancement. The method comprises: obtaining human-object interaction triple description and boundary box conditions; based on a semantic library pre-constructed by a large language model, expanding the triple description into multi-slot structured semantic description and encoding; performing intra-slot semantic aggregation on each slot to compress into a single slot semantic vector; performing cross-slot relationship modeling on the slot semantic vector to output a global action semantic representation through self-attention fusion; performing feature modulation on the subject and object semantic representations based on the global action semantics; fusing the modulated representations with spatial position features to form a conditional representation vector, injecting the diffusion model denoising network to generate an interaction image. Through coarse-grained semantic fine-grained structured reconstruction and hierarchical coding, the application significantly improves the semantic fidelity and interaction controllability of generated images in complex interaction scenarios.

Description

Technical Field

[0001] This invention relates to the field of image generation technology, and in particular to a method and apparatus for generating human-object interaction images based on structured semantic enhancement. Background Technology

[0002] With diffusion models demonstrating powerful capabilities in text-to-image generation tasks, achieving finely controlled, conditional generation has become a hot research topic. Particularly in human-object interaction image generation tasks, models are required to simultaneously respond to external constraints such as text prompts and spatial coordinate boxes, generating images that simultaneously satisfy reasonable spatial layout, accurate action semantics, and consistent interaction relationships.

[0003] Existing methods primarily rely on the original human-object interaction triples as semantic control conditions and introduce structural information such as bounding boxes into the diffusion model to achieve spatial control. However, traditional human-object interaction triples typically only contain highly compressed semantic labels such as subject, action, and object, resulting in coarse-grained representations that struggle to depict fine-grained visual semantic information closely related to interactive actions, such as body posture, contact methods, spatial relationships, and behavioral intentions. Simultaneously, most existing methods treat interactive action semantics as a single text-based injection into the generation model, lacking structured organization and hierarchical modeling of multi-dimensional action semantic information. This approach makes the model prone to insufficient action response, ambiguous interaction relationships, or semantic drift when faced with semantically similar but visually detailed interactive actions, thus limiting the action discriminability and semantic consistency of the generated images.

[0004] Therefore, there is an urgent need to provide a method and apparatus for generating human-object interaction images based on structured semantic enhancement. Summary of the Invention

[0005] The purpose of this application is to provide a method and apparatus for generating human-object interaction images based on structured semantic enhancement, thereby solving the aforementioned technical problems. This purpose is achieved through the following technical solution: The first aspect of this application proposes a method for generating human-object interaction images based on structured semantic enhancement, including: Obtain the human-object interaction triplet description and bounding box conditions of human-object interaction images; Based on the semantic library pre-built by the large language model, the human-object interaction triplet description is extended into a multi-slot structured semantic description, and the text of each slot is encoded into a semantic embedding vector through a text encoder. Perform intra-slot semantic aggregation on the semantic embedding vector of each slot, and compress the token sequence of each slot into a single slot semantic vector; Cross-slot relationship modeling is performed on the slot semantic vector, and multi-slot semantic information is fused through a self-attention mechanism to output a global action semantic representation; Obtain the subject semantic representation and the object semantic representation from the slot semantic vector; Based on the global action semantic representation, modulation parameters are generated, and feature modulation is performed on the subject semantic representation and the object semantic representation respectively to obtain the modulated subject semantic representation and object semantic representation; The modulated subject semantic representation, object semantic representation, and global action semantic representation are fused with their corresponding spatial location features to form a conditional representation vector, which is then injected into the denoising network of the diffusion model to generate an interactive image.

[0006] Preferably, the multi-slot structured semantic description includes the following slot types: subject information slot, action information slot, object information slot, posture information slot, contact method information slot, spatial relationship information slot, behavioral intention information slot, and time duration feature information slot.

[0007] Preferably, the semantic library pre-built based on the large language model extends the human-object interaction triplet description into a multi-slot structured semantic description, including: For all action-object combinations in the dataset, a fine-grained, multi-slot structured semantic description is generated offline using a large language model and stored to construct the semantic library. During the training and inference phases, action-object pairs are extracted based on the input human-object interaction triplet description, and indexed queries are performed in the semantic database to obtain the corresponding multi-slot structured semantic description.

[0008] Preferably, the intra-slot semantic aggregation and the cross-slot relationship modeling constitute a hierarchical coding structure: The first stage is the semantic aggregation within the slot, which realizes semantic compression from the word level to the slot level, and aggregates multiple tokens in each slot into a slot semantic vector. The cross-slot relationship modeling is the second stage, which realizes semantic fusion from the slot level to the interaction level. Global self-attention encoding is performed on all slot semantic vectors to output the global action semantic representation that integrates multi-slot information.

[0009] Preferably, the in-slot semantic aggregation includes: A learnable query vector is introduced, and the contribution of each token in each slot to the learnable query vector is calculated through a multi-head attention mechanism. Key semantic information is adaptively aggregated, and all token sequences in each slot are compressed into a slot semantic vector.

[0010] Preferably, the step of generating modulation parameters based on the global action semantic representation and performing feature modulation on the subject semantic representation and the object semantic representation respectively includes: The global action semantic representation is used to generate scaling and offset parameters through linear mapping; Using the scaling and offset parameters, residual modulation based on the FiLM mechanism is performed on the subject semantic representation and the object semantic representation, respectively, so that the action semantics have a reverse influence on the feature representation of the subject and the object.

[0011] Preferably, the training process of the denoising network includes: During the diffusion training process, global semantic consistency loss, spatial coordinate regression loss, and action discrimination contrast loss are jointly applied to perform multi-layer control optimization on the denoising network; the semantic alignment between the generated image and the text description is constrained by the global semantic consistency loss. The spatial geometric structure of human-object interaction in the generated image is constrained by spatial coordinate regression loss. By using action discrimination contrast loss, the model's ability to distinguish fine-grained semantics of specific interactive actions is enhanced.

[0012] Preferably, the calculation of the action discrimination contrast loss includes: Using a cross-attention mechanism, the global action semantic representation is used as the query vector to extract action-related region features from the intermediate layer visual features of the denoising network; The action-related region features are compared and learned with the corresponding action text semantic features to maximize the similarity of correct action pairs and minimize semantic confusion between different actions.

[0013] Preferably, the calculation of the spatial coordinate regression loss includes: The bounding box coordinates of the subject features, object features and action features are predicted by the regression head, respectively, to obtain the predicted bounding box of the person, the predicted bounding box of the object and the predicted bounding box of the middle region of the action. The predicted bounding boxes of the person, the predicted bounding boxes of the object, and the predicted bounding boxes of the middle region of the action are respectively subjected to L1 loss supervision with the ground truth bounding boxes.

[0014] The second aspect of this application proposes a human-object interaction image generation device based on structured semantic enhancement, comprising: A hierarchical action semantic encoding module is used to obtain the human-object interaction triplet description and bounding box conditions of human-object interaction images; based on the semantic library pre-built by the large language model, the human-object interaction triplet description is extended into a multi-slot structured semantic description, and the text of each slot is encoded into a semantic embedding vector by a text encoder; the semantic embedding vector of each slot is subjected to in-slot semantic aggregation, and the token sequence of each slot is compressed into a single slot semantic vector; The fusion module is used to perform cross-slot relationship modeling on the slot semantic vector, fuse multi-slot semantic information through a self-attention mechanism, and output a global action semantic representation. A diffusion model denoising network is used to obtain subject semantic representation and object semantic representation from the slot semantic vector; generate modulation parameters based on the global action semantic representation, and perform feature modulation on the subject semantic representation and the object semantic representation respectively to obtain modulated subject semantic representation and object semantic representation; fuse the modulated subject semantic representation, object representation and global action semantic representation with the corresponding spatial location features to form a conditional representation vector, and inject it into the diffusion model denoising network to generate an interactive image.

[0015] This invention provides a method and apparatus for generating human-object interaction images based on structured semantic enhancement. First, by introducing a fine-grained action semantic decomposition mechanism based on a large language model, the traditional highly compressed interaction triplet labels are expanded into a structured semantic description encompassing multiple information such as subject, action, object, posture, contact method, spatial relationship, interaction intent, and temporal attributes. This addresses the lack of detailed information in coarse-grained semantic expressions at the data source, enabling the model to perceive subtle visual differences between different interactive actions. Second, the constructed hierarchical action semantic encoding module adopts a two-stage Transformer architecture. First, learnable attention pooling is used to achieve adaptive key information extraction and compression from the word level to the slot level. Then, a cross-slot self-attention mechanism captures the inherent dependencies between different semantic dimensions and completes global information fusion from the slot level to the interaction level. This progressive modeling approach retains local fine-grained features while forming a high-level semantic representation with global interaction understanding capabilities. Furthermore, the action semantic modulation strategy based on the FiLM mechanism enables the global action representation to dynamically adjust the feature representations of the subject and objects in the form of conditional signals, strengthening the logical consistency between the character's visual performance and action execution. Finally, by constructing a multi-layered control and optimization system that includes global image-text semantic alignment, spatial geometric structure regression, and fine-grained action discrimination comparison, the generation process is jointly constrained from multiple perspectives such as perceptual quality, spatial layout, and semantic distinguishability, thereby synergistically improving the overall fidelity, interactive rationality, and semantic consistency of the generated images. Attached Figure Description

[0016] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Throughout the drawings, the same reference numerals denote the same components. Obviously, the drawings described below are merely some embodiments described in this application, and those skilled in the art can obtain other drawings based on these drawings.

[0017] Figure 1 This is a schematic diagram of the overall architecture of a human-object interaction image generation method based on structured semantic enhancement, provided in an embodiment of this application.

[0018] Figure 2 This is a schematic diagram of the hierarchical action semantic encoding module provided in an embodiment of this application.

[0019] Figure 3 This diagram illustrates the comparison of experimental results between the human-object interaction image generation method based on structured semantic enhancement provided in this application embodiment and other models.

[0020] Figure 4 This is a schematic diagram illustrating the correspondence between the structured semantic description and the generated image provided in the embodiments of this application.

[0021] Figure 5 This is a flowchart illustrating a method for generating human-object interaction images based on structured semantic enhancement, as provided in an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. It should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application. Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in this application.

[0023] First, the application scenarios of this application will be explained in further detail.

[0024] Conditionally controllable image generation: Early image generation research primarily focused on unconditional visual content synthesis, aiming to learn the distribution of real image data and generate highly realistic images. As diffusion models have demonstrated significant advantages in generation quality, training stability, and semantic consistency, research has gradually shifted towards conditionally controlled image generation. This involves introducing external control signals during the generation process, enabling the model to more precisely regulate object location, spatial layout, and local structure. To achieve this goal, various diffusion generation methods based on structural conditions such as bounding boxes, pose, depth maps, and semantic segmentation maps have been proposed.

[0025] Representative works such as ControlNet significantly enhance the model's responsiveness to external control signals by introducing parallel trainable branches outside the pre-trained diffusion model backbone and explicitly injecting structural conditions such as pose, edge, and depth into the denoising process. GLIGEN, on the other hand, achieves good results in target-level control by jointly modeling text descriptions with spatial constraints such as bounding boxes, enabling the model to generate semantically corresponding target content in a specified region. Building on this, methods such as InstanceDiffusion and MIGC further explore multi-instance generation scenarios, improving target localization accuracy and relationship expression capabilities under complex layout conditions by constructing independent instance condition representations or explicitly modeling relationships between instances.

[0026] These methods significantly improve the ability of diffusion models in spatial layout control and multi-instance generation, laying the foundation for controllable image generation in complex scenes. However, most existing research focuses on object position constraints, structural alignment, and instance combination, paying insufficient attention to higher-level information such as semantic differences, contact relationships, and action intentions in interactive actions. When generation tasks further evolve from multi-object layout to multi-object interaction, relying solely on structural conditions or single-layer text prompts is no longer sufficient to meet the requirements of semantic accuracy and interaction rationality in complex interactive scenarios.

[0027] Human-object interaction image generation: Human-object interaction image generation requires the model to generate images that simultaneously contain people, objects, and their interactive behaviors, given text prompts and structural constraints. Compared with general conditionally controllable image generation tasks, this task not only requires the model to correctly control the positional relationship between people and objects, but also requires the generated results to accurately express the action relationship and interaction logic between them. Therefore, it places higher demands on the multi-condition collaborative modeling capabilities of the generative model.

[0028] To address this task, related technologies have achieved controllable generation of interactive actions by explicitly introducing person-object-action triples into the diffusion model, providing a representative research paradigm for HOI image generation. However, this method mainly relies on the original HOI triples for conditional modeling, while commonly used HOI triples typically only contain compressed semantics such as subject, action, and object, resulting in coarse-grained expression that is difficult to fully describe fine-grained action information such as posture, contact method, spatial relationship, and interaction intention. Therefore, in complex action scenarios, such methods are prone to problems such as insufficient action response, ambiguous interaction relationships, or semantic shift.

[0029] The relevant technologies are still limited to action text enhancement, conditional decoupling, or high-level semantic supplementation, lacking systematic and structured modeling of the fine-grained semantic components of complex interactive actions. Especially in action scenarios with similar semantics but different interaction details, the models often struggle to simultaneously ensure action distinguishability, semantic consistency, and interaction rationality.

[0030] This application proposes a method and apparatus for generating human-object interaction images based on structured semantic enhancement. Without altering the core structure of the latent space diffusion model, it performs fine-grained semantic reconstruction of action text. First, a fine-grained action semantic decomposition module is designed, using a large language model to expand the original triples into multi-slot semantic descriptions containing information such as subject, action, object, posture, and contact method. Second, a hierarchical action semantic encoding module is constructed, using a two-stage Transformer to aggregate and model multi-slot semantics, enhancing the model's understanding of complex interaction relationships. Finally, a multi-layered control optimization strategy is combined to jointly optimize the model from three levels: spatial constraints, cross-modal semantic consistency, and action discrimination, improving the semantic fidelity and interactive controllability of the generated images. The method is described first, followed by the evaluation and apparatus.

[0031] Example 1: See Figure 5 This embodiment provides a method for generating human-object interaction images based on structured semantic enhancement, including: S100, Obtain the human-object interaction triplet description and bounding box conditions of the human-object interaction image; The Human-Object Interaction (HOI) triple description typically consists of three terms: subject, action, and object, such as "person-riding-motorcycle" or "person-feeding-giraffe," providing a coarse-grained summary of the interaction content. Bounding box conditions specify the spatial areas that people and objects should occupy in the image, usually given in normalized coordinates, such as the left bounding box coordinates of the person and the right bounding box coordinates of the object. These conditions serve as the basis for subsequent semantic expansion and spatial constraints, acting as external control signals for the model to generate interactive images.

[0032] S200, based on a semantic library pre-built from a large language model, extends the human-object interaction triplet description into a multi-slot structured semantic description, and encodes the text of each slot into a semantic embedding vector through a text encoder; Specifically, before training and use, all possible action-object combinations in the dataset have been processed offline using a large language model, generating fine-grained text descriptions covering multiple semantic dimensions for each action-object pair, forming a semantic library. In step S200, action-object pairs are extracted based on the input triplet descriptions, and the corresponding multi-slot structured semantic descriptions are obtained directly from the semantic library via index query. These slots can include subject information, action information, object information, posture information, contact method information, spatial relationship information, behavioral intent information, etc., thus expanding the triplet information, which originally contained only a few words, into a text set containing multi-dimensional interaction details. Subsequently, the content of each slot in this text set is converted into a fixed-length semantic embedding vector sequence by a pre-trained text encoder (e.g., a CLIP-based text encoder), with each slot corresponding to a set of token sequences. This process elevates coarse-grained category labels into structured semantic signals with rich visual cues, laying the data foundation for subsequent hierarchical semantic modeling.

[0033] S300 performs intra-slot semantic aggregation on the semantic embedding vector of each slot, compressing the token sequence of each slot into a single slot semantic vector; Since the text description of each slot generates more tokens after encoding, directly retaining all tokens not only introduces redundancy but also increases the complexity of subsequent cross-slot modeling. Therefore, this step performs semantic aggregation independently for each slot. This aggregation can be achieved through a learnable attention pooling mechanism or a pooling strategy with weighted allocation. The goal is to adaptively select the most representative key semantic information from the slot description, compressing the token sequence within a slot into a compact slot semantic vector. For example, for the "posture" slot, its token sequence contains details about standing, bending over, and reaching out; the aggregated vector can highlight the posture features most relevant to the current interaction action. This step completes the abstraction from the word level to the slot level, refining the information of each semantic dimension, which helps enhance the model's understanding of fine-grained descriptions and reduce noise interference.

[0034] S400, perform cross-slot relationship modeling on the slot semantic vector, fuse multi-slot semantic information through a self-attention mechanism, and output a global action semantic representation; Specifically, this step projects all slot semantic vectors onto the same dimension and can introduce global convergence markers such as [CLS]. A self-attention-based encoder then models the global relationships between all slot vectors. This self-attention mechanism allows the representation of each slot to dynamically adjust based on information from other slots. For example, when perceiving the action of "riding," it automatically strengthens posture and contact details such as "leaning forward," forming a close semantic association. Finally, it outputs a global action semantic representation representing the core semantics of the entire interaction. This vector integrates information from posture changes, contact methods, spatial topology, and behavioral intentions, achieving semantic fusion from the slot level to the interaction level.

[0035] S500, obtain the subject semantic representation and object semantic representation from the slot semantic vector; generate modulation parameters based on the global action semantic representation, and perform feature modulation on the subject semantic representation and the object semantic representation respectively to obtain the modulated subject semantic representation and object semantic representation; It can be assumed that the semantic representation of the subject is usually taken from the vector after aggregating the subject information slots, and the semantic representation of the object is taken from the vector after aggregating the object information slots. However, the visual representation of the subject and the object is highly dependent on specific actions. For example, the same motorcycle should reflect the close connection between the rider and the motorcycle when "riding," and should reflect the sense of force when "pushing." To address this, using the global action semantic representation obtained from S400, two types of modulation parameters, scaling and offset, are generated through linear mapping. These parameters are then applied to the subject semantic representation and the object semantic representation using residual adjustment based on feature linear modulation. This modulation does not require re-extracting features but directly intervenes in the direction of feature expression through action semantics, causing the character features to change dynamically with the action semantics, thereby strengthening the consistency between action execution and character performance.

[0036] S600, the modulated subject semantic representation, object semantic representation and global action semantic representation are fused with the corresponding spatial location features to form a conditional representation vector, which is then injected into the denoising network of the diffusion model to generate an interactive image.

[0037] Spatial location features are typically high-dimensional vectors obtained by mapping bounding box coordinates through Fourier position encoding or fully connected layers, corresponding to the person region, object region, and intermediate regions where interactive actions may occur, respectively. Fusion can be achieved through element-wise addition, concatenation, or weighted summation to form a unified conditional representation vector. For example, injecting the conditional representation vector into the U-Net denoising network of a diffusion model allows the model to be guided to generate visual content conforming to structured semantic descriptions at specified spatial locations during each denoising step through cross-attention or gated self-attention mechanisms. Because the semantic vectors are rich in action details and the spatial features indicate layout relationships, the model can accurately generate images with consistent interaction and reasonable spatial arrangement.

[0038] To address the issue of coarse semantic granularity, the knowledge reserves of a large language model are leveraged to automatically expand single action labels into multiple slots. Each slot independently carries a type of fine-grained visual semantics, enabling the model to deal with rich, visually descriptive, and structured text rather than dry labels, thus enhancing the expressive power of semantic input.

[0039] To address the lack of structured organization and hierarchical modeling, a two-level hierarchical processing approach was adopted to achieve structured organization. First, intra-slot semantic aggregation compresses the token sequence within each slot into a single slot semantic vector. This is equivalent to refining information within each fine-grained dimension (such as posture and contact), forming a compact semantic representation for that dimension, avoiding noise and redundancy caused by directly injecting long text tokens. Second, cross-slot relationship modeling uses a self-attention mechanism to interact among all slot semantic vectors, outputting a global action semantic representation. This allows different slots to calibrate with each other, forming a holistic and structured understanding of interactive actions. Third, to address insufficient action response and ambiguous interaction relationships, modulation parameters are generated using global action semantics to modulate the semantic representations of the subject and object separately. Specifically, semantic decoupling explicitly separates the semantic representations of the subject, object, and global action, avoiding their overlap. Fourth, action conditional modulation involves generating modulation parameters from the global action semantics, rather than simply concatenating them, to conditionally transform the semantic features of the subject and object. After modulation, the images are fused with their respective spatial location features, achieving alignment between semantic content and spatial layout. This ensures that the generated interactive images satisfy bounding box constraints while maintaining semantic consistency in the interactive relationships.

[0040] In summary, the technical solution provided in this embodiment achieves a seamless transformation from coarse-grained triples to fine-grained structured semantic signals. Multi-slot semantic extension fills in the gaps in traditional triples, such as gesture, contact, and intent—key visual information. Hierarchical aggregation and cross-slot modeling ensure that semantics across dimensions mutually reinforce each other rather than being expressed in isolation. Action semantic modulation ensures the matching of character features with interaction logic, while spatial feature fusion lays out semantics in precise geometric positions. The conditional signals constructed in this way far surpass simple text triples in fidelity and discriminability, thereby greatly improving the semantic accuracy and spatial rationality of generated images in complex human-object interaction scenarios.

[0041] In an exemplary embodiment, the multi-slot structured semantic description includes the following slot types: subject information slot, action information slot, object information slot, posture information slot, contact method information slot, spatial relationship information slot, behavioral intention information slot, and time duration feature information slot.

[0042] Thus, by decomposing interactive actions into multiple semantics, covering key elements such as action core, physical contact, spatial topology, and behavioral context, the comprehensiveness and structure of fine-grained semantic description are ensured, providing rich and highly distinguishable basic information for subsequent semantic encoding and spatial fusion.

[0043] In an exemplary embodiment, the semantic library pre-built based on a large language model extends the human-object interaction triplet description into a multi-slot structured semantic description, including: For all action-object combinations in the dataset, a fine-grained, multi-slot structured semantic description is generated offline using a large language model and stored to construct the semantic library. During the training and inference phases, action-object pairs are extracted based on the input human-object interaction triplet description, and indexed queries are performed in the semantic database to obtain the corresponding multi-slot structured semantic description.

[0044] Therefore, the time-consuming semantic generation process is placed in the offline preprocessing stage, and only efficient index query operations are needed when executing online. This significantly improves the richness of semantic description while avoiding the negative impact on model training or inference speed.

[0045] In an exemplary embodiment, the intra-slot semantic aggregation and the cross-slot relationship modeling constitute a hierarchical coding structure: The first stage is the semantic aggregation within the slot, which realizes semantic compression from the word level to the slot level, and aggregates multiple tokens in each slot into a slot semantic vector. The cross-slot relationship modeling is the second stage, which realizes semantic fusion from the slot level to the interaction level. Global self-attention encoding is performed on all slot semantic vectors to output the global action semantic representation that integrates multi-slot information.

[0046] Therefore, by first extracting key information within each semantic dimension and then modeling interaction dependencies across dimensions, the final action semantic representation can both retain local detailed features and capture global interaction consistency.

[0047] In one exemplary embodiment, the in-slot semantic aggregation includes: A learnable query vector is introduced, and the contribution of each token in each slot to the learnable query vector is calculated through a multi-head attention mechanism. Key semantic information is adaptively aggregated, and all token sequences in each slot are compressed into a slot semantic vector.

[0048] Thus, unlike simple mean pooling, the attention pooling mechanism based on learnable query vectors can automatically focus on the core words in the slot description, effectively filter out redundant noise, and improve the representation quality of slot-level semantic vectors.

[0049] In an exemplary embodiment, the step of generating modulation parameters based on the global action semantic representation and performing feature modulation on the subject semantic representation and object semantic representation respectively includes: The global action semantic representation is used to generate scaling and offset parameters through linear mapping; Using the scaling and offset parameters, residual modulation based on the FiLM mechanism is performed on the subject semantic representation and the object semantic representation, respectively, so that the action semantics have a reverse influence on the feature representation of the subject and the object.

[0050] This allows action semantics (such as "riding" or "pushing") to finely adjust the feature representation of interactive subjects and objects in the form of conditional signals, so that the same object exhibits feature changes that conform to the interaction logic under different actions, thereby strengthening the dynamic consistency between character features and action semantics.

[0051] In one exemplary embodiment, the training process of the denoising network includes: During the diffusion training process, global semantic consistency loss, spatial coordinate regression loss, and action discrimination contrast loss are jointly applied to perform multi-layer control optimization on the denoising network; the semantic alignment between the generated image and the text description is constrained by the global semantic consistency loss. The spatial geometric structure of human-object interaction in the generated image is constrained by spatial coordinate regression loss. By using action discrimination contrast loss, the model's ability to distinguish fine-grained semantics of specific interactive actions is enhanced.

[0052] Therefore, this multi-layer control optimization strategy jointly constrains the diffusion generation process from three complementary perspectives: semantic, spatial, and action. This overcomes the limitation of conventional noise prediction loss, which only focuses on pixel-level distribution fitting and ignores high-level semantic alignment, and systematically improves the overall quality of the generated image.

[0053] In an exemplary embodiment, the calculation of the action discrimination contrast loss includes: Using a cross-attention mechanism, the global action semantic representation is used as the query vector to extract action-related region features from the intermediate layer visual features of the denoising network; The action-related region features are compared and learned with the corresponding action text semantic features to maximize the similarity of correct action pairs and minimize semantic confusion between different actions.

[0054] Therefore, through the action semantic-driven attention query mechanism, the model can actively focus on the spatial region in the image that is most relevant to the interactive action, avoiding interference from background information. This allows for more accurate enhancement of the discriminative power of action features through comparative learning, effectively alleviating the semantic ambiguity problem of similar actions.

[0055] In an exemplary embodiment, the calculation of the spatial coordinate regression loss includes: The bounding box coordinates of the subject features, object features and action features are predicted by the regression head, respectively, to obtain the predicted bounding box of the person, the predicted bounding box of the object and the predicted bounding box of the middle region of the action. The predicted bounding boxes of the person, the predicted bounding boxes of the object, and the predicted bounding boxes of the middle region of the action are respectively subjected to L1 loss supervision with the ground truth bounding boxes.

[0056] Therefore, this loss directly establishes a mapping relationship between semantic encoding and geometric space, forcing the semantic features to explicitly retain the spatial location information of the subject, object and their interaction area, thereby ensuring the accuracy of the spatial structure and geometric rationality of the interaction relationship in the generated image.

[0057] As an example, a method for generating human-object interactions based on structured semantic enhancement is proposed, the overall architecture of which is as follows: Figure 1 As shown, while maintaining the core structure of the diffusion model, the focus is on reconstructing the semantic input format to enhance the model's ability to express, distinguish, and control complex action semantics. Specifically, firstly, the interaction semantics are expanded into a multi-dimensional structure through a fine-grained action semantic decomposition module (FSD); then, a hierarchical action semantic encoding module (HSE) is used to aggregate multi-dimensional semantics and model action information; finally, a multi-level control optimization strategy (MCO) is combined to jointly constrain the model from three levels: global semantic consistency, spatial structure, and action distinction, thereby improving the diffusion model's ability to understand and generate complex interaction semantics.

[0058] Its network structure: This section employs a latent space diffusion modeling strategy, where the model receives data including real images during the training phase. Text description HOI triplet description and the bounding box set of people and objects. Among them, HOI triples are used to retrieve fine-grained semantic descriptions generated by a pre-built large language model, serving as a source of structured semantic input; the bounding boxes adopt normalized coordinate forms. These represent the coordinates of the top-left and bottom-right corners of the bounding box, respectively, used to depict the spatial layout relationship between people and objects. The goal of the model is to learn the image distribution in the latent space and generate image results that satisfy specified interaction relationships under the joint constraints of semantic and spatial conditions.

[0059] The input image is first encoded by a pre-trained variational autoencoder (VAE). Representations of latent variables are obtained in the latent space. ,in During the training phase Add random Gaussian noise Obtaining noisy latent variables As the main input to the diffusion denoising network, and the time step The embedded data is fed into the U-Net denoising network for noise prediction learning. The network learns by progressively predicting noise. To recover the latent variable representation To approximate the real data distribution. Finally, it is decoded by a VAE decoder. The recovered latent variables are mapped back to pixel space to generate an image. .

[0060] In terms of condition construction, the action semantic library is first pre-built through the FSD module. Used to retrieve sets of fine-grained action descriptions generated offline from large language models. For each pair of object-action combinations The semantic library stores a set of fine-grained descriptions generated offline by a large language model. In the input HOI triples... Subsequently, the retrieved fine-grained semantic description And mapped to token-level semantic embedding vectors through a pre-trained CLIP text encoder. Subsequently, the HSE module is used to refine the multi-dimensional fine-grained description vector. Cross-sentence modeling and semantic compression are performed to form a unified structured semantic representation. The bounding box coordinates are then mapped to high-dimensional spatial features through Fourier position encoding. After fusing fine-grained semantic features with spatial location features, a final conditional representation vector is formed, which is then input into the U-Net network of the diffusion model for the model to learn.

[0061] In the diffusion denoising stage, U-Net uses noisy latent variables and time-step embeddings as basic inputs, and fuses structured semantic conditions and spatial conditions through a gated self-attention mechanism during feature propagation, thereby achieving collaborative modeling of action semantics and geometric structure. This application does not change the overall structure within the diffusion backbone network, but rather enhances the expressive quality of the input control conditions themselves, enabling the model to achieve stronger interactive semantic understanding under the original conditional fusion mechanism. After conditional guidance and multi-layer feature propagation, the network outputs noise prediction results, which are used to complete denoising learning in the latent space.

[0062] Fine-grained action semantic decomposition module: In human-object interaction generation tasks, traditional HOI triples are typically represented as: While this representation can provide basic semantic constraints, it is essentially a discrete category label, making it difficult to characterize the fine-grained differences in actions regarding posture changes, contact methods, spatial topological relationships, and behavioral intentions. To enhance the model's ability to express complex interaction semantics, this application first constructs a fine-grained action semantic decomposition module (FSD), which expands the original HOI combination into a multi-dimensional structured semantic representation, thereby providing richer and more distinguishable conditional information for subsequent diffusion modeling.

[0063] The semantic library of the FSD module is built based on all valid action-object combinations in the HICO-DET dataset. Let all distinct action-object combinations in the dataset... Form a set ,in This represents the number of combinations. For each combination... Construct text prompt template And input the large language model to generate the corresponding structured fine-grained description. The generated results are then reorganized into a set of slots consisting of eight semantic dimensions:

[0064] in Indicates the subject of the action (SUBJECT). Indicates the verb form (ACTION), Indicates the object (OBJECT). Indicates body posture (POSTURE) Indicates the contact method (CONTACT). Indicates spatial relationships (SPACE). Indicates behavioral intent. Represents the duration of time (TIME). All generated results constitute an offline semantic library. The semantic library is pre-built before training; during the training and inference phases, it only performs index queries and does not participate in gradient updates.

[0065] The eight semantic dimensions mentioned above were chosen because human-object interaction in visual generation can be decomposed into five key factors: subject, action core, physical contact, spatial topology, and behavioral context. , , Maintain consistency with the original HOI triple semantic skeleton to ensure stable constraints at the category level; and This directly corresponds to the distribution of human body posture and local contact structures, and is an important factor in distinguishing similar movements such as sitting on, lying on, and learning on. Describing the spatial topological relationships between the subject and objects helps to enhance the consistency of the geometric layout; It provides functional semantic supplements to actions, enhancing the model's ability to abstractly understand behavioral context; This decomposes the continuous features of the action, providing supplementary information for dynamic semantics. Through this structural decomposition, the originally single HOI triple is mapped into a set of multi-dimensional semantic vectors, thereby expanding the conditional expression space.

[0066] During the training and inference phases, given the input HOI triples First, extract action-object pairs. and in semantic library Perform an index query to obtain the corresponding fine-grained description. The text in each slot is then CLIP encoded into a set of text sequences. This provides input for subsequent semantic encoding modules. The overall mapping process can be represented as:

[0067] in, , Indicates that a batch contains Each sample contains a maximum of [number] samples. Group human-object interaction pairs Indicates the number of slots. The length of the CLIP token. For text embedding dimensions.

[0068] Through the fine-grained action semantic decomposition module, the original discrete category labels are expanded into multi-dimensional semantic expressions, laying the foundation for subsequent semantic fusion and diffusion modulation.

[0069] Hierarchical action semantic encoding module: Building upon the fine-grained action semantic decomposition module that obtains multi-slot semantic descriptions, this application further proposes a hierarchical action semantic encoding module (HSE) for structural modeling and cross-role interaction fusion of multi-slot semantic information, such as... Figure 2 As shown, the HSE module uses a two-stage Transformer structure to process the high-dimensional semantic vector output by the FSD module. Achieve progressive abstraction and integration from word level to slot level and then to interaction level.

[0070] Since the text description of each slot typically contains multiple tokens, directly retaining all tokens would introduce redundant information and hinder subsequent cross-slot modeling. Therefore, the first stage employs LearnableSlotPooling to adaptively aggregate the token sequence within each slot. For each fine-grained text description... Introducing a learnable query vector By calculating the contribution of each token to the query through multi-head attention, key semantics are adaptively aggregated to obtain a compressed semantic representation. :

[0071] Subsequently, the output is normalized using LayerNorm to obtain the slot-level semantic representation. Learnable attention pooling compresses the 77 tokens in each slot into a single semantic vector, achieving the first hierarchical modeling from the word level to the slot level. Compared to simple mean pooling, attention pooling can highlight the core words in the description, thus more effectively preserving fine-grained semantic information.

[0072] However, while the first-stage Learnable SlotPooling can yield independent slot-level semantic vectors, explicit relationship modeling between these vectors has not yet been established. In a complete human-object interaction process, there are close dependencies between the subject, action, object, and their fine-grained semantic attributes. Therefore, the second stage designs a cross-slot Transformer to model cross-slot relationships between slot-level semantics. First, a cross-slot Transformer is designed... Projected to model dimension ,get For each pair of interaction actions, a learnable [CLS] tag is added before the slot vector to construct an input sequence. :

[0073] Here, CLS is a learnable semantic vector, serving as the convergence point of global semantics. Subsequently, the CLS token sequence is input into a two-layer Transformer Encoder for cross-slot self-attention modeling, and finally, the vector at the [CLS] position of the output sequence is taken as the global action semantic representation of the interaction pair. :

[0074] The global action semantic representation is denoted as It integrates the semantics of multiple slots, including posture changes, contact methods, spatial (topological) relationships, and behavioral intentions, to depict the core meaning of interaction. To further integrate the semantics of subject, object, and action with spatial location information, the HSE module explicitly separates the semantic representations of action subject and object from the slot representation. Let the 0th slot indices correspond to the subject semantics and the 2nd slot to the object semantics; these are then mapped to the same dimension as the action semantics using linear projection.

[0075]

[0076] in, The projected linear matrix is ​​used to learn the semantic representations of the action subject and the object, which are then separated. .

[0077] However, the roles of subjects and objects in interactions are determined by actions. For example, in the interaction between "a person riding a motorcycle" and "a person pushing a motorcycle," the visual representation of the subject and object should change with the action. To enable action semantics to guide the feature representation of subjects and objects, the HSE module introduces a semantic modulation mechanism based on the FiLM mechanism. Specifically, it modulates global action semantics... Through linear mapping to modulation vector and Then, residual modulation is performed on the semantics of the subject and the object:

[0078]

[0079] in Learnable scale and offset parameters are used to stabilize the modulation process; The small-scale coefficient is used to control the intensity of action injection and avoid excessive interference with the original semantics. This process enables action semantics to modulate the role representation in reverse, thereby enhancing semantic consistency.

[0080] Ultimately, the HSE module obtained three types of semantic vectors: modulated subject representations. Global action representation Modulated object representation This representation needs to be combined with the corresponding spatial location features (i.e., spatial coding features) to form a unified conditional signal. Therefore, each semantic vector is paired with its corresponding spatial coding feature. The data are fed separately into a multilayer perceptron for fusion, and then concatenated into a unified feature vector to obtain the final conditional representation. :

[0081] This vector is then injected into the diffusion model as a condition controlling the generation process of human-object interaction images.

[0082] Through the hierarchical encoding described above, the HSE module transforms the original discrete HOI text description into a continuous semantic representation with internal structural relationships and cross-role dependencies. The first stage achieves word-level to slot-level abstraction, the second stage completes the fusion from slot-level to interaction-level, and the modulation of role semantics further enhances semantic consistency. The final representation retains fine-grained details while possessing global interaction information, providing stable and highly discriminative control conditions for subsequent interaction generation tasks.

[0083] Multi-layer control optimization module section: In the process of generating diffusion-based human-object interaction images, traditional noise prediction loss (such as MSE) only constrains the model's ability to fit the noise distribution at the latent space level, but cannot directly guarantee the consistency between the generated result and the text semantics. Especially after introducing fine-grained action semantic decomposition, relying solely on diffusion reconstruction targets is insufficient to fully constrain the alignment relationship between action and object semantics. Therefore, this application proposes a multi-layer control optimization module (MCO) to jointly constrain the model from three levels: global semantic consistency, spatial structure rationality, and action semantic discriminability, thereby enhancing the semantic controllability and generation stability of the model.

[0084] To address the CLIP reward loss and facilitate explicit supervision of semantic alignment between generated images and text descriptions, the MCO module introduces a CLIP-based reward loss. This loss calculates image-text similarity in pixel space, but directly using full sampling to generate images incurs significant memory overhead. Therefore, we employ a single-step efficient sampling strategy: first, we sample the training images... Encode the latent representation using a pre-trained VAE autoencoder. Then a small amount of noise is applied. (i.e., adding noise at a smaller time step) to obtain The aim is to disrupt its consistency with the text and restore the image using single-step denoising. :

[0085] in, The cumulative product coefficients of the diffusion scheduling function are represented. and These are text and HOI control condition inputs, respectively.

[0086] After obtaining the generated image Then, the image is decoded into a pixel-space image using a pre-trained VAE autoencoder and uniformly scaled to the CLIP input resolution (224×224). Subsequently, the pre-trained CLIP image encoder and text encoder are called to extract the image embeddings. With text embedding L2 normalization is then performed. Finally, a bidirectional InfoNCE contrastive loss is constructed:

[0087] in For temperature coefficient, The loss function matches the labels. By comparing negative samples within a batch, this loss function maximizes the similarity of correct image-text pairs and minimizes the similarity of incorrect image-text matches, thereby providing explicit global semantic consistency supervision for the model while maintaining the stability of diffusion training.

[0088] To address the coordinate regression loss and further ensure the spatial rationality of human-object interactions in the generated images, the MCO module introduces coordinate regression loss after the HSE module outputs features. By explicitly predicting human bounding boxes, object bounding boxes, and action mid-region bounding boxes, the semantic structure is mapped to the geometric space, enhancing the geometric consistency of human-object interactions in the generated images.

[0089] Specifically, in obtaining the main characteristics Object characteristics and action characteristics Then, through a lightweight regression head Map it to the corresponding bounding box prediction and the actual annotation box Perform L1 loss supervision:

[0090]

[0091] From the perspective of representation learning, if the semantic decomposition features truly encode the positions of people, objects, and interaction areas, then their corresponding geometric positions should be recoverable through simple linear mapping. Introducing coordinate regression loss forces the semantic embedding to retain spatial discriminative information, preventing semantic tokens from degenerating into abstract vectors with only textual semantics and lacking geometric constraints, thereby improving the spatial accuracy of the generated results.

[0092] While global image-text alignment constraints can improve overall semantic consistency, they still fall short in distinguishing fine-grained interactions. Diffusion models often focus more on the overall appearance, which can easily lead to action blurring or semantic drift. To address this, the MCO module further proposes an action classification contrastive loss. By explicitly extracting action-related image features in the intermediate layers of U-Net and comparing them with the corresponding action text features, the model's ability to discriminate action semantics is strengthened.

[0093] Let the intermediate layer visual features of the denoising U-Net network be... In order to obtain visual representations that focus only on action-related regions from visual features. This application employs a cross-attention mechanism, using the action semantic features output by the HSE module as the query vector and visual features as the key and value.

[0094] This operation uses action semantics to query features across the entire image, thereby focusing on the spatial region most relevant to the current action. Compared to directly using image features from global average pooling, this mechanism can filter out irrelevant background information, allowing the visual representation to focus more on the interactive area.

[0095] Subsequently, visual motion features Text action features L2 normalization is performed separately, and InfoNCE contrastive loss is used for supervision to ensure that each visual action feature is matched only with its corresponding text action feature.

[0096] in For temperature coefficient, The cosine similarity is used. This loss maximizes the similarity between correct action pairs by comparing negative samples within a batch, while minimizing confusion between different actions, thereby improving the ability to distinguish action semantics.

[0097] Overall, the MCO module collaboratively optimizes the generation process from different dimensions through three loss functions: CLIP reward loss ensures overall semantic consistency between text and images, action classification comparison loss enhances fine-grained discrimination at the action level, and coordinate regression loss constrains the rationality of spatial geometric structure. These three functions complement each other, jointly improving the semantic accuracy and structural stability of human-object interaction image generation tasks.

[0098] Loss function part: To ensure stable training and multi-dimensional constraints of the model in human-object interaction generation tasks, this application further introduces semantic consistency constraints, action discrimination constraints, and spatial structure constraints on the basis of the standard diffuse noise prediction objective, and constructs a joint optimization multi-objective loss function system.

[0099] First, the basic training objective of the diffusion model still uses the mean squared error loss for noise prediction. Let the original latent variables be... , at time step Adding Gaussian noise yields The model prediction noise is The actual noise is The basic diffusion loss is defined as:

[0100] This loss term directly constrains the model's ability to fit the noise distribution during forward diffusion, and serves as a fundamental supervisory signal to ensure the quality of the generated image and the stability of training convergence.

[0101] Building upon this, to enhance the overall consistency between the generated image and the text semantics, this application proposes three auxiliary loss functions for the MCO module. These functions jointly constrain the model from three levels: global semantic consistency, spatial structure rationality, and action semantic discriminability. The final total loss function is defined as follows:

[0102] Among them, hyperparameters , , These are the weighting coefficients for the corresponding loss terms, used to balance the impact of semantic consistency, action discrimination ability, and spatial structure constraints on the overall training objective.

[0103] Through the above multi-objective joint optimization strategy, the model maintains the stability of diffusion generation while coordinating constraints from three dimensions: global semantic consistency, action level discrimination ability, and spatial geometric rationality, thereby achieving simultaneous improvement in semantic expression and structural layout of human-object interaction images.

[0104] As another example, an experimental analysis is conducted on the technical solution provided in Embodiment 1.

[0105] Data set and evaluation metrics: This example demonstrates the experimental validation of the proposed method on the HICO-DET dataset. HICO-DET is one of the widely adopted standard datasets in the field of human-object interaction, derived from the MS COCO image collection, containing 47,776 natural scene images, with 38,118 images in the training set and 9,658 images in the test set. This dataset defines 80 object categories and 117 action categories, which combine to form 600 human-object interaction categories. Unlike datasets that only provide image-level labels, HICO-DET provides fine-grained instance-level annotation information, including bounding boxes of people and objects and corresponding interaction relationship annotations. This provides direct spatial control signals and semantic supervision information for the human-object interaction image generation task based on text prompts and coordinate frame conditions, as described in this application.

[0106] To comprehensively evaluate the generation quality, interaction controllability, and action semantic consistency of the proposed method, this application employs four evaluation metrics for experimental analysis: Fréchet Inception Distance (FID), Kernel Inception Distance (KID), HOI Detection Score, and Action Detection Score. FID and KID evaluate the difference in feature distribution between the generated image and the real image; lower values ​​indicate higher quality and a distribution closer to the real data. HOI Detection Score evaluates the detectability and accuracy of human-object interaction structures in the generated results; higher values ​​indicate greater accuracy in human-object localization and action category matching. Action Detection Score further evaluates the action semantic consistency in the generation task, measuring the model's ability to express complex interactive actions by comparing the semantic matching degree between the generated image and the input text; higher values ​​indicate stronger consistency between the generated result and the text prompt at the action semantic level. The HOI Detection Score evaluates generated images using existing HOI detectors, measuring the model's control over human-object interaction relationships from an interaction detection perspective. The Action Detection Score addresses the shortcomings of the traditional HOI Detection Score, which relies on discrete action classification and struggles to characterize fine-grained semantic differences in actions. This metric first uses a visual language model to perform text inversion on the generated image, obtaining the corresponding action description. Then, it uses a pre-trained language model to extract the semantic representations of the inverted text and the original prompt text, calculating their similarity. The degree of matching in the continuous semantic space characterizes the consistency between the generated action and the text conditions. Based on these metrics, this application can systematically evaluate the performance of the proposed method from multiple perspectives, including image quality, interaction structure rationality, and action semantic accuracy.

[0107] Experiment setup section: In this application, all images were uniformly scaled to 512×512 resolution during model training and evaluation. Model training employed the AdamW optimizer for parameter updates, with an initial learning rate of 0.00005. The total training steps were 500,000, with the first 10,000 steps using a linear warm-up strategy to mitigate gradient oscillations in the early training phase. The batch size was set to 8, and all training was completed on a single RTX 4090 GPU, with an overall training time of approximately 150 hours. A dynamic learning rate scheduling strategy was employed to improve model convergence stability. The inference phase employs a 50-step diffusion sampling method, accelerated by using the PLMS sampler, with a Classifier-Free Guidance coefficient set to 4.0. In the semantic construction phase, the fine-grained action descriptions required in this work are generated by the thinking patterns of the large language model DeepSeek-V3.2, and structured semantic input is provided through semantic library retrieval during both the training and inference phases. Unless otherwise specified in this application, all experiments were conducted on a server environment with a single RTX 4090 GPU and 64GB of physical memory. The algorithm is implemented using Python 3.9 and PyTorch 2.4.1 to ensure experimental reproducibility and efficient use of computational resources.

[0108] Overall performance evaluation section: To verify the effectiveness of the proposed structured semantic enhancement-based human-object interaction generation method, this application conducted systematic experiments on the HICO-DET dataset and compared it with various mainstream methods, including Stable Diffusion, GLIGEN, InteractDiffusion, and EmIT, on three benchmarks: FGAHOI, RLIPv2 [29,30], and Action DetectionScore. Specific results are shown in Tables 1, 2, and 3. It should be noted that since some methods (such as EmIT) have not yet released their code, this application only cites the metrics reported in their original papers under the same experimental settings for comparison. Evaluation results not provided are indicated by "-" in the tables.

[0109] Table 1 Comparison of experimental results of the method in this application on the FGAHOI benchmark.

[0110] In the FGAHOI benchmark, the proposed method achieved optimal performance across different detectors and evaluation modes. Under the Swin-Tiny detector, the proposed method achieved a Full subset mAP of 31.24 and a Rare subset mAP of 24.75 in the Default setting, representing a further improvement over DDP-Diffusion's 30.73 and 24.06, respectively. In the Known mode, the proposed method achieved Full subset mAPs of 33.35 and Rare subset mAPs of 26.53, maintaining its leading advantage. Under the higher-performance Swin-Large detector, the proposed method achieved a Full subset mAP of 33.14 in the Default mode, further improving to 34.59 in the Known mode, and a Rare subset mAP of 28.46, with overall performance superior to existing methods such as InteractDiffusion and GLIGEN. Compared to the detection results of real data, the method in this application has approached or even surpassed the detection performance of real samples under certain settings. This indicates that the structured fine-grained semantic description introduced in this application method can effectively enhance the model's ability to jointly model the relationship between people, objects and actions, thereby improving the performance of the generated results in terms of interaction structure rationality and spatial consistency.

[0111] In the RLIPv2 benchmark, the proposed method also demonstrated stable performance advantages. Under the RLIPv2-Tiny detector, the proposed method achieved scores of 36.13 and 29.65 on the Full and Rare subsets, respectively, outperforming most baseline models overall. Under the RLIPv2-Large detector, the proposed method achieved a score of 33.15 on the Rare subset, higher than InteractDiffusion's 32.18, indicating a significant improvement in the interactive representation ability of low-frequency action categories. Regarding generation quality, the proposed method also achieved the best results in FID and KID metrics, with an FID of 16.64 and a KID of 0.00453, demonstrating that structured semantic enhancement can effectively improve the overall realism and distribution consistency of the diffused generated images.

[0112] Table 2 Comparison of experimental results of the method in this application on the RLIPv2 benchmark.

[0113] Table 3 Comparison of experimental results of different methods on the Action Detection Score metric

[0114] In the Action Detection Score evaluation, the proposed method also achieved significant improvements in image-text semantic consistency. Experimental results show that the average semantic similarity of the proposed method reaches 0.7314, which is significantly higher than DDP-Diffusion's 0.7194 and InteractDiffusion's 0.7061. Furthermore, the proposed method exhibits higher matching ratios at different similarity thresholds, reaching 45.63% for Sim@0.75, 23.41% for Sim@0.8, and 8.15% for Sim@0.85. At the more stringent Sim@0.9 and Sim@0.95 thresholds, the ratios reach 3.64% and 0.59% respectively, all significantly better than existing methods. Compared to the detection results on real data, the proposed method also shows good semantic consistency in high-threshold regions, indicating that structured semantic information can effectively enhance the action semantic correspondence between the generated image and the text prompt, enabling the model to exhibit better controllability and semantic fidelity at the fine-grained action expression level.

[0115] In summary, the method proposed in this application outperforms existing methods in all three evaluation dimensions: FGAHOI, RLIPv2, and Action Detection Score. This demonstrates that by introducing structured semantic information generated by a large language model, the diffusion model's ability to model complex human-object interactions can be significantly enhanced, resulting in improved interaction structure rationality, image quality, and action semantic consistency.

[0116] Ablation experiment section: To further verify the effectiveness of each module design, this example conducted ablation experiments on the HICO-DET dataset, as shown in Table 4, and analyzed the two structural modules and three loss functions respectively. The experimental settings were consistent with previous experiments, and the evaluation metrics included Full, Rare, Similarity, and Sim@0.75. For module ablation, the original HOI triples were used as semantic input when removing the FSD module, and the MLP mapping was replaced when removing the HSE module to keep its dimensionality consistent with the subsequent network.

[0117] Table 4 shows the performance of the proposed method on different modules of the HICO-DET dataset.

[0118] First, a combined ablation experiment was conducted on the FSD and HSE modules. As shown in Table 4, after removing both modules, the model relied only on the original HOI triples and MLP mapping. At this point, Full and Rare scores were 33.13 and 26.42, respectively, with a Similarity of 0.7054. After introducing only the FSD module, the model significantly improved its semantic expressive power by leveraging the fine-grained semantic descriptions generated by the large language model. Full score increased to 35.62, Rare to 28.17, and Similarity to 0.7255, indicating that fine-grained semantics plays an important role in understanding complex actions. Only retaining the HSE module... When using the SE module, the original HOI triples are structured and encoded using a hierarchical Transformer, resulting in a Full score of 34.98 and a Rare score of 28.02, indicating that hierarchical modeling also contributes to feature enhancement. Simultaneously introducing both modules yields the best performance: a Full score of 36.13, a Rare score of 29.65, a Similarity score of 0.7314, and a Sim@0.75 score of 45.63%. These results demonstrate that the FSD module enhances expressive power by enriching semantics, while the HSE module strengthens semantic organization through structural modeling; the two complement each other and jointly improve HOI generation performance.

[0119] Table 5 shows the performance of the proposed method on the HICO-DET dataset with different loss functions.

[0120] In the loss function ablation experiments, all models were built based on the complete FSD and HSE modules. As shown in Table 5, when using only the original noise prediction loss of the diffusion model, the Full score was 34.96 and the Rare score was 28.15. After adding the CLIP reward loss and constraining the semantic consistency between text and images, the Full score increased to 35.54 and the Similarity score increased to 0.7285, enhancing the semantic matching ability. Further adding coordinate regression loss to explicitly supervise the spatial location of people, objects, and action regions improved the Full score to 36.01 and the Rare score to 28.95, enhancing the model's spatial structure learning ability. Finally, action contrast loss was introduced to strengthen action semantic discrimination through comparative learning of visual and text action features, achieving the optimal model: Full score of 36.13, Rare score of 29.65, Similarity score of 0.7314, and Sim@0.75 score of 45.63%. This indicates that action contrast learning can further improve the model's ability to distinguish fine-grained action semantics, making the generated results more accurate in terms of action-level expression. Therefore, the FSD module brings the most significant gain through fine-grained semantics, while the HSE module strengthens semantic structure modeling; the three losses are optimized synergistically from the perspectives of semantic consistency, spatial constraints and action discrimination, respectively, which together verify the effectiveness of the method in this application.

[0121] Results visualization section: Figure 3 and Figure 4 The paper presents a comparison of the generation results of the method in this application with those of existing methods, as well as the correspondence between the structured semantic description and the generation results.

[0122] from Figure 3It can be seen that GLIGEN can generate corresponding person and object instances in the corresponding region based on a given bounding box, but its generation results are more of a spatial placement of the target, lacking clear and reasonable interaction relationships. For example, in the "holding a bicycle" scene, although GLIGEN generates a person and a bicycle, a clear control relationship is not formed between the person and the object; in the "feeding a horse" scene, the model also fails to show a clear feeding action, indicating that relying solely on target-level control is insufficient to effectively characterize complex interaction semantics. In contrast, InteractDiffusion can generate images with human-object interaction relationships to some extent, but it still has shortcomings in fine-grained action expression. In the "holding a bicycle" scene, an extra bicycle appears in the generation result, and the person's posture is closer to "riding a bicycle" than "holding a bicycle," reflecting that the model's ability to distinguish similar action semantics is still limited; in the "feeding a horse" scene, although there is a certain spatial proximity between the person and the horse, a clear feeding action is not formed, indicating that the model still has deviations in understanding complex action semantics.

[0123] In contrast, the method proposed in this application demonstrates more accurate interactive semantic control capabilities across multiple scenarios. In the "holding a bicycle" scenario, the method generates results more consistent with the text description, establishing a reasonable support and control relationship between the person and the bicycle, indicating that the structured semantic enhancement mechanism effectively improves the model's ability to model fine-grained action differences. In scenarios such as "washing a dog" and "lying on a bed," the method also generates interactive results more consistent with the text semantics and bounding box constraints, demonstrating good semantic consistency and spatial rationality. In the more challenging example of "feeding a horse," although the model failed to explicitly generate the typical feeding posture completely, the person's forward lean, the relative position between the person and the horse, and the contact trend already reflect the interactive semantics required for "feeding" to some extent, showing a tendency to generate semantics closer to the target compared to baseline methods. This indicates that the method proposed in this application still has strong semantic guidance capabilities in complex and long-tailed interactive scenarios, but there is still room for improvement in the accurate modeling of some high-difficulty actions.

[0124] Figure 4This demonstrates the correspondence between structured semantic descriptions and generated images. For examples such as "holding a bicycle" and "feeding a giraffe," the fine-grained semantic descriptions generated by the large language model not only retain the action and object information from the original text but also further supplement multi-dimensional semantic attributes such as posture, contact method, spatial relationship, action intention, and duration. For instance, in the "holding a bicycle" scenario, the structured description provides key information such as "standing beside the bicycle," "hands on handlebars or frame," and "bicycle is stationary or moving beside," while the standing posture of the person in the generated image, the contact method between the person and the bicycle, and their spatial relationship all maintain a high degree of consistency with these descriptions. In the "feeding a giraffe" scenario, descriptions in the structured semantics such as "standing or reaching upward," "hand offering food," and "giraffe is above or at a distance" are also clearly reflected in the generated results. The results show that the fine-grained action semantic decomposition module proposed in this application can extend the original HOI triples into a more expressive structured semantic representation, while the hierarchical action semantic encoding module can further explore the intrinsic relationship between these semantic elements, thereby providing a more stable and accurate control signal for the diffusion model.

[0125] In summary, this application addresses the challenges of insufficient modeling of coarse-grained and fine-grained action information in the original HOI triple semantic representation during complex human-object interaction image generation, as well as limitations in action discriminability and semantic consistency of the generated results. It proposes a human-object interaction image generation method based on structured semantic enhancement. This method, without altering the core structure of the latent space diffusion model, performs fine-grained semantic reconstruction of the original action description at the control condition expression level. A fine-grained action semantic decomposition module expands the compressed interaction description into a multi-slot structured semantic representation (multi-slot structured semantic description). Furthermore, it combines a hierarchical action semantic encoding module with a multi-layered control optimization strategy to enhance the model's understanding and modeling ability of complex interaction relationships and fine-grained action differences. Experimental results on the HICO-DET dataset demonstrate that the proposed method achieves good results in terms of image quality, semantic consistency, and interaction rationality, validating the effectiveness of the structured semantic enhancement-based method in human-object interaction image generation tasks and providing a feasible research approach for controllable image generation in complex interaction scenarios.

[0126] Example 2.

[0127] This embodiment provides a human-object interaction image generation device based on structured semantic enhancement, including: A hierarchical action semantic encoding module is used to obtain the human-object interaction triplet description and bounding box conditions of human-object interaction images; based on the semantic library pre-built by the large language model, the human-object interaction triplet description is extended into a multi-slot structured semantic description, and the text of each slot is encoded into a semantic embedding vector by a text encoder; the semantic embedding vector of each slot is subjected to in-slot semantic aggregation, and the token sequence of each slot is compressed into a single slot semantic vector; The fusion module is used to perform cross-slot relationship modeling on the slot semantic vector, fuse multi-slot semantic information through a self-attention mechanism, and output a global action semantic representation. A diffusion model denoising network is used to obtain subject semantic representation and object semantic representation from the slot semantic vector; generate modulation parameters based on the global action semantic representation, and perform feature modulation on the subject semantic representation and the object semantic representation respectively to obtain modulated subject semantic representation and object semantic representation; fuse the modulated subject semantic representation, object representation and global action semantic representation with the corresponding spatial location features to form a conditional representation vector, and inject it into the diffusion model denoising network to generate an interactive image.

[0128] In an exemplary embodiment, the multi-slot structured semantic description includes the following slot types: subject information slot, action information slot, object information slot, posture information slot, contact method information slot, spatial relationship information slot, behavioral intention information slot, and time duration feature information slot. In an exemplary embodiment, the semantic library pre-built based on a large language model extends the human-object interaction triplet description into a multi-slot structured semantic description, including: For all action-object combinations in the dataset, a fine-grained, multi-slot structured semantic description is generated offline using a large language model and stored to construct the semantic library. During the training and inference phases, action-object pairs are extracted based on the input human-object interaction triplet description, and indexed queries are performed in the semantic database to obtain the corresponding multi-slot structured semantic description.

[0129] In an exemplary embodiment, the intra-slot semantic aggregation and the cross-slot relationship modeling constitute a hierarchical coding structure: The first stage is the semantic aggregation within the slot, which realizes semantic compression from the word level to the slot level, and aggregates multiple tokens in each slot into a slot semantic vector. The cross-slot relationship modeling is the second stage, which realizes semantic fusion from the slot level to the interaction level. Global self-attention encoding is performed on all slot semantic vectors to output the global action semantic representation that integrates multi-slot information. In one exemplary embodiment, the in-slot semantic aggregation includes: A learnable query vector is introduced, and the contribution of each token in each slot to the learnable query vector is calculated through a multi-head attention mechanism. Key semantic information is adaptively aggregated, and all token sequences in each slot are compressed into a slot semantic vector. In an exemplary embodiment, the step of generating modulation parameters based on the global action semantic representation and performing feature modulation on the subject semantic representation and object semantic representation respectively includes: The global action semantic representation is used to generate scaling and offset parameters through linear mapping; Using the scaling and offset parameters, residual modulation based on the FiLM mechanism is performed on the subject semantic representation and the object semantic representation, respectively, so that the action semantics have a reverse influence on the feature representation of the subject and the object. In one exemplary embodiment, the training process of the denoising network includes: During the diffusion training process, global semantic consistency loss, spatial coordinate regression loss, and action discrimination contrast loss are jointly applied to perform multi-layer control optimization on the denoising network; the semantic alignment between the generated image and the text description is constrained by the global semantic consistency loss. The spatial geometric structure of human-object interaction in the generated image is constrained by spatial coordinate regression loss. By using action discrimination contrast loss, the model's ability to distinguish fine-grained semantics of specific interactive actions is enhanced.

[0130] In an exemplary embodiment, the calculation of the action discrimination contrast loss includes: Using a cross-attention mechanism, the global action semantic representation is used as the query vector to extract action-related region features from the intermediate layer visual features of the denoising network; The action-related region features are compared and learned with the corresponding action text semantic features to maximize the similarity of correct action pairs and minimize semantic confusion between different actions.

[0131] In an exemplary embodiment, the calculation of the spatial coordinate regression loss includes: The bounding box coordinates of the subject features, object features and action features are predicted by the regression head, respectively, to obtain the predicted bounding box of the person, the predicted bounding box of the object and the predicted bounding box of the middle region of the action. The predicted bounding boxes of the person, the predicted bounding boxes of the object, and the predicted bounding boxes of the middle region of the action are respectively subjected to L1 loss supervision with the ground truth bounding boxes.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A method for generating human-object interaction images based on structured semantic enhancement, characterized in that, include: Obtain the human-object interaction triplet description and bounding box conditions of human-object interaction images; Based on the semantic library pre-built by the large language model, the human-object interaction triplet description is extended into a multi-slot structured semantic description, and the text of each slot is encoded into a semantic embedding vector through a text encoder. Perform intra-slot semantic aggregation on the semantic embedding vector of each slot, and compress the token sequence of each slot into a single slot semantic vector; Cross-slot relationship modeling is performed on the slot semantic vector, and multi-slot semantic information is fused through a self-attention mechanism to output a global action semantic representation; Obtain the subject semantic representation and the object semantic representation from the slot semantic vector; Based on the global action semantic representation, modulation parameters are generated, and feature modulation is performed on the subject semantic representation and the object semantic representation respectively to obtain the modulated subject semantic representation and object semantic representation; The modulated subject semantic representation, object semantic representation, and global action semantic representation are fused with their corresponding spatial location features to form a conditional representation vector, which is then injected into the denoising network of the diffusion model to generate an interactive image.

2. The method according to claim 1, characterized in that, The multi-slot structured semantic description includes the following slot types: subject information slot, action information slot, object information slot, posture information slot, contact method information slot, spatial relationship information slot, behavioral intention information slot, and time duration feature information slot.

3. The method according to claim 2, characterized in that, The semantic library pre-built based on the large language model extends the human-object interaction triplet description into a multi-slot structured semantic description, including: For all action-object combinations in the dataset, a fine-grained, multi-slot structured semantic description is generated offline using a large language model and stored to construct the semantic library. During the training and inference phases, action-object pairs are extracted based on the input human-object interaction triplet description, and indexed queries are performed in the semantic database to obtain the corresponding multi-slot structured semantic description.

4. The method according to claim 1, characterized in that, The intra-slot semantic aggregation and the cross-slot relationship modeling constitute a hierarchical coding structure: The first stage is the semantic aggregation within the slot, which realizes semantic compression from the word level to the slot level, and aggregates multiple tokens in each slot into a slot semantic vector. The cross-slot relationship modeling is the second stage, which realizes semantic fusion from the slot level to the interaction level. Global self-attention encoding is performed on all slot semantic vectors to output the global action semantic representation that integrates multi-slot information.

5. The method according to claim 4, characterized in that, The semantic aggregation within the slot includes: A learnable query vector is introduced, and the contribution of each token in each slot to the learnable query vector is calculated through a multi-head attention mechanism. Key semantic information is adaptively aggregated, and the entire token sequence of each slot is compressed into a slot semantic vector.

6. The method according to claim 4, characterized in that, The step of generating modulation parameters based on the global action semantic representation and performing feature modulation on the subject semantic representation and object semantic representation respectively includes: The global action semantic representation is used to generate scaling and offset parameters through linear mapping; Using the scaling and offset parameters, residual modulation based on the FiLM mechanism is performed on the subject semantic representation and the object semantic representation, respectively, so that the action semantics have a reverse influence on the feature representation of the subject and the object.

7. The method according to claim 1, characterized in that, The training process of the denoising network includes: During the diffusion training process, global semantic consistency loss, spatial coordinate regression loss, and action discrimination contrast loss are jointly applied to perform multi-layer control optimization on the denoising network; the semantic alignment between the generated image and the text description is constrained by the global semantic consistency loss. The spatial geometric structure of human-object interaction in the generated image is constrained by spatial coordinate regression loss. By using action discrimination contrast loss, the model's ability to distinguish fine-grained semantics of specific interactive actions is enhanced.

8. The method according to claim 7, characterized in that, The calculation of the action discrimination contrast loss includes: Using a cross-attention mechanism, the global action semantic representation is used as the query vector to extract action-related region features from the intermediate layer visual features of the denoising network; The action-related region features are compared and learned with the corresponding action text semantic features to maximize the similarity of correct action pairs and minimize semantic confusion between different actions.

9. The method according to claim 7, characterized in that, The calculation of the spatial coordinate regression loss includes: The bounding box coordinates of the subject features, object features and action features are predicted by the regression head, respectively, to obtain the predicted bounding box of the person, the predicted bounding box of the object and the predicted bounding box of the middle region of the action. The predicted bounding boxes of the person, the predicted bounding boxes of the object, and the predicted bounding boxes of the middle region of the action are respectively subjected to L1 loss supervision with the ground truth bounding boxes.

10. A human-object interaction image generation device based on structured semantic enhancement, characterized in that, include: The hierarchical action semantic encoding module is used to obtain the human-object interaction triplet description and bounding box conditions of human-object interaction images; Based on the semantic library pre-built by the large language model, the human-object interaction triplet description is extended into a multi-slot structured semantic description, and the text of each slot is encoded into a semantic embedding vector by a text encoder; the semantic embedding vector of each slot is subjected to in-slot semantic aggregation, and the token sequence of each slot is compressed into a single slot semantic vector. The fusion module is used to perform cross-slot relationship modeling on the slot semantic vector, fuse multi-slot semantic information through a self-attention mechanism, and output a global action semantic representation. A diffusion model denoising network is used to obtain subject semantic representation and object semantic representation from the slot semantic vector; generate modulation parameters based on the global action semantic representation, and perform feature modulation on the subject semantic representation and the object semantic representation respectively to obtain modulated subject semantic representation and object semantic representation; fuse the modulated subject semantic representation, object representation and global action semantic representation with the corresponding spatial location features to form a conditional representation vector, and inject it into the diffusion model denoising network to generate an interactive image.

Images