Spatial constraint action understanding generation method and related equipment

Abstract

The invention provides a spatial constraint action understanding generation method and related equipment. The method comprises the steps of obtaining control condition information; inputting control condition information into the trained relationship understanding diffusion model to obtain a character action target image; wherein the relation understanding diffusion model is composed of a diffusion model backbone network, a space control module and an action understanding attention module, and the diffusion model backbone network is used for finally generating a target image; the space control module is used for guiding the generation of a target person and an object in the bounding box by fusing the mask into the feature map; and the action understanding attention module is used for fusing text description and bounding box information into image features by adjusting a calculation mode of a self-attention layer and extracting and learning action-related information from the image features. By combining the bounding box information and the text description information and utilizing the specific model structure, the method achieves the precise control of the generated image, and is good in image generation quality and accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and image generation, and in particular to a spatially constrained action understanding generation method and related equipment. Background Art

[0002] In recent years, image generation has made significant progress in the field of computer vision, especially in image generation models trained with large-scale Internet data. Deep learning-based image generation methods, especially latent diffusion models (LDMs), have shown their ability to generate high-quality images, especially based on multi-conditional control (such as prompt text, bounding boxes, edges, or graffiti). However, in specific applications, current image generation techniques still have certain limitations, especially in generating images that conform to expected actions and relationships between people and objects.

[0003] This is because LDM will suffer from semantic loss when generating dynamic human actions or complex object interactions, especially when generating human actions based on text prompts. For example, when using prompt text to generate images of human actions, LDM usually only focuses on part of the information in the text (such as the action description) and ignores other equally important text information (such as the object of the action). The above semantic loss phenomenon will cause the generated image to be inconsistent with the actual expectations, seriously affecting the quality and accuracy of image generation.

[0004] Based on this, the present application provides a spatial constraint action understanding generation method and related equipment. Summary of the invention

[0005] In response to the problems existing in the prior art, the present application provides a spatial constraint action understanding generation method and related equipment.

[0006] The purpose of this application is achieved by the following technical solutions:

[0007] In a first aspect, the present application provides a method for generating spatial constraint action understanding, the method comprising:

[0008] Acquiring control condition information, wherein the control condition information includes bounding box information and text description information;

[0009] Inputting the control condition information into a trained relational understanding diffusion model to obtain a character action target image;

[0010] Among them, the relationship understanding diffusion model is composed of a diffusion model backbone network, a spatial control module and an action understanding attention module. The diffusion model backbone network is used to predict the noise added in the image generation process, and finally generate the target image through multi-level information transmission; the spatial control module is used to guide the model to generate target people and objects within the bounding box by integrating masks into the feature map; the action understanding attention module is used to fuse text descriptions and bounding box information into image features by adjusting the calculation method of the self-attention layer, and extract and learn action-related information therefrom.

[0011] In some possible implementations, the diffusion model backbone network adopts a Unet network structure, and each layer consists of a self-attention layer and a cross-attention layer; jump connections are used between each layer to directly pass low-level features to the decoder.

[0012] In some possible implementations, the bounding box information includes a person bounding box and an action object bounding box, and the space control module is used to:

[0013] For the continuous multiple middle layers of the Unet network structure, receiving the image features extracted by the previous Unet network and the bounding box information of the model input, and obtaining the overlapping part of the person bounding box and the action object bounding box through the bounding box information to obtain the action bounding box; merging the person bounding box, the action object bounding box and the action bounding box into a triple bounding box;

[0014] Downsample each bounding box using linear interpolation, downsample the bounding box to a predetermined size, and generate a mask at a corresponding position according to the bounding box coordinates;

[0015] The feature map is subjected to mask constraints according to the mask, and the mask and the feature map of the same size are fused by element-by-element multiplication, and the fused feature map is decomposed into multiple independent feature maps, each of which is used to focus on a type of information in a person, an object, or an action;

[0016] Input each independent feature map into the corresponding attention module according to the category of aggregated information to obtain instance features and action features;

[0017] According to the instance features and the action features, the foreground feature map and the background features are integrated through a fusion attention layer to supplement the completeness of the scene, and a fusion feature map combined with mask constraints is obtained; wherein the background features are provided by the image features.

[0018] In some possible implementations, the step of inputting each independent feature map into a corresponding attention module according to the category of aggregated information to obtain instance features and action features includes:

[0019] The independent feature maps of clustered people and objects are input into the instance attention layer to generate instance features; the independent feature maps of clustered actions are input into the action attention layer to generate action features.

[0020] In some possible implementations, the triplet feature includes a person condition feature, an object condition feature, and an action condition feature, and the action understanding attention module is used to:

[0021] Acquire the triplet feature, and concatenate and calculate the control condition and the fusion feature map according to the triplet feature to obtain a corresponding matrix;

[0022] The calculated matrix is ​​replaced into the self-attention layer of the outer layer of the relational understanding diffusion model, so that the model focuses on action-related information in the generation phase.

[0023] In some possible implementations, the triple feature is obtained by:

[0024] Extracting features of the control condition information through a self-attention layer according to the types of characters, objects, and actions, and obtaining a triplet feature representation of the control conditions including character conditions, object conditions, and action conditions;

[0025] The calculation formula for concatenating and calculating the control condition and the fusion feature map according to the triple feature representation is as follows:

[0026] Q h =γ h ·(SelfAttn h (c h )+c h ) (5)

[0027] Q o =γ o ·(SelfAttn o (c o )+c o ) (6)

[0028] Q a =γ a ·(SelfAttn a (c a )+c a ) (7)

[0029] Among them, Q h ,Q o ,Q a is the triple feature representation of the control condition, c h 、c o and c aare the control conditions of character type, object type and action type after the control condition information is disassembled, γ h , γ o , γ a is the adaptive weight and γ h , γ o The initial weight is less than γ a The initial weights are used to strengthen the model's response to action type c a of learning.

[0030] In some possible implementations, the control condition and the fusion feature map are concatenated and calculated according to the triple feature to obtain a corresponding matrix, including:

[0031] The control condition is transformed dimensionally using a linear layer to align the dimensions of the fused feature map;

[0032] The treated control condition c h 、c o 、c a It is concatenated with the fusion feature map and input into the action understanding attention module for calculation to obtain the matrix F q 、F k With F v , the corresponding calculation formula is as follows:

[0033] F q =(X+Q h +Q a )·W q (8)

[0034] F k =(X+Q o +Q a )·W k (9)

[0035] F v =(X+Q h +Q a )⊙(X+Q o +Q a )·W v (10)

[0036] Among them, ⊙ is element-by-element multiplication, · is matrix transformation, W q , W k and W v is the weight matrix, and X is the fused feature map.

[0037] In the second aspect, the present application also provides a relationship understanding diffusion model for implementing the spatial constraint action understanding generation method described in any one of the first aspects, wherein the relationship understanding diffusion model consists of a diffusion model backbone network, a spatial control module and an action understanding attention module.

[0038] In a third aspect, the present application further provides an electronic device, comprising a memory and at least one processor, wherein the memory stores a computer program, and the at least one processor is configured to implement a method as described in any one of the first aspects when executing the computer program.

[0039] In a fourth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by at least one processor, the steps of the method described in the first aspect are implemented.

[0040] In combination with the above technical solutions and the technical problems solved, the technical solutions to be protected by the present invention and the positive effects possessed are:

[0041] Taking into account the challenges faced by LDM in generating dynamic character movements or complex object interactions, especially the problem of semantic loss when generating character movements based on text prompts, resulting in the generated image not being consistent with actual expectations, this application introduces a spatial control module and an action understanding attention module. During the generation process, a mask mechanism is used to perform targeted constraints on the bounding boxes of characters and action objects, so that the positions of characters and action objects generated by the model are more in line with expectations; an action understanding attention module is designed and introduced to enhance the model's ability to understand actions by performing in-depth analysis of the action information in the text description information, so that the characters' movements in the generated images are more realistic and the relationship with the action objects is more natural.

[0042] In summary, the provided spatially constrained action understanding generation method achieves precise control of the generated image by combining bounding box information and text description information and utilizing a specific model structure, thereby improving the quality of image generation and the naturalness of the action, and the quality and accuracy of image generation are good. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present application is further described below in conjunction with the accompanying drawings and implementation methods.

[0044] Figure 1 It is a flowchart of a method for understanding and generating spatially constrained actions provided in an embodiment of the present application.

[0045] Figure 2 It is a network framework diagram of a relationship understanding diffusion model provided in an embodiment of the present application.

[0046] Figure 3 It is a schematic diagram of a space control module provided in an embodiment of the present application.

[0047] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] Below, in conjunction with the accompanying drawings and specific implementation methods, the present application is further described. It should be noted that, under the premise of no conflict, the various embodiments described below or the various technical features can be arbitrarily combined to form a new embodiment. The implementation procedures of the present application will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation procedures, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0049] The following briefly describes the relevant terms in the embodiments of the present application to facilitate understanding by those skilled in the art.

[0050] Latent diffusion model (LDM) is a technology widely used in the field of image generation in recent years. LDM effectively improves the quality and efficiency of generated images by introducing the concept of latent space. In LDM, the image is mapped to a lower dimensional latent space through the encoder of variational autoencoder (VAE), the diffusion model is used for image generation in the latent space, and then the image is reversely reconstructed by the decoder of VAE. This method avoids the problem that the diffusion model needs to process high-dimensional data during the generation process by gradually reversely generating clear images in the latent space. However, as described in the background technology, LDM may have the problem of semantic loss when generating dynamic character movements or complex object interactions, especially when generating character movements based on text prompts, which seriously affects the quality and accuracy of image generation.

[0051] In order to solve this problem, some methods are proposed in related technologies to enhance the semantic understanding ability of LDM when processing action generation. As an example, Interact Diffusion is one of the methods. Interact Diffusion processes the input bounding box information and text information through Fourier transform, uses a multi-layer linear network (MLP) to fuse the two types of information, and then generates three embeddings containing different information based on the triple of person, action, and action object and inputs them into the model. It reduces the semantic loss phenomenon of the model in the process of generating character actions to a certain extent, but the character actions in the generated images are still not clear enough, and lack the natural interaction between the character and the action object.

[0052] The difference between this application and the related art is that, considering how to focus on making the model understand the character's actions and generate images that meet the expected conditions, and avoiding the bottleneck of the model in understanding and generating complex action relationships, a spatial constraint action understanding generation method and related equipment are provided, aiming to enhance the model's understanding of the character's actions and generate images that meet the expected conditions. The following first describes the method embodiment, and then describes the model and equipment.

[0053] Method embodiment.

[0054] See also Figure 1 and Figure 2 , Figure 1 It is a flowchart of a method for understanding and generating spatially constrained actions provided in an embodiment of the present application. Figure 2 : is a network framework diagram of a relationship understanding diffusion model provided in an embodiment of the present application. The method is used for image generation, comprising:

[0055] S101, obtaining control condition information, where the control condition information includes bounding box information and text description information;

[0056] S102, inputting the control condition information into a trained relational understanding diffusion model to obtain a character action target image.

[0057] Among them, the relationship understanding diffusion model is composed of a diffusion model backbone network, a spatial control module and an action understanding attention module. The diffusion model backbone network is used to predict the noise added in the image generation process, and finally generate the target image through multi-level information transmission; the spatial control module is used to guide the model to generate target people and objects within the bounding box by integrating masks into the feature map; the action understanding attention module is used to fuse text descriptions and bounding box information into image features by adjusting the calculation method of the self-attention layer, and extract and learn action-related information therefrom.

[0058] The technical solution provided by this embodiment and the spatial constraint action understanding generation method provided can be used in the field of image generation to generate a target image containing a target person action and an action object. It mainly includes the acquisition of control condition information and the application of a relational understanding diffusion model.

[0059] During the acquisition of control condition information, the user provides the model with bounding box information of people and action objects, which indicates the expected positions of people and objects in the image. The user also provides textual hints describing the relationship between the expected generated person actions and objects.

[0060] In specific applications, the diffusion model backbone network in the relational understanding diffusion model is a network based on the Unet structure, which is used to predict the noise in the image generation process, and uses the self-attention layer and the cross-attention layer (ie SA) to extract image and text features, thereby generating clear and expected images. The spatial control module guides the model to generate target characters and objects within the bounding box specified by the user by integrating the mask into the feature map, ensuring that the positions of the generated characters and objects are consistent with the bounding box input by the user, and improving the accuracy of the spatial layout of the generated image. The action understanding attention module adjusts the calculation method of the self-attention layer, integrates the text description and bounding box information into the image features, focuses on extracting and learning information related to the action, and enhances the model's ability to understand the semantics of the action, making the actions of the characters in the generated image more realistic and the relationship with the action object more natural.

[0061] The beneficial effect of this embodiment is that, considering the challenges faced by LDM in generating dynamic character actions or complex object interactions, especially when generating character actions based on text prompts, there will be a problem of semantic loss, resulting in the generated image not being consistent with the actual expectations. This embodiment introduces a spatial control module and an action understanding attention module, which can more accurately generate characters and objects within a specified bounding box, and generate more natural and realistic character actions, significantly improving the quality of the generated image. Through the use of the spatial control module, the model can more accurately generate characters and objects within a specified bounding box, improving the accuracy of the spatial layout of the image. The action understanding attention module enables the model to understand and learn character actions more deeply, and the character actions in the generated image are more realistic and the relationship with the action object is more natural. Through the application of multi-level information transfer and self-attention mechanism, the model can generate higher quality images with clearer details and textures. Through the introduction of hierarchical features of control conditions, the model's understanding of actions is more delicate and accurate, thereby improving the consistency of actions in generated images.

[0062] In summary, the provided spatially constrained action understanding generation method achieves precise control of the generated image by combining bounding box information and text description information and utilizing a specific model structure, thereby improving the quality of image generation and the naturalness of the action, and the quality and accuracy of image generation are good.

[0063] In an exemplary embodiment, the diffusion model backbone network adopts a Unet network structure, and each layer consists of a self-attention layer and a cross-attention layer; jump connections are used between each layer to directly pass low-level features to the decoder.

[0064] The Unet network structure includes an encoder-decoder architecture. The encoder implements downsampling through convolutional layers and is responsible for gradually extracting high-level features of the image; the decoder restores the potential spatial resolution of the image by gradually upsampling. The technical solution provided in this embodiment includes a self-attention layer and a cross-attention layer in each layer of the network. The self-attention layer can capture the long-distance dependencies within the image and enhance the model's understanding of the global structure of the image. The cross-attention layer focuses on fusing external control conditions (such as text descriptions, bounding boxes, etc.) with image features to generate images that meet specific conditions. The jump connection directly connects the low-level feature map in the encoder to the corresponding layer in the decoder, allowing low-level features (such as edges, textures, and other detail information) to be passed directly to the decoder, which helps to generate more refined images.

[0065] The beneficial effect of this embodiment is that, through the combination of self-attention layer and cross-attention layer, the model can more effectively fuse the internal features of the image and the external control conditions to generate an image that is more in line with expectations. The jump connection helps to retain the detailed information of the image, making the generated image more refined in quality. The introduction of self-attention layer and cross-attention layer enhances the model's understanding and expression ability of complex scenes. It not only helps the model retain more detailed information, but also effectively avoids the loss of details caused by over-compression of information during the generation process.

[0066] In an exemplary embodiment, the bounding box information includes a person bounding box and an action object bounding box, and the space control module is used to:

[0067] For the continuous multiple middle layers of the Unet network structure, receiving the image features extracted by the previous Unet network and the bounding box information of the model input, and obtaining the overlapping part of the person bounding box and the action object bounding box through the bounding box information to obtain the action bounding box; merging the person bounding box, the action object bounding box and the action bounding box into a triple bounding box;

[0068] Downsample each bounding box using linear interpolation, downsample the bounding box to a predetermined size, and generate a mask at a corresponding position according to the bounding box coordinates;

[0069] The feature map is subjected to mask constraints according to the mask, and the mask and the feature map of the same size are fused by element-by-element multiplication, and the fused feature map is decomposed into multiple independent feature maps, each of which is used to focus on a type of information in a person, an object, or an action;

[0070] Input each independent feature map into the corresponding attention module according to the category of aggregated information to obtain instance features and action features; the categories of aggregated information are people, objects and actions;

[0071] According to the instance features and the action features, the foreground feature map and the background features are integrated through a fusion attention layer to supplement the completeness of the scene, and a fusion feature map combined with mask constraints is obtained; wherein the background features are provided by the image features.

[0072] The technical solution provided by the present embodiment is that the spatial control module is used to ensure that the positions of the generated characters and action objects are consistent with the bounding boxes specified by the user when generating a target image containing character actions, and the action relationship is natural and reasonable. Specifically, the bounding box information of the image features extracted from the upper layer of the Unet network and the model input is received, including the character bounding box and the action object bounding box. By calculating the overlapping part of the character bounding box and the action object bounding box, the action bounding box is obtained, which helps to determine the interaction area between the character and the action object. The character bounding box, the action object bounding box and the action bounding box are merged into a triple bounding box, which helps the model to better understand the characters, action objects and the interaction relationship between them. In order to align the dimensions of the image features, each bounding box is downsampled to a predetermined size (e.g., 64×64 pixels) using linear interpolation. According to the bounding box coordinates after downsampling, a mask is generated at the corresponding position, and the mask and the feature map can be fused by element-by-element multiplication, so that the feature map is constrained by the mask in space, ensuring that the generated characters and action objects are in the correct position. The fused feature map is decomposed into multiple independent feature maps, each of which focuses on a type of information in a person, object, or action, which helps the model process different information more finely.

[0073] Then, each independent feature map is input into the corresponding attention module, such as the instance attention layer and the action attention layer, to obtain instance features and action features. The foreground feature map and background features are integrated through the fusion attention layer to supplement the completeness of the scene and generate a fusion feature map combined with mask constraints, which helps to generate a more natural and complete image.

[0074] The beneficial effect of this embodiment is that, through mask constraints and bounding box processing, the positions of the generated characters and action objects are more accurate and consistent with the bounding boxes specified by the user. By merging the triple bounding box and the action bounding box, the model can better understand and generate natural and reasonable character action relationships. The model can more accurately control the positions and actions of the characters and action objects in the generated image, improving the controllability of image generation. By paying special attention to the interactive area between the characters and action objects, the model can more accurately generate interactive actions, improving the accuracy of action recognition and generation.

[0075] In summary, this technical solution significantly improves the spatial accuracy and naturalness of action relationships in image generation tasks through precise spatial constraints and feature fusion, while also improving the overall quality of generated images.

[0076] In an exemplary embodiment, the step of inputting each independent feature map into a corresponding attention module according to the category of the aggregated information to obtain instance features and action features includes:

[0077] The independent feature maps of clustered people and objects are input into the instance attention layer to generate instance features; the independent feature maps of clustered actions are input into the action attention layer to generate action features.

[0078] By decomposing the feature maps and processing them separately, unnecessary calculations can be reduced and the computational efficiency of the model can be improved; by inputting the feature maps into dedicated attention layers separately, the model can more accurately extract features related to people, objects, and actions, improving the accuracy and reliability of feature extraction; the generated instance features and action features can be used to more accurately control the image generation process to ensure that the generated images meet the user's expectations and control conditions; by specifically processing the features of people, objects, and actions, especially in scenes with person actions and object interactions, the generated images are visually more natural and realistic.

[0079] In an exemplary embodiment, the triplet feature includes a person condition feature, an object condition feature, and an action condition feature, and the action understanding attention module is used to:

[0080] Acquire the triplet feature, and concatenate and calculate the control condition and the fusion feature map according to the triplet feature to obtain a corresponding matrix;

[0081] The calculated matrix is ​​replaced into the self-attention layer of the outer layer of the relational understanding diffusion model, so that the model focuses on action-related information in the generation phase.

[0082] The technical solution provided in this embodiment, the action understanding attention module (Action Relational Attention, ARA) is to enhance the accuracy and naturalness of the image generation model when processing scenes containing human actions. When working, first extract triple features from the control conditions, including human condition features, object condition features and action condition features. The triple features are spliced ​​with the fused feature map (which has been combined with the feature map of background and foreground information) to enhance the model's understanding of the relationship between the various elements in the action scene. The spliced ​​features are input into the ARA module for calculation to obtain the corresponding matrix. The calculated matrix is ​​used to replace the original matrix of the self-attention layer of the external layer in the relational understanding diffusion model. Through the replacement operation, the model can focus more on information related to the action in the generation stage, such as the posture of the character, the direction of the action, and the interaction between the character and the object, thereby generating more accurate and realistic action scene images.

[0083] The beneficial effect of this embodiment is that, through the specially designed action understanding attention module, the model can more deeply understand and learn the characteristics of character actions, improving the accuracy of action recognition and generation. Focusing on action-related information makes the generated image more natural and realistic in action execution, improving the overall quality of the image. The model can respond to control conditions more accurately, especially in complex action scenes, providing more detailed control of character actions in generated images.

[0084] In an exemplary embodiment, the triple feature is obtained by:

[0085] Extracting features of the control condition information through a self-attention layer according to the types of characters, objects, and actions, and obtaining a triplet feature representation of the control conditions including character conditions, object conditions, and action conditions;

[0086] The calculation formula for concatenating and calculating the control condition and the fusion feature map according to the triple feature representation is as follows:

[0087] Q h =γ h ·(SelfAttn h (c h )+c h ) (5)

[0088] Q o =γ o ·(SelfAttn o (c o )+c o ) (6)

[0089] Q a =γ a ·(SelfAttn a (c a )+c a ) (7)

[0090] Among them, Q h , Q o , Q a is the triple feature representation of the control condition, c h 、c o and c a are the control conditions of character type, object type and action type after the control condition information is disassembled, γ h , γ o , γ a is the adaptive weight and γ h , γ o The initial weight is less than γ a The initial weights are used to strengthen the model's response to action type c alearning;

[0091] The control condition and the fusion feature map are spliced ​​and calculated according to the triple feature to obtain a corresponding matrix, including:

[0092] The control condition is transformed dimensionally using a linear layer to align the dimensions of the fused feature map;

[0093] The treated control condition c h 、c o 、c a It is concatenated with the fusion feature map and input into the action understanding attention module for calculation to obtain the matrix F q 、F k With F v , the corresponding calculation formula is as follows:

[0094] F q =(X+Q h +Q a )·W q (8)

[0095] F k =(X+Q o +Q a )·W k (9)

[0096] F v =(X+Q h +Q a )⊙(X+Q o +Q a )·W v (10)

[0097] Among them, ⊙ is element-by-element multiplication, · is matrix transformation, W q , W k and W v is the weight matrix, and X is the fused feature map.

[0098] The technical solution provided by this embodiment is that the action understanding attention module (ARA module) enhances the image generation model's ability to understand and generate action-related features by processing triple features. Specifically, the control condition information is broken down into three parts: person, object, and action, and features are extracted through the self-attention layer to obtain triple feature representations. Then, the extracted features are adjusted through adaptive weights, where γ h ,γ o The initial weight is less than γ a The initial weight of a The linear layer can be used to control the condition c h 、c o 、ca The dimensions are transformed to ensure that they are aligned with the dimensions of the fused feature map X.

[0099] The processed control conditions are concatenated with the fusion feature map X, and then input into the ARA module for calculation to obtain the matrix F q 、F k With F v .

[0100] Specifically, the ARA module calculates F through formulas (8), (9) and (10): q 、F k With F v . Among them, element-by-element multiplication (⊙) and matrix transformation (·) are used to combine the feature map and control condition information. The calculated matrix F q 、F k With F v It is used to replace the original matrix of the self-attention layer in the relational understanding diffusion model, allowing the model to focus more accurately on action-related information during the generation phase.

[0101] The beneficial effect of this embodiment is that, through the specially designed action understanding attention module, the model can more deeply understand and learn the characteristics of human actions, improving the accuracy of action recognition and generation. The model can more accurately focus on action-related information during the generation phase, thereby improving the authenticity and action consistency of the generated image.

[0102] That is to say, in the method embodiment, the control condition information input by the user is processed using the relational understanding diffusion model to obtain the character action target image. It can be understood that the process of generating actions by the relational understanding diffusion model can be decoupled into two parts:

[0103] The first part is that according to the manually provided bounding box, the model needs to generate corresponding people and objects in the specified area and ensure the degree of fit between the people and the bounding box, so that the generated image is more consistent with the expected spatial distribution;

[0104] The second part is that when generating actions, there must be a real and reasonable relationship between the character and the action object. For example, when the model describes the action of "feeding the cat", the character and the cat must face each other, and the character needs to hold food in his hand and is feeding or preparing to feed the cat, so as to accurately express the action of "feeding the cat". If the character and the cat are only generated within the bounding box, but lack the corresponding interactive relationship, an effective action cannot be formed.

[0105] The relational understanding diffusion model optimizes the process of LDM generating images for the above two parts respectively, aiming to improve the accuracy and consistency of the model when generating character action images.

[0106] Specifically, first, the model generates a corresponding mask based on the input bounding box, and integrates the characteristics of the LDM middle layer network that pays more attention to semantic information and structural information, and introduces the mask into the middle layer of LDM. In this way, the mask can impose spatial constraints on the image generation process, so that the generated characters and objects are more in line with the expected position requirements.

[0107] Secondly, in order to solve the problem that the model lacks interactive learning with control conditions when generating actions, the self-attention layer of the external layer of LDM is improved. The traditional self-attention layer mainly learns the information of the image itself, but lacks a deep interactive understanding of external control conditions (such as text and bounding boxes). To this end, the technical solution provided in this application integrates text information and bounding box information in the input of the self-attention layer, thereby improving the model's ability to understand the semantics of actions and model the relationship between people and objects. At the same time, since the external layer of LDM pays more attention to texture information, adjustments can focus on the self-attention layer of the external layer.

[0108] The positive effects of the technical solution to be protected by this application include at least:

[0109] During the generation process, a mask mechanism is used to impose targeted constraints on the bounding boxes of people and action objects, so that the positions of people and action objects generated by the model are more in line with expectations; an action understanding attention module is designed and introduced to enhance the model's ability to understand actions by performing in-depth analysis of action information in text description information, making the actions of people in the generated images more realistic and their relationship with action objects more natural.

[0110] Model Embodiment.

[0111] An embodiment of the present application provides a relationship understanding diffusion model, which is used for the spatial constraint action understanding generation method described in any one of the method embodiments. The relationship understanding diffusion model is composed of a diffusion model backbone network, a spatial control module and an action understanding attention module. The specific implementation method of the relationship understanding diffusion model is consistent with the implementation method and technical effect achieved by the relationship understanding diffusion model recorded in the above-mentioned method implementation method, and some contents will not be repeated here.

[0112] In an exemplary embodiment, the training method of the relational understanding diffusion model includes:

[0113] A network training set is obtained, wherein the network training set includes a plurality of training pictures including bounding box information and text description information, wherein the training pictures are training pictures of interaction between people and objects taken in the real world, and the bounding boxes indicated by the boundary information are manually marked; and the relationship understanding diffusion model is trained using the network training set.

[0114] As an example, see Figure 2 , provides a relational understanding diffusion model (RelDiffusion), which consists of a diffusion model backbone network, a spatial control module, and an action understanding attention module. The backbone network of the relational understanding diffusion model is a variant of Unet, and each layer consists of a self-attention layer and a cross-attention layer to predict the noise added during the image generation process. The spatial control module guides the model to generate target people (person in the figure) and objects (cat in the figure) within the bbox (Bounding Box) by integrating the mask of the instance into the feature map. The action understanding guidance module adjusts the calculation method of the self-attention layer to fuse the text description (Text Caption) information (A person is feeding a cat) and the bounding box information into the image features, focusing on extracting and learning action-related information from it.

[0115] Regarding the backbone network of the diffusion model, in the image generation task, a strong feature representation is the key to ensure the generation of high-quality images. In related technologies, Unet, as a classic image segmentation network, has been shown to perform well in a variety of visual tasks. It can be considered that the success of Unet is attributed to its unique encoding-decoding structure and efficient feature fusion mechanism. Therefore, in the relational understanding diffusion model, a backbone network with a Unet structure is adopted.

[0116] The structure of Unet can be regarded as a typical encoder-decoder architecture, in which the encoder part implements downsampling through convolutional layers and is responsible for gradually extracting high-level features of the image, while the decoder restores the potential spatial resolution of the image through gradual upsampling. The original Unet network is usually composed of convolutional neural networks, which is sufficient for many image segmentation tasks, but considering that such a network structure is not enough to generate high-quality images in image generation tasks, in the relational understanding diffusion model mentioned in this application, each layer of Unet has been improved and replaced with a module composed of a self-attention layer and a cross-attention layer, so that the network can more effectively extract image and text features, thereby more accurately predicting noise and improving the quality and consistency of generated images. At the same time, in order to enhance the representation ability of the network, Unet uses jump connections between each layer to pass low-level features directly to the decoder, which not only helps the model retain more detail information, but also effectively avoids the loss of details caused by over-compression of information during the generation process.

[0117] In summary, within the framework of the relational understanding diffusion model, the Unet backbone network is used to predict the noise in the image generation process and gradually improve the generation effect through multi-level information transfer, ultimately producing a clear and expected image.

[0118] For the spatial control module, in the character interaction action generation, the user will provide control condition information for the relational understanding diffusion model. The control condition information includes text description information (global text prompt t) and bounding box information. The bounding box information includes: character bounding box and action object bounding box in And the corresponding character and action object description for each bounding box

[0119] Based on user input (to provide control condition information), the model needs to generate an image I. During the generation process, ensure that each bounding box B in the image h and B o Endogenous generation and description and Consistent characters or action objects, and an action relationship that conforms to the global text prompt t is generated between each pair of character-action objects. When processing bounding box information, the model in the related art usually processes the bounding box information of the image modality as an embedding similar to the text modality, and finally integrates it with the text information through the cross-attention layer to integrate the image information. Obviously, the conditional control of the text modality will cause the position of the characters in the generated image to be inconsistent with the input bounding box, resulting in a phenomenon similar to semantic loss. To solve this problem, this example introduces a mask-guided spatial control module (MGSC) in the network, which converts the input bounding box information into a mask and integrates it into the feature map. In this way, the mask provides clear position information in the image modality, helping the network to more accurately generate the expected character position, thereby effectively improving the accuracy of position matching.

[0120] Specifically, see Figure 3 , Figure 3 It is a schematic diagram of a space control module provided in an embodiment of the present application.

[0121] The spatial control module receives the bounding box information from the model input and the image features (feature map) extracted by the previous Unet network. First, the input character bounding box B is calculated. h and the action object bounding box B o The overlapping part of the action bounding box B is obtained a , and then merge the three bounding boxes into a triple bounding box In order to align the dimensions of image features, MGSC first downsamples the bounding box using linear interpolation, downsamples the bounding box size to 64×64, and generates corresponding masks at corresponding positions according to the bounding box coordinates. After the mask is generated, the feature map X is then constrained by masking. The mask and feature map of the same size are fused by element-by-element multiplication. The fused feature map is decomposed into multiple independent feature maps. Each independent feature map focuses on a type of information in a person, object, or action. Subsequently, feature maps of different categories are input into the corresponding attention modules: person and object feature maps enter the instance attention layer (Instance Attention, corresponding to Formula 2), and action feature maps enter the action attention layer (Action Attention, corresponding to Formula 3) to generate each instance feature. and action features

[0122] Since the feature map generated separately only contains foreground information and lacks background features, a fused attention layer (corresponding to Formula 4) can be introduced. The fused attention layer complements the integrity of the scene by integrating the foreground feature map and background information. The background features are provided by the original image features X input by the spatial control module. Finally, MGSC generates a fused feature map X combined with mask constraints, realizing the organic combination of foreground and background information, and ensuring the accuracy of the generated image in terms of spatial layout and semantic consistency.

[0123] For ease of understanding, the above process can be described by formulas (1), (2), (3), and (4) as follows:

[0124]

[0125] Where ⊙ represents the element-by-element multiplication of matrices and + represents the addition of matrices. The spatial control module is used in the middle four layers of the Unet network.

[0126] For the action understanding attention module, considering the performance of the self-attention mechanism in computer vision tasks, it can fully capture the information inside the image. However, the related technologies lack the ability to pay precise attention to certain specific information, especially in tasks involving complex semantic scenes, such as understanding and generating character actions. To make up for this deficiency, this example proposes a new attention calculation method called Action Relational Attention (ARA), which aims to enhance the model's ability to understand and generate character actions. It can be understood that the core idea of ​​ARA is to introduce the control condition c into the self-attention calculation to strengthen the model's learning performance on action-related features.

[0127] Specifically, we first disassemble the control condition c into three parts: c h (Character), c o (object), c a(Action). This decomposition method conforms to the natural hierarchical structure of action understanding, that is, the character is the subject of the action, the object is the associated element of the action, and the action itself is the main target information.

[0128] Then, the control conditions of each part are extracted through the self-attention layer, and finally the triple feature representation Q of the control conditions is obtained. h ,Q o ,Q a The calculation formulas (5), (6) and (7) are as follows:

[0129] Q h =γ h ·(SelfAttn h (c h )+c h )(5)

[0130] Q o =γ o ·(SelfAttn o (c o )+c o ) (6)

[0131] Q a =γ a ·(SelfAttn a (c a )+c a ) (7)

[0132] Among them, γ h ,γ o ,γ a is the adaptive weight, γ h ,γ o The initial weight is set to 1.0, γ a The initial weight is 1.5, which is used to strengthen the model's response to action c a of learning.

[0133] Then, the treated control condition c h , c o , c a Concatenate it with the fusion feature map X and input it into ARA to calculate and get the matrix F q 、F k With F k The calculation formulas (8), (9), and (10) of ARA are as follows:

[0134] F q =(X+Q h +Q a )·W q (8)

[0135] F k =(X+Q o +Q a )·W k (9)

[0136] F v =(X+Q h +Q a )⊙(X+Q o +Q a )·W v (10)

[0137] Among them, ⊙ is the element-by-element multiplication and · is the matrix transformation. Before entering the ARA calculation, a linear layer can be used to change the dimension of the control condition c to align the dimension of the feature map x.

[0138] Finally, the matrix F calculated by ARA is q 、F k 、F v By replacing the self-attention layer in the outer layer of the relational understanding diffusion model, this improvement enables the model to focus more accurately on action-related information during the generation phase, thereby improving the realism and action consistency of the generated images.

[0139] As another example, experimental results and analysis using the relational understanding diffusion model provided by the present application are provided. The tracker in the experiment was implemented using Python 3.8 and PyTorch 2.0.1 framework, and the experiment was conducted on a graphics processing unit (GPU).

[0140] The network training set comes from the large-scale human-object interaction dataset HICO-DET, which contains 47,776 pictures of human-object interaction actions taken in the real world. The interaction action triples are divided into 600 categories, consisting of 80 object categories and 117 gerund categories. The bounding boxes of the targets are all manually marked, with a total of more than 15 action interaction annotations. The dataset is preprocessed in the same way as the baseline model Interact Diffusion, and the training image size and the generated image size are both 512×512. During the training process, the diffusion model backbone network is initialized by the LDM model pre-trained on the LAION-400M dataset. The specific process is not repeated here.

[0141] This example uses the adaptive momentum-based adam optimizer, with an initial learning rate of 0.000005 and a learning rate warm-up of 10,000 steps. The entire process includes training on 1 GPU, 1,000,000 iterations, and the GPU processes 8 images per iteration, which is approximately equal to 106 rounds of training.

[0142] During the test, we first generated the corresponding text prompts based on the HOI triples provided by the training set, and then generated 9658 images of size 512×512 in combination with the bounding boxes of the training set. For the images generated by the model, we used the swin_tiny pre-trained weights of the FGAHOI model to test the specific locations of objects and actions generated in the images. The images in the test set were uniformly cropped and resized to 512×512 pixels, and the quality of the generated images was evaluated using FID and KID, and the similarity of the data distribution between the generated images and the original test set images was calculated.

[0143] RelDiffusion was evaluated on the public human behavior interaction recognition dataset HICO-DET in comparison with other methods (StableDiffusion, GLIGEN, GLIGEN*, InteractDiffusion). In order to further improve the generation capability of the diffusion model under specific conditions, external control information (such as human posture, edge map, etc.) can be introduced through Control Net, so that the diffusion model can generate images that meet specific requirements. By adding a replica of the Unet network to LDM and fine-tuning the replica separately, the model can better capture the target content and structure when generating images, thereby achieving further optimization in generation quality and control accuracy. GLIGEN adds a pluggable module to LDM to learn the control conditions of external input, showing strong flexibility in complex scenarios.

[0144] Interact Diffusion further improves the model's ability to understand the interactions between human actions and targets in images based on GLIGEN. This method combines bounding box information with text prompts, and uses triples to represent (human, action, object), which strengthens the model's semantic understanding and interactive learning capabilities when generating human actions. By introducing this interactive mechanism, Interact Diffusion solves the problem of modeling the relationship between human and action objects in the generation process to a certain extent, and generates more expected high-quality action images.

[0145] Detailed experimental results are shown below:

[0146] Human interaction detection is evaluated using the most advanced model FGAHOI. The model detects the generated images and compares them with the labeled data in the HICO-DET dataset to calculate the mAP score. The mAP score is divided into two categories: default settings and known objects. The default setting is more challenging because it requires the model to distinguish irrelevant objects in the image.

[0147] For the quality of generated images, we use the commonly used evaluation indicators FID and KID in the field of image generation. They quantify the quality of generated images by comparing the similarity between generated images and real images. The detailed experimental results are shown in Tables 1 and 2. In terms of the two evaluation indicators of image quality and task interaction detection, it is proved that RelDiffusion can meet the expected requirements.

[0148] Table 1 Image quality score results under HICO-Det dataset

[0149]

[0150] Table 2 Experimental results of different diffusion models under FGAHOI benchmark

[0151]

[0152] Obviously, the relational understanding diffusion model provided in this example is used for the spatial constraint action understanding generation method. By introducing a mask mechanism, the control conditions are directly integrated into the image features, and the action understanding attention module is used to guide the model to learn the character action information more efficiently. It can make full use of the input control conditions to achieve precise control of the final generated image, while retaining the global perception ability of the self-attention mechanism. By introducing the hierarchical features of the control conditions, the model's understanding of the action is more delicate and accurate. The above experimental evaluation proves that the above method has significant advantages in generation effect and action consistency, and verifies the effectiveness of the network. Furthermore, through this method, the input control condition information can be fully utilized to control the final generated image.

[0153] Storage Medium Embodiments.

[0154] This embodiment provides a computer-readable storage medium on which instructions are stored. When the instructions are executed by a processor, the processor implements any method described in the method embodiment. The specific embodiments are consistent with the embodiments and technical effects recorded in the above method embodiments, and some contents are not repeated here.

[0155] Electronic device embodiments.

[0156] This embodiment provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and when the computer program is executed by the processor, the steps of the method described in any one of the method embodiments are implemented.

[0157] See also Figure 4 , Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0158] The electronic device 10 may include, for example, at least one memory 11 , at least one processor 12 , and a bus 13 connecting different platform systems.

[0159] The memory 11 may include a (computer) readable medium in the form of a volatile memory, such as a random access memory (RAM) 111 and / or a cache memory 112 , and may further include a read-only memory (ROM) 113 .

[0160] The memory 11 also stores a computer program, which can be executed by the processor 12 so that the processor 12 implements the steps of any of the above methods.

[0161] The memory 11 may also include a utility 114 having at least one program module 115, such program module 115 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0162] Accordingly, the processor 12 may execute the above-mentioned computer program, and may execute the utility 114 .

[0163] The processor 12 may adopt one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic components.

[0164] The bus 13 may be a local bus representing one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or any of a variety of bus architectures.

[0165] The electronic device 10 may also communicate with one or more external devices such as a keyboard, a pointing device, a Bluetooth device, etc., and may also communicate with one or more devices that can interact with the electronic device 10, and / or communicate with any device (such as a router, a modem, etc.) that enables the electronic device 10 to communicate with one or more other computing devices. Such communication may be performed through an input / output interface 14. In addition, the electronic device 10 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter 15. The network adapter 15 may communicate with other modules of the electronic device 10 through the bus 13. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 10 in actual applications, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0166] Computer Program Product Embodiments.

[0167] The embodiment of the present application also provides a computer program product, the specific embodiments of which are consistent with the embodiments recorded in the above method embodiments and the technical effects achieved, and some contents will not be repeated here.

[0168] The computer program product includes a computer program, and when the computer program is executed by at least one processor, the steps of any of the above methods are implemented. The computer program product can be a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the computer program product of the present invention is not limited to this, and the computer program product can be any combination of one or more computer-readable media.

[0169] It should be noted that, in the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b and c can be single or multiple. It is worth noting that "at least one" can also be interpreted as "one or more items".

[0170] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are configured to distinguish similar objects, and do not have to be configured to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0171] This application is explained from the perspectives of purpose of use, effectiveness, progress and novelty, and has met the functional enhancement and usage requirements emphasized by the Patent Law. The above description and drawings of this application are only the preferred embodiments of this application, and are not intended to limit this application. Therefore, all structures, devices, features, etc. that are similar or identical to this application, that is, all equivalent replacements or modifications made in accordance with the scope of the patent application of this application, should fall within the scope of protection of the patent application of this application.

Claims

1. A spatially constrained action understanding generation method, characterized in that: The method comprises: Acquiring control condition information, wherein the control condition information includes bounding box information and text description information; Inputting the control condition information into a trained relational understanding diffusion model to obtain a character action target image; Among them, the relationship understanding diffusion model is composed of a diffusion model backbone network, a spatial control module and an action understanding attention module. The diffusion model backbone network is used to predict the noise added in the image generation process, and finally generate the target image through multi-level information transmission; the spatial control module is used to guide the model to generate target people and objects within the bounding box by integrating masks into the feature map; the action understanding attention module is used to fuse text descriptions and bounding box information into image features by adjusting the calculation method of the self-attention layer, and extract and learn action-related information therefrom.

2. The spatial constraint action understanding generation method according to claim 1, characterized in that: The diffusion model backbone network adopts the Unet network structure, and each layer consists of a self-attention layer and a cross-attention layer; jump connections are used between each layer to directly transfer low-level features to the decoder.

3. The spatial constraint action understanding generation method according to claim 2, characterized in that: The bounding box information includes a character bounding box and an action object bounding box, and the space control module is used to: For the continuous multiple middle layers of the Unet network structure, receiving the image features extracted by the previous Unet network and the bounding box information of the model input, and obtaining the overlapping part of the person bounding box and the action object bounding box through the bounding box information to obtain the action bounding box; merging the person bounding box, the action object bounding box and the action bounding box into a triple bounding box; Downsample each bounding box using linear interpolation, downsample the bounding box to a predetermined size, and generate a mask at a corresponding position according to the bounding box coordinates; The feature map is subjected to mask constraints according to the mask, and the mask and the feature map of the same size are fused by element-by-element multiplication, and the fused feature map is decomposed into multiple independent feature maps, each of which is used to focus on a type of information in a person, an object, or an action; Input each independent feature map into the corresponding attention module according to the category of aggregated information to obtain instance features and action features; According to the instance features and the action features, the foreground feature map and the background features are integrated through a fusion attention layer to supplement the completeness of the scene, and a fusion feature map combined with mask constraints is obtained; wherein the background features are provided by the image features.

4. The spatial constraint action understanding generation method according to claim 3, characterized in that: The step of inputting each independent feature map into the corresponding attention module according to the category of the aggregated information to obtain instance features and action features includes: The independent feature maps of clustered people and objects are input into the instance attention layer to generate instance features; the independent feature maps of clustered actions are input into the action attention layer to generate action features.

5. The spatial constraint action understanding generation method according to claim 3, characterized in that: The triplet feature includes a person condition feature, an object condition feature and an action condition feature, and the action understanding attention module is used to: Acquire the triplet feature, and concatenate and calculate the control condition and the fusion feature map according to the triplet feature to obtain a corresponding matrix; The calculated matrix is ​​replaced into the self-attention layer of the outer layer of the relational understanding diffusion model, so that the model focuses on action-related information in the generation phase.

6. The spatial constraint action understanding generation method according to claim 5, characterized in that: The method of obtaining the triple feature includes: Extracting features of the control condition information through a self-attention layer according to the types of characters, objects, and actions, and obtaining a triplet feature representation of the control conditions including character conditions, object conditions, and action conditions; The calculation formula for concatenating and calculating the control condition and the fusion feature map according to the triple feature representation is as follows: Q h =γ h ·(selfAttn h (w h )+c h ) (5) Q o =γ o ·(selfAttn o (w o )+c o ) (6) Q a =γ a ·(selfAttn a (w a )+c a ) (7) Among them, Q h ,Q o ,Q a is the triple feature representation of the control condition, c h 、c o and c a are the control conditions of character type, object type and action type after the control condition information is disassembled, γ h , γ o , γ a is the adaptive weight and γ h , γ o The initial weight is less than γ a The initial weights are used to strengthen the model's response to action type c a of learning.

7. The spatial constraint action understanding generation method according to claim 6, characterized in that: The control condition and the fusion feature map are spliced ​​and calculated according to the triple feature to obtain a corresponding matrix, including: The control condition is transformed dimensionally using a linear layer to align the dimensions of the fused feature maps; The treated control condition c h 、c o 、c a It is concatenated with the fusion feature map and input into the action understanding attention module for calculation to obtain the matrix F q 、F k With F v , the corresponding calculation formula is as follows: F q =(X+Q h +Q a )·W q (8) F k =(X+Q o +Q a )·W k (9) F v =(X+Q h +Q a )⊙(X+Q o +Q a )·W v (10) Among them, ⊙ is element-by-element multiplication, · is matrix transformation, W q , W k and W v is the weight matrix, and X is the fused feature map.

8. A relational understanding diffusion model, used to implement the spatial constraint action understanding generation method according to any one of claims 1 to 7, characterized in that: The relationship understanding diffusion model consists of a diffusion model backbone network, a spatial control module and an action understanding attention module.

9. An electronic device, characterized in that: The electronic device comprises a memory and at least one processor, the memory stores a computer program, and the at least one processor is configured to implement the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by at least one processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images