Character interaction detection method based on spatial fine-grained context interaction feature fusion

Abstract

The invention relates to the technical field of computer vision, and discloses a figure interaction detection method based on spatial fine-grained context interaction feature fusion, which comprises the following steps of: firstly, performing target detection on an input image to obtain a target detection result set and a person-object pairing feature; performing gridding projection on the image to obtain image global features, and inputting the image global features into a spatial fine-grained feature learning module to obtain spatial fine-grained features; inputting the spatial fine-grained features and the human-object pairing features into a spatial context interaction feature fusion module to obtain spatial context interaction features, and inputting the spatial context interaction features and the spatial fine-grained features into a visual encoder to obtain enhanced human-object pairing features; obtaining an interaction category score according to the feature and a text embedding feature of a character interaction category; and iteratively optimizing the character interaction detection model until convergence. According to the method, the description of local details of a visual space is enhanced, the context relationship between a person and an object is enhanced, and the accuracy of person interaction detection is improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for detecting human interaction based on spatial fine-grained contextual interaction feature fusion. Background Technology

[0002] With the rapid development of embodied intelligence technology, intelligent agents equipped with robotic arms, mobile platforms, or multiple sensors are increasingly being applied to scenarios such as home services, assisted operation, and object manipulation. To perform tasks safely, reliably, and efficiently in real-world environments, embodied intelligence systems must be able to accurately understand scene structure from single-frame visual input, particularly recognizing the interaction relationships between people and objects, such as handing over objects, picking up tools, and touching devices. These interactions not only reflect operational intentions and object uses but also directly influence the agent's action planning and task execution, representing crucial semantic information in the embodied intelligence decision-making chain. In many practical deployment scenarios, due to sensor bandwidth, computational latency, or privacy limitations, embodied intelligence systems cannot rely on long-term video sequences for analysis and often can only complete semantic judgments based on single-frame images. This necessitates that HOI (Human-Object Interaction) detection models extract fine-grained spatial structure, local object attributes, and the contextual relationships between people and objects from limited visual cues. However, existing human interaction detection methods typically rely on global visual features or shallow semantic patterns, which can easily lead to inaccurate interaction category predictions when dealing with scenarios with severe occlusion, blurred object details, or significant changes in interaction posture.

[0003] To address this challenge, existing methods mainly fall into three categories: First, methods based on instance detection and global context modeling. These typically employ a two-stage paradigm, first using object detection networks (such as Faster R-CNN) to detect all human and object instances in an image, then fusing visual features, spatial location information (such as relative coordinates of bounding boxes), and global scene context to learn joint representations between humans and objects, and finally using a classifier to determine interaction relationships. Feature fusion methods are diverse, including simple feature concatenation, attention-based weighted fusion, or designing specialized interaction proposal networks to generate candidate pairs. Second, methods incorporating prior knowledge of human anatomy and language. To provide richer discriminative cues, some studies use keypoint representations of human posture as input, using positional encodings of parts such as hands and torso to assist in judging interaction actions. Simultaneously, using verb-noun semantic relationships (language priors) learned from large-scale corpora to constrain and correct visual prediction results has also become a common strategy to improve model performance. Thirdly, there is the structured relationship reasoning method based on graph neural networks. Generally, the interaction scene is modeled through a graph structure, where people and objects are nodes and the interaction relationships between nodes are edges. Through graph neural networks, multiple rounds of message passing and feature aggregation are performed between nodes to explicitly model the interdependencies between all instances, thereby inferring the final category of human interaction. The graph can be constructed based on visual similarity, spatial proximity, or semantic association, aiming to capture complex spatial structural contextual relationships.

[0004] The limitations of the current method are mainly reflected in the following two points: 1) The model's local perception capability is insufficient, making it difficult for the visual encoder to capture the texture features and spatial details of key areas. For example, for small objects, tools, or hand details in interactive actions, there is a lack of dedicated local enhancement mechanisms, causing the visual representation to be biased towards the background and ignore key semantics. At the same time, relying solely on the broad semantic knowledge contained in the pre-trained model of a large model cannot accurately depict the mutual influence between people and objects in different interactions in the spatial dimension. For example, different hand trajectories, contact points, or different functional areas of objects (such as the different interaction areas of pushing a cart and riding a bicycle) will result in weak discrimination of detailed features of interaction categories. 2) Existing methods are often too coarse-grained when capturing the semantic interaction between people and objects. For example, they associate human and object regions through feature splicing or attention layers, which cannot express more complex interaction dependencies and is difficult to handle the subtle differences between human actions and object states. When the model needs to identify patterns with similar semantics but different interaction categories (such as pushing and pulling doors), it lacks more complex interaction semantic modeling.

[0005] Based on the above analysis, there is an urgent need to construct a new detection framework that can simultaneously enhance local spatial perception and human interaction modeling capabilities, enabling it to accurately identify human relationships under single-frame image input conditions, and providing reliable technical support for real-time interactive task planning of embodied intelligence systems. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, the present invention aims to provide a human interaction detection method based on spatial fine-grained contextual interaction feature fusion. This method can not only significantly enhance the visual encoder's ability to perceive local object details, spatial layout and structural texture, but also effectively capture the contextual relationship between people and objects through interactive semantic reasoning. This enables the model to more accurately distinguish multiple interaction modes and subtle action differences, thereby improving the detection accuracy of known interaction categories and the robustness of the model in complex scenes, and ensuring the accuracy and stability of human interaction detection results in diverse application scenarios.

[0007] This invention employs the following technical solution: a method for detecting human interaction based on spatial fine-grained contextual interaction feature fusion, comprising the following steps: Step 1: Initialize the object detector, text encoder, and visual encoder; input the image into the object detector to obtain the object detection result set; obtain human-object pairing features based on the object detection result set; perform gridded projection on the image to obtain global image features; Step 2: Construct a spatial fine-grained feature learning module, with the input being the set of global image features and object detection results, and the output being spatial fine-grained features; Step 3: Construct a spatial context interaction feature fusion module, with inputs being fine-grained spatial features and human-object pairing features, and output being spatial context interaction features; Step 4: After generating a sentence description for the character interaction category name using a predefined text template, obtain the text embedding features through a text encoder; input the spatial fine-grained features and spatial context interaction features into the visual encoder to obtain enhanced person-object pairing features; calculate the similarity between the text embedding features and the enhanced person-object pairing features to obtain the character interaction category score; Step 5: Calculate the focus loss function based on the human interaction category score and the actual interaction category. Use the stochastic gradient descent algorithm to train the human interaction detection model, which consists of a spatial fine-grained feature learning module, a spatial context interaction feature fusion module, a visual encoder, and a text encoder. For other images, execute steps 1 to 4 in sequence to obtain the spatial position of all people and objects and their paired interaction categories until the human interaction detection model converges, thus completing the training of the human interaction detection model. Step 6: Input the new image into the trained human interaction detection model to obtain the predicted human bounding box, object bounding box, and interaction category.

[0008] Furthermore, step 1 specifically includes: Step 11: Initialize the object detector, CLIP text encoder, CLIP visual encoder, and predefined text templates for the input image. Detection was performed using a pre-trained DETR object detector, and the results were obtained. The set of detection results is denoted as the target detection result set. The bounding box coordinates Represented as Target category Belongs to the category set , Total number of target categories; confidence score The target global feature vector output by the target detector decoder is ; Represents the first element in the set of target detection results. One goal; Step 12: Person-object screening and candidate construction for person-object pairing: From the target detection result set Targets with the category "human" are selected from the data, and a human index set is constructed. Index for each person , with all other target indexes ( and By forming pairs, a set of person-object pairing indices is obtained. The total number of pairs is ; Step 13: Obtaining Person-Object Pairing Features: Extracting the person-object pairing index set. The first in A pair Extracting global human feature vectors from the target detection result set and the global feature vector of the object The two are concatenated and then mapped to paired features through a two-layer feedforward network: , This is a splicing operation; among which The feedforward neural network function contains two fully connected layers. The weights of the first layer are... The weight of the second layer is All paired features constitute person-object paired features. Its submatrices are all stored column-wise; Step 14: Global Image Feature Acquisition: The input image... It is divided into a non-overlapping, fixed-size grid, with each grid cell being 16×16 pixels, thus forming The grid in which and Depending on the input image resolution; each grid is then flattened and projected onto... Dimensions, after adding location information, yield global image features. Simultaneously, a learnable class embedding vector is randomly initialized. Used to summarize global image features; to pair human-object features global features of the image With learnable category embedding vectors The complete sequence is obtained by splicing. This sequence will be used for subsequent layer-by-layer processing.

[0009] Furthermore, step 2 specifically includes: Step 21: The spatial fine-grained feature learning module includes: spatial feature projection, spatial location embedding, multi-scale convolutional aggregation, and multi-scale fusion with spatial location features; Step 22: Spatial Feature Projection: Projecting global features of the image Projection as spatial mapping image features Then, a dimension-reduced FFN with weights is applied. ,Will Projected onto a more compact dimension ,Right now ; Step 23: Spatial Location Embedding: For each spatial location in the spatially mapped image features From the set of target detection results Find the detection target index that covers this location. Extract bounding box coordinates Confidence score and target category Input the target category into the CLIP text encoder to obtain the text embedding. ;Will , , After concatenating the three elements, the spatial location embedding vector is obtained through FFN mapping. Apply the above steps to each spatial location to obtain the spatial location embedding matrix. ; and Spatial embedding features are obtained after fusion and layer normalization. , This is a layer normalization function, used to obtain spatial location information of objects in the image; Step 24: Multi-scale convolutional aggregation: Create Convolutional layers with different kernels The set of convolution kernel sizes is (generally ); Spatial embedding features use Different types of convolutions, with stride=1 and padding to maintain spatial dimensions, yield multi-scale feature sets. Each scale is mapped to FFN. Multi-scale spatial perception features are obtained by combining and summing the dimensions. ; Step 25: Multi-scale and spatial location feature fusion; integrating multi-scale spatial perception features Projection Back Dimensions: Through weighted residual connection: ,in This indicates element-wise multiplication. These are learnable parameters; the optimal weights are automatically learned during training. It represents a fine-grained spatial feature.

[0010] Furthermore, step 3 specifically includes: Step 31: The spatial context interaction feature fusion module includes a region feature extraction module, an interaction context extraction module, an interaction pattern capture module, and a spatial interaction feature fusion module. By analyzing the association patterns between people and object regions, it identifies the occurrence and mode of interaction. Step 32: Region Feature Extraction Module: Extracting the Person-Object Pairing Index Set The first in Personal-item pairing From the perspective of spatial fine-grained features Extracting Region Features – Using ROI Align Operation , ,in and These are the bounding boxes for people and objects, respectively. These are the human region features and the object region features, respectively; then, the compact region features of the human and object are obtained through FFN. , Flatten is the operation for flattening; Step 33: Interaction Context Extraction Module: For the first step obtained in step 32... Person-object pairing of compact region features of people and objects, constructing A learnable query Extracting contextual information related to human-object interaction through cross-attention. and : , Cross-attention calculation ,in and Depend on or Obtained by linear transformation; , Used to capture interaction-related features and filter out irrelevant details; Step 34: Interaction Mode Capture Module: For the first step obtained in step 33... Person-object pairing involves contextualizing person-object interactions and calculating interaction features through cross-attention. , ; Indicates for the first Person-object pairing integrates human interaction features with object characteristics; This indicates object interaction features that incorporate human characteristics; this operation captures the interaction patterns between humans and objects. Step 35: Interaction Feature Enhancement Module: Similarly, for the first... Person-object pairing, extracting pairing features Dimensionality reduction is obtained by linear mapping. ;by As a query, it extracts enhanced interaction features from human-object interaction features; human enhanced interaction features Enhanced interactive features of objects The two enhanced interaction features mentioned above are concatenated and projected back. Dimensions followed by learnable parameters Weighted residual connection ; All processed After matching features, spatial context interaction features are obtained. .

[0011] Furthermore, step 4 specifically includes: Step 41: Insert the spatial fine-grained feature learning module from Step 2 and the spatial context interaction feature fusion module from Step 3 into multiple levels of the CLIP visual encoder. Specifically, in the first step... Layer CLIP visual encoder Previously, the global features of the image were used as the output of the previous layer, which is also the input of the current layer. Human-object pairing characteristics and learnable category embeddings The spatial fine-grained feature learning module is called to obtain... The spatial context interaction feature fusion module obtains and will Learnable category embeddings and After sequential assembly, it is sent to the first... The CLIP visual encoder of the layer obtains ,in and These are enhanced human-object pairing features and global image features, respectively. Step 42: Generate interactive category text embeddings: Obtain the set of interactive categories from the dataset. Number of categories For each interaction category Based on the category verb and object category, a templated text description, "A photo of a person [verb-ing] a / an [object]", is generated using a predefined text template. The CLIP text encoder then converts this text sequence into a token ID sequence using a tokenizer. ,in For the first The number of tokens in the text class. Use the token ID; insert K learnable tokens before the text: assign a trainable vector representation to each learnable token. This leads to a set of learnable tokens. The aforementioned learnable token is inserted before the text token embedding to form an extended text sequence. ,in This represents the word embedding layer of the CLIP text encoder, used to obtain text token embeddings; finally, the CLIP text encoder, which expands the text sequence input, obtains the word embedding. The final text embedding vectors for each category , For text encoders; text embedding vectors of each category constitute the text embedding features of the character interaction category. Each of its submatrices is stored column-wise; Step 43: Calculate the person-object interaction classification score: Extract the person-object pairing features from the last layer of the CLIP visual encoder. Computation and text embedding features cosine similarity ,in For temperature parameters; specifically, Indicates the first Personal-object pairing and the first Similarity between categories; probability matrix of interaction category scores obtained through sigmoid activation. ,in For the first Personal-object pairing belongs to the first The probability of each category.

[0012] Furthermore, step 5 specifically includes: Step 51: Construct training labels: For the training images, the first... Person-object pairing, if a pair of ground-truth bounding boxes exists Make and Then it is considered that the person-object pairing matches the ground-truth, and if the match is found and the ground-truth label contains a category. ,make Otherwise Thus, a binary label matrix is ​​constructed. ; Step 52: Calculate the focus loss: Use the focus loss to mitigate class imbalance. The formula is as follows: , in, , For balancing parameters, For focusing parameters; Step 53: Model Training and Convergence: Train the model on the given image dataset and its included human interaction categories using stochastic gradient descent. The model consists of a spatial fine-grained feature learning module, a spatial context interaction feature fusion module, a visual encoder, and a text encoder. In each iteration, randomly sample a batch of images and execute steps 1 to 4 to obtain the human interaction category score probability matrix. Calculate the loss The parameters are updated by backpropagation; training stops when the validation set loss does not decrease for five consecutive rounds; training uses only seen categories, and the human interaction detection model generalizes to unseen categories through the semantic space of CLIP.

[0013] Furthermore, step 6 specifically includes: Input a new image and execute steps 1 to 4 to obtain the probability matrix of character interaction categories. The final interaction confidence is obtained by incorporating the detection confidence. ,in and Confidence scores for people and objects are determined based on the first... Personal-item pairing From the object detection result set Obtained from; For hyperparameters; pairing each person with another person Select the category with the highest score: ;like Output , These are the bounding boxes for the detected people and objects, respectively. For the first in the set of interaction categories Categories For interactive confidence.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The human interaction detection method based on spatial fine-grained contextual interaction feature fusion of the present invention utilizes a spatial fine-grained feature learning module to perform spatial location embedding, multi-scale convolution aggregation and feature fusion on global image features. By injecting fine-grained spatial information, it enhances local spatial perception capabilities, enabling the visual encoder to capture local object details, spatial layout and structural texture, and enabling the model to extract texture features and spatial details of key regions from the image.

[0015] 2. The human interaction detection method based on spatial fine-grained contextual interaction feature fusion of the present invention utilizes a spatial contextual interaction feature fusion module to perform interactive semantic reasoning on human-object pairing features through region feature extraction, interaction context extraction, interaction pattern capture, and interaction feature enhancement, thereby capturing the contextual relationship between humans and objects. It can model more complex interaction dependencies, enabling the model to accurately distinguish patterns with similar semantics but different interaction categories, thus improving overall detection accuracy. This provides an effective visual input solution for efficient interactive semantic support of embodied intelligent systems in real-time task planning in real-world environments. Attached Figure Description

[0016] Figure 1 This is a flowchart of the character interaction detection method based on spatial fine-grained contextual interaction feature fusion of the present invention; Figure 2 This is a flowchart illustrating the principle of the character interaction detection method based on spatial fine-grained contextual interaction feature fusion of the present invention. Detailed Implementation

[0017] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0018] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for detecting human interaction based on spatial fine-grained contextual interaction feature fusion.

[0019] Example 1 A character interaction detection method based on spatial fine-grained contextual interaction feature fusion, referring to Figures 1-2 As shown, it includes the following steps: Step 1: Initialize the object detector, text encoder, and visual encoder; input the image into the object detector to obtain the object detection result set; obtain human-object pairing features based on the object detection result set; perform gridded projection on the image to obtain global image features; Step 2: Construct a spatial fine-grained feature learning module, with the input being the set of global image features and object detection results, and the output being spatial fine-grained features; Step 3: Construct a spatial context interaction feature fusion module, with inputs being fine-grained spatial features and human-object pairing features, and output being spatial context interaction features; Step 4: After generating a sentence description for the character interaction category name using a predefined text template, obtain the text embedding features through a text encoder; input the spatial fine-grained features and spatial context interaction features into the visual encoder to obtain enhanced person-object pairing features; calculate the similarity between the text embedding features and the enhanced person-object pairing features to obtain the character interaction category score; Step 5: Calculate the focus loss function based on the human interaction category score and the actual interaction category. Use the stochastic gradient descent algorithm to train the human interaction detection model, which consists of a spatial fine-grained feature learning module, a spatial context interaction feature fusion module, a visual encoder, and a text encoder. For other images, execute steps 1 to 4 in sequence to obtain the spatial position of all people and objects and their paired interaction categories until the human interaction detection model converges, thus completing the training of the human interaction detection model. Step 6: Input the new image into the trained human interaction detection model to obtain the predicted human bounding box, object bounding box, and interaction category.

[0020] Specifically, step 1 includes: Step 11: Initialize the object detector, CLIP (Contrastive Language-Image Pre-Training) text encoder, CLIP visual encoder, and predefined text templates for the input image. Detection is performed using a pre-trained DETR (Detection Transformer) object detector, resulting in... The set of detection results is denoted as the target detection result set. The bounding box coordinates Represented as Target category Belongs to the category set , Total number of target categories; confidence score The target global feature vector output by the target detector decoder is ( Generally, 256 is taken. Represents the first element in the set of target detection results. One goal; Step 12: Person-object screening and candidate construction for person-object pairing: From the target detection result set Targets with the category "human" are selected from the data, and a human index set is constructed. Index for each person , with all other target indexes ( and By forming pairs, a set of person-object pairing indices is obtained. The total number of pairs is ; Step 13: Obtaining Person-Object Pairing Features: Extracting the person-object pairing index set. The first in A pair Extracting global human feature vectors from the target detection result set and the global feature vector of the object The two are concatenated and then mapped to paired features through two layers of feed-forward networks (FFN): , This is a splicing operation; among which The feedforward neural network function contains two fully connected layers. The weights of the first layer are... ( (Generally taken as 1024), the weight of the second layer is... ( (Generally, 512 is used); all paired features constitute person-object paired features. Its submatrices are all stored column-wise; Step 14: Global Image Feature Acquisition: The input image... It is divided into a non-overlapping, fixed-size grid, each grid typically 16×16 pixels, thus forming The grid in which and Depending on the input image resolution; each grid is then flattened and projected onto... Dimensions, after adding location information, yield global image features. Simultaneously, a learnable class embedding vector is randomly initialized. Used to summarize global image features; to pair human-object features global features of the image With learnable category embedding vectors The complete sequence is obtained by splicing. This sequence will be used for subsequent layer-by-layer processing.

[0021] Specifically, step 2 includes: Step 21: The spatial fine-grained feature learning module includes: spatial feature projection, spatial location embedding, multi-scale convolution aggregation, and multi-scale and spatial location feature fusion. The purpose of this module is to incorporate fine-grained spatial information at the feature level, so that the global representation of CLIP is enhanced to capture fine-grained object details and spatial structure, which is more conducive to subsequent ROI feature extraction and interactive judgment. Step 22: Spatial Feature Projection: Projecting global features of the image Projection as spatial mapping image features Then, a dimension-reduced FFN with weights is applied. ,Will Projected onto a more compact dimension ,Right now , usually take Dimensionality reduction is mainly used to reduce computational cost and extract more compact feature representations; Step 23: Spatial Location Embedding: For each spatial location in the spatially mapped image features From the set of target detection results Find the detection target index that covers this location. Extract bounding box coordinates Confidence score and target category Input the target category into the CLIP text encoder to obtain the text embedding. ;Will , , After concatenating the three elements, the spatial location embedding vector is obtained through FFN mapping. Apply the above steps to each spatial location to obtain the spatial location embedding matrix. ; and After fusion, spatial embedding features are obtained through layer normalization. , This is a layer normalization function, used to obtain spatial location information of objects in the image; Step 24: Multi-scale convolutional aggregation: Create Convolutional layers with different kernels The set of convolution kernel sizes is (generally ); Spatial embedding features use Different types of convolutions, with stride=1 and padding to maintain spatial dimensions, yield multi-scale feature sets. Each scale is mapped to FFN. Multi-scale spatial perception features are obtained by combining and summing the dimensions. ; Step 25: Multi-scale and spatial location feature fusion; integrating multi-scale spatial perception features Projection Back Dimensions: Through weighted residual connection: ,in This indicates element-wise multiplication. These are learnable parameters; the optimal weights are automatically learned during training. It represents a fine-grained spatial feature.

[0022] Specifically, step 3 includes: Step 31: The spatial context interaction feature fusion module includes a region feature extraction module, an interaction context extraction module, an interaction pattern capture module, and a spatial interaction feature fusion module. By analyzing the association patterns between people and object regions, it identifies the occurrence and mode of interaction. Step 32: Region Feature Extraction Module: Extracting the Person-Object Pairing Index Set The first in Personal-item pairing From the perspective of spatial fine-grained features Extracting region features—using ROI (Region of Interest) Alignment (uniform sampling within the bounding box) Region features are extracted using a bilinear interpolation operation at each location. , ,in and These are the bounding boxes for people and objects, respectively. These are the human region features and the object region features, respectively; then, the compact region features of the human and object are obtained through FFN. , Flatten is the operation for flattening; Step 33: Interaction Context Extraction Module: For the first step obtained in step 32... Person-object pairing of compact region features of people and objects, constructing A learnable query (generally ); Extracting context related to human-object interaction through cross-attention and : , Cross-attention calculation ,in and Depend on or Obtained by linear transformation; , Used to capture interaction-related features and filter out irrelevant details; Step 34: Interaction Mode Capture Module: For the first step obtained in step 33... Person-object pairing involves contextualizing person-object interactions and calculating interaction features through cross-attention. , ; Indicates for the first Person-object pairing integrates human interaction features with object characteristics; This indicates object interaction features that incorporate human characteristics; this operation captures the interaction patterns between humans and objects. Step 35: Interaction Feature Enhancement Module: Similarly, for the first... Person-object pairing, extracting pairing features Dimensionality reduction is obtained by linear mapping. ;by As a query, it extracts enhanced interaction features from human-object interaction features; human enhanced interaction features Enhanced interactive features of objects The two enhanced interaction features mentioned above are concatenated and projected back. Dimensions followed by learnable parameters Weighted residual connection ; All processed After matching features, spatial context interaction features are obtained. .

[0023] Specifically, step 4 includes: Step 41: Insert the spatial fine-grained feature learning module from Step 2 and the spatial context interaction feature fusion module from Step 3 into multiple levels of the CLIP visual encoder (total). (layer), specifically, in the first layer Layer CLIP visual encoder Previously, the global features of the image were used as the output of the previous layer, which is also the input of the current layer. Human-object pairing characteristics and learnable category embeddings The spatial fine-grained feature learning module is called to obtain... The spatial context interaction feature fusion module obtains and will Learnable category embeddings and After sequential assembly, it is sent to the first... The CLIP visual encoder of the layer obtains ,in and These are enhanced human-object pairing features and global image features, respectively; this mechanism allows global image features and human-object pairing features to be repeatedly enhanced at multiple levels. Step 42: Generate interactive category text embeddings: Obtain the set of interactive categories from the dataset. Number of categories For each interaction category Based on the category verb and object category, a templated text description, "A photo of a person [verb-ing] a / an [object]", is generated using a predefined text template. The CLIP text encoder then converts this text sequence into a token ID sequence using a tokenizer. ,in For the first The number of tokens in the text class. Use the token ID; insert K learnable tokens before the text: assign a trainable vector representation to each learnable token. This leads to a set of learnable tokens. The aforementioned learnable token is inserted before the text token embedding to form an extended text sequence. ,in This represents the word embedding layer of the CLIP text encoder, used to obtain text token embeddings; finally, the CLIP text encoder, which expands the text sequence input, obtains the word embedding. The final text embedding vectors for each category , For text encoders; text embedding vectors of each category constitute the text embedding features of the character interaction category. Each of its submatrices is stored column-wise; Step 43: Calculate the person-object interaction classification score: Extract the person-object pairing features from the last layer of the CLIP visual encoder. Computation and text embedding features cosine similarity ,in Temperature parameter (usually taken as...) ); specifically, Indicates the first Personal-object pairing and the first Similarity between categories; probability matrix of interaction category scores obtained through sigmoid activation. ,in For the first Personal-object pairing belongs to the first The probability of each category.

[0024] Specifically, step 5 includes: Step 51: Construct training labels: For the training images, the first... Person-object pairing, if a pair of ground-truth bounding boxes exists Make and ( And usually take If the person-object pairing matches the ground truth, then the pairing is considered to match if the ground truth tag includes a category. ,make Otherwise Thus, a binary label matrix is ​​constructed. ; Step 52: Calculate the focus loss: Use the focus loss to mitigate class imbalance. The formula is as follows: , in, , For balance parameters (usually) ), Focusing parameters (usually) Focus loss assigns greater weight to hard-to-classify samples and reduces the weight of easy-to-classify samples. Step 53: Model Training and Convergence: Train the model on the given image dataset and its included human interaction categories using stochastic gradient descent. The model consists of a spatial fine-grained feature learning module, a spatial context interaction feature fusion module, a visual encoder, and a text encoder. In each iteration, randomly sample a batch of images and execute steps 1 to 4 to obtain the human interaction category score probability matrix. Calculate the loss The parameters are updated by backpropagation; training stops when the validation set loss does not decrease for five consecutive rounds; training uses only seen categories, and the human interaction detection model generalizes to unseen categories through the semantic space of CLIP.

[0025] Specifically, step 6 includes: Input a new image and execute steps 1 to 4 to obtain the probability matrix of character interaction categories. The final interaction confidence is obtained by incorporating the detection confidence. ,in and Confidence scores for people and objects are determined based on the first... Personal-item pairing From the object detection result set Obtained from; Hyperparameters (usually) Pairing up each character. Select the category with the highest score: ; like ( And usually take ), output , These are the bounding boxes for the detected people and objects, respectively. For the first in the set of interaction categories Categories For interactive confidence.

[0026] The present invention provides a method for detecting human interaction based on spatial fine-grained contextual interaction feature fusion, referring to... Figures 1-2 As shown, it first performs object detection on the input image to obtain an object detection result set; then, based on the object detection result set, it constructs a person-object pairing index set and extracts person-object pairing features accordingly; simultaneously, it segments and processes the input image to obtain global image features; the global image features are input into a spatial fine-grained feature learning module to obtain spatial fine-grained features; then, the spatial fine-grained features and person-object pairing features are input into a spatial context interaction feature fusion module to obtain spatial context interaction features; the spatial fine-grained feature learning module and the spatial context interaction feature fusion module are embedded before the CLIP visual encoder, and the above steps are executed multiple times to strengthen the person-object pairing features; finally, the visual encoder is used to extract the human-object pairing features. The last layer of the encoder outputs human-object pairing features, and calculates their similarity with the features output by the text encoder to obtain a human interaction score, thereby determining the interaction category. Finally, a new image is input, and the previous steps are repeated until the model converges. The new image data is input into the trained model to obtain predicted human bounding boxes, object bounding boxes, interaction categories, and corresponding confidence scores. This method utilizes a spatial fine-grained feature learning module, enabling the model to learn the perception of local object details, spatial layout, and structural texture. Then, a spatial context interaction feature fusion module is used to capture the contextual relationship between humans and objects, allowing the model to more accurately distinguish multiple interaction modes and improve the overall classification accuracy.

[0027] The content described in this embodiment is merely an enumeration of the implementation forms of the inventive concept. The scope of protection of this invention should not be regarded as limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for detecting character interactions based on spatial fine-grained contextual interaction feature fusion, characterized in that, Includes the following steps, Step 1: Initialize the object detector, text encoder, and visual encoder. Input the image into the object detector to obtain the object detection result set, and obtain the human-object pairing features based on the object detection result set. Global features of an image are obtained by performing a gridded projection on the image. Step 2: Construct a spatial fine-grained feature learning module, with the input being the set of global image features and object detection results, and the output being spatial fine-grained features; Step 3: Construct a spatial context interaction feature fusion module, with inputs being fine-grained spatial features and human-object pairing features, and output being spatial context interaction features; Step 4: After generating a sentence description for the character interaction category name using a predefined text template, obtain the text embedding features through a text encoder; input the spatial fine-grained features and spatial context interaction features into the visual encoder to obtain enhanced human-character pairing features; Calculate the similarity between text embedding features and enhanced person-object pairing features to obtain a person interaction category score; Step 5: Calculate the focus loss function based on the human interaction category score and the actual interaction category. Use the stochastic gradient descent algorithm to train the human interaction detection model, which consists of a spatial fine-grained feature learning module, a spatial context interaction feature fusion module, a visual encoder, and a text encoder. For other images, execute steps 1 to 4 in sequence to obtain the spatial position of all people and objects and their paired interaction categories until the human interaction detection model converges, thus completing the training of the human interaction detection model. Step 6: Input the new image into the trained human interaction detection model to obtain the predicted human bounding box, object bounding box, and interaction category.

2. The character interaction detection method based on spatial fine-grained contextual interaction feature fusion according to claim 1, characterized in that, Step 1 specifically includes: Step 11: Initialize the object detector, CLIP text encoder, CLIP visual encoder, and predefined text templates for the input image. Detection was performed using a pre-trained DETR object detector, and the results were obtained. The set of detection results is denoted as the target detection result set. The bounding box coordinates Represented as Target category Belongs to the category set , Total number of target categories; confidence score The target global feature vector output by the target detector decoder is ; Represents the first element in the set of target detection results. One goal; Step 12: Person-object screening and candidate construction for person-object pairing: From the target detection result set Targets with the category "human" are selected from the data, and a human index set is constructed. Index for each person , with all other target indexes ( and By forming pairs, a set of person-object pairing indices is obtained. The total number of pairs is ; Step 13: Obtaining Person-Object Pairing Features: Extracting the person-object pairing index set. The first in A pair Extracting global human feature vectors from the target detection result set and global feature vector of the object The two are concatenated and then mapped to paired features through a two-layer feedforward network: , This is a splicing operation; among which The feedforward neural network function contains two fully connected layers. The weights of the first layer are... The weight of the second layer is All paired features constitute person-object paired features. Its submatrices are all stored column-wise; Step 14: Global Image Feature Acquisition: The input image... It is divided into a non-overlapping, fixed-size grid, with each grid cell being 16×16 pixels, thus forming The grid in which and Depending on the input image resolution; each grid is then flattened and projected onto... Dimensions, after adding location information, yield global image features. Simultaneously, a learnable class embedding vector is randomly initialized. Used to summarize global image features; to pair human-object features global features of the image With learnable category embedding vectors The complete sequence is obtained by splicing. This sequence will be used for subsequent layer-by-layer processing.

3. The character interaction detection method based on spatial fine-grained contextual interaction feature fusion according to claim 2, characterized in that, Step 2 specifically includes: Step 21: The spatial fine-grained feature learning module includes: spatial feature projection, spatial location embedding, multi-scale convolutional aggregation, and multi-scale fusion with spatial location features; Step 22: Spatial Feature Projection: Projecting global features of the image Projection as spatial mapping image features Then, a dimension-reduced FFN with weights is applied. ,Will Projected onto a more compact dimension ,Right now ; Step 23: Spatial Location Embedding: For each spatial location in the spatially mapped image features From the set of target detection results Find the detection target index that covers this location Extract bounding box coordinates Confidence score and target category Input the target category into the CLIP text encoder to obtain the text embedding. ;Will , , After concatenating the three elements, the spatial location embedding vector is obtained through FFN mapping. Apply the above steps to each spatial location to obtain the spatial location embedding matrix. ; and Spatial embedding features are obtained after fusion and layer normalization. , This is a layer normalization function, used to obtain spatial location information of objects in the image; Step 24: Multi-scale convolutional aggregation: Create Convolutional layers with different kernels The set of convolution kernel sizes is (generally ); Spatial embedding features use Different types of convolutions, with stride=1 and padding to maintain spatial dimensions, yield multi-scale feature sets. Each scale is mapped to FFN. Multi-scale spatial perception features are obtained by combining and summing the dimensions. ; Step 25: Multi-scale and spatial location feature fusion; integrating multi-scale spatial perception features Projection Back Dimensions: Through weighted residual connection: ,in This indicates element-wise multiplication. These are learnable parameters; the optimal weights are automatically learned during training. It represents a fine-grained spatial feature.

4. The character interaction detection method based on spatial fine-grained contextual interaction feature fusion according to claim 3, characterized in that, Step 3 specifically includes: Step 31: The spatial context interaction feature fusion module includes a region feature extraction module, an interaction context extraction module, an interaction pattern capture module, and a spatial interaction feature fusion module. By analyzing the association patterns between people and object regions, it identifies the occurrence and mode of interaction. Step 32: Region Feature Extraction Module: Extracting the Person-Object Pairing Index Set The first in Personal-item pairing From the perspective of spatial fine-grained features Extracting Region Features – Using ROI Align Operation , ,in and These are the bounding boxes for people and objects, respectively. These are the human region features and the object region features, respectively; then, the compact region features of the human and object are obtained through FFN. , Flatten is the operation for flattening; Step 33: Interaction Context Extraction Module: For the first step obtained in step 32... Person-object pairing of compact region features of people and objects, constructing A learnable query Extracting contextual information related to human-object interaction through cross-attention. and : , Cross-attention calculation ,in and Depend on or Obtained by linear transformation; , Used to capture interaction-related features and filter out irrelevant details.

5. The character interaction detection method based on spatial fine-grained contextual interaction feature fusion according to claim 4, characterized in that, Step 3 also includes: Step 34: Interaction Mode Capture Module: For the first step obtained in step 33... Person-object pairing involves contextualizing person-object interactions and calculating interaction features through cross-attention. , ; Indicates for the first Person-object pairing integrates human interaction features with object characteristics; This indicates object interaction features that incorporate human characteristics; this operation captures the interaction patterns between humans and objects. Step 35: Interaction Feature Enhancement Module: Similarly, for the first... Person-object pairing, extracting pairing features Dimensionality reduction is obtained by linear mapping. ;by As a query, it extracts enhanced interaction features from human-object interaction features; human enhanced interaction features Enhanced interactive features of objects The two enhanced interaction features mentioned above are concatenated and projected back. Dimensions followed by learnable parameters Weighted residual connection ; All processed After matching features, spatial context interaction features are obtained. .

6. The character interaction detection method based on spatial fine-grained contextual interaction feature fusion according to claim 5, characterized in that, Step 4 specifically includes: Step 41: Insert the spatial fine-grained feature learning module from Step 2 and the spatial context interaction feature fusion module from Step 3 into multiple levels of the CLIP visual encoder. Specifically, in the first step... Layer CLIP visual encoder Previously, the global features of the image were used as the output of the previous layer, which is also the input of the current layer. Human-object pairing characteristics and learnable category embeddings The spatial fine-grained feature learning module is called to obtain... The spatial context interaction feature fusion module obtains and will Learnable category embeddings and After sequential assembly, it is sent to the first... The CLIP visual encoder of the layer obtains ,in and These are enhanced human-object pairing features and global image features, respectively. Step 42: Generate interactive category text embeddings: Obtain the set of interactive categories from the dataset. Number of categories For each interaction category Based on the category verb and object category, a templated text description, "A photo of a person [verb-ing] a / an [object]", is generated using a predefined text template. The CLIP text encoder then converts this text sequence into a token ID sequence using a tokenizer. ,in For the first The number of tokens in the text class. Use the token ID; insert K learnable tokens before the text: assign a trainable vector representation to each learnable token. This leads to a set of learnable tokens. The aforementioned learnable token is inserted before the text token embedding to form an extended text sequence. ,in This represents the word embedding layer of the CLIP text encoder, used to obtain text token embeddings; finally, the CLIP text encoder, which expands the text sequence input, obtains the word embedding. The final text embedding vectors for each category , For text encoders; text embedding vectors of each category constitute the text embedding features of the character interaction category. Each of its submatrices is stored column-wise.

7. The character interaction detection method based on spatial fine-grained contextual interaction feature fusion according to claim 6, characterized in that, Step 4 also includes: Step 43: Calculate the person-object interaction classification score: Extract the person-object pairing features from the last layer of the CLIP visual encoder. Computation and text embedding features cosine similarity ,in For temperature parameters; specifically, Indicates the first Personal-object pairing and the first Similarity between categories; probability matrix of interaction category scores obtained through sigmoid activation. ,in For the first Personal-object pairing belongs to the first The probability of each category.

8. The character interaction detection method based on spatial fine-grained contextual interaction feature fusion according to claim 7, characterized in that, Step 5 specifically includes: Step 51: Construct training labels: For the training images, the first... Person-object pairing, if a pair of ground-truth bounding boxes exists Make and Then it is considered that the person-object pairing matches the ground-truth, and if the match is found and the ground-truth label contains a category. ,make Otherwise Thus, a binary label matrix is ​​constructed. .

9. The character interaction detection method based on spatial fine-grained contextual interaction feature fusion according to claim 8, characterized in that, Step 5 also includes: Step 52: Calculate the focus loss: Use the focus loss to mitigate class imbalance. The formula is as follows: , in, , For balancing parameters, For focusing parameters; Step 53: Model Training and Convergence: Train the human interaction detection model on the given image dataset and its included human interaction categories using the stochastic gradient descent algorithm. In each iteration, randomly sample a batch of images and execute steps 1 to 4 to obtain the human interaction category score probability matrix. Calculate the loss The parameters are updated by backpropagation; training stops when the validation set loss does not decrease for five consecutive rounds; training uses only seen categories, and the human interaction detection model generalizes to unseen categories through the semantic space of CLIP.

10. The character interaction detection method based on spatial fine-grained contextual interaction feature fusion according to claim 9, characterized in that, Step 6 specifically includes: Input a new image and execute steps 1 to 4 to obtain the probability matrix of character interaction categories. The final interaction confidence is obtained by incorporating the detection confidence. ,in and Confidence scores for people and objects are determined based on the first... Personal-item pairing From the object detection result set Obtained from; For hyperparameters; pairing each person with another person Select the category with the highest score: ; like Output , These are the bounding boxes for the detected people and objects, respectively. For the first in the set of interaction categories Categories For interactive confidence.

Images