Multi-modal fusion character interaction recognition method and system based on large model

Abstract

The invention provides a multi-modal fusion character interaction recognition method and system based on a large model, and the method comprises the steps: carrying out the target detection, posture estimation and Caption extraction of each frame of image of a target video, so as to obtain the visual features, geometric features and text features of the target video; then extracting structured geometric features and visual features of fused semantics through a feature extraction model; and taking the structured geometric features as different types of entities, obtaining a hidden state of a time sequence dimension through a bidirectional recurrent neural network BiRNN, realizing intra-class and inter-class information interaction between the entities through a message attention mechanism, and finally outputting a character interaction recognition result through decision-level fusion. According to the method, topological association between skeleton joints and entities is fully excavated, deep interaction of semantic and visual information is realized in combination with a semantic-visual dual-guidance module, and the alignment degree between modals is improved; and the robustness and the recognition accuracy of the model in a complex scene are improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for multimodal fusion human interaction recognition based on a large model. Background Technology

[0002] Human-Object Interaction (HOI) recognition is a core task in computer vision, aiming to identify the ternary relationship of <person, interaction category, object> from images or videos. Existing HOI recognition methods mostly focus on static images, while most real-world interactions exhibit temporal dynamics, such as picking up, putting down, and grasping. Static images contain almost no temporal information about people and objects, making it difficult to distinguish such interactions using HOI detection methods for static images. Therefore, human interaction recognition based on video data becomes crucial for solving this problem.

[0003] The core challenge of video character interaction lies in the fact that occlusion and perspective changes in the video data itself, dynamic background interference, and the ambiguity of fine-grained movements all make it impossible for any single modality to capture these subtle differences. Multimodal fusion is considered a potential solution to overcome the limitations of single modality, and human posture, as an abstract representation of character movements, has become the preferred choice.

[0004] Existing technologies, such as 2G-GCN and Hier-GAT, have improved the performance of video character interaction recognition by incorporating human pose. However, these methods often use simple graph structures to shallowly encode pose information, failing to fully explore the topological relationships between skeletal joints. Semantic modalities, as a bridge connecting visual features and external knowledge, can help models maintain robust understanding of interactive behaviors in data-limited or complex environments. Although visual-language models (VLMs) have made significant progress in recent years, their large number of parameters leads to high fine-tuning costs for downstream tasks. Some studies have attempted to use VLMs to generate image captions to guide semantic learning; however, the inter-frame correlation of dynamic interactions in HOI videos poses an accuracy challenge to frame-by-frame caption generation. Furthermore, the essential characteristics of multimodal data differ significantly, with complementarity and redundancy existing between modalities. Existing methods (such as simple feature concatenation) struggle to adaptively balance the contribution weights of different modalities, especially prone to modal conflicts in complex scenarios. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multimodal fusion method and system for human interaction recognition based on a large model. This invention employs a multi-fusion strategy involving message attention, feature-level, and decision-level fusion to effectively fuse different modalities, thereby achieving accurate segmentation and recognition of video HOIs.

[0006] In a first aspect, the present invention provides a multimodal fusion method for human interaction recognition based on a large model, comprising the following steps:

[0007] S1) Acquire the target video and perform target detection, pose estimation and caption extraction on each frame of the target video to obtain the visual features, geometric features and text features of the target video;

[0008] S2) Construct a feature extraction model, which includes a geometric feature extraction module and a semantic-visual dual-guided module; use the geometric feature extraction module to extract structured geometric features from the geometric features; and use the semantic-visual dual-guided module to perform multi-head cross-attention calculation on text semantic features, HOI action tags, and visual features to achieve deep interaction between semantic and visual information and obtain visual features that fuse semantics.

[0009] S3) Construct a spatiotemporal graph centered on vision, treat the structured geometric features as different categories of entities, obtain the hidden states of the temporal dimension through a bidirectional recurrent neural network (BiRNN), realize information interaction between entities within and between categories through a message attention mechanism, and finally output the human interaction recognition result through decision-level fusion.

[0010] Preferably, in step S1), the extraction of visual features from the target video specifically includes the following steps:

[0011] S11) Obtain feature maps for each frame of image using a pre-trained object detector. and the bounding boxes of all objects in that frame image. ;

[0012] S12) Using a pose estimator, obtain the skeleton point matrix of all human entities in each frame of the image. ;

[0013] Calculate the skeletal point matrix for each person. Find the minimum bounding rectangle of the human body entity, and define the corresponding bounding box of the minimum bounding rectangle. ; Get the bounding boxes of all human entities ;

[0014] S13) Using the bounding box of the human body entity bounding box of the object entity Perform feature map Perform ROIAlign and global average pooling processing; human visual feature set and object visual feature set.

[0015] Preferably, in step S1), the extraction of geometric features of the target video is specifically as follows:

[0016] Using the diagonal points of the bounding box as object keypoints, the human skeleton point matrix is... Combine human body key points and object key points to construct a multi-interaction graph set to obtain geometric features;

[0017] Build human-centered graphs for every person and every object ;

[0018] Build an object relationship graph for all objects. ;

[0019] Constructing a human connection diagram for all human bodies ;

[0020] Build a global location graph for all entities ;

[0021] In all constructed graphs, in addition to the coordinates of each bone point and keypoint, velocity information is additionally introduced, represented by a one-step forward difference. , , and Together they constitute geometric features.

[0022] Preferably, in step S2), for the graph First, calculate its adaptive adjacency matrix. ;

[0023] Then, through A learnable state transition matrix, for the adaptive adjacency matrix and picture Perform graph convolution operation to obtain the first... Individual-centered geometric features;

[0024] The geometric features of all the obtained graphs are flattened and embedded into the same dimension in the MLP. All human-centered geometric features are added to the corresponding human visual features through a learnable scalar weight to obtain human visual-geometric features.

[0025] Object relationship diagram Perform the same operation to obtain the visual-geometric features of the object;

[0026] For human body connection diagram Global location map , and and By splicing and mapping, interactive geometric features are obtained.

[0027] Preferably, step S3) is as follows:

[0028] For the temporal aspect, a bidirectional recurrent neural network (BiRNN) is used for extraction. Human visual-geometric features at any moment Two-way hidden state ;

[0029] For spatial edges, a message attention mechanism is designed to enable message passing between entities within and between classes.

[0030] Information on human categories and object categories is aggregated to obtain aggregated human body information and aggregated object information;

[0031] And it uses ASSIGN's segment-level network to output segment-level recognition results;

[0032] When the spatiotemporal graph outputs visually centered human-object interaction sub-action recognition results, it is combined with the visual features fused with semantics. Decision-level fusion is achieved through weighted summation, and the final recognition result is output.

[0033] Secondly, the present invention provides a multimodal fusion human interaction recognition system based on a large model, comprising:

[0034] The preliminary feature extraction module is used to perform target detection, pose estimation and caption extraction on each frame of the target video to obtain the visual features, geometric features and text features of the target video;

[0035] The feature extraction and fusion module calls the feature extraction model and performs structured geometric feature extraction through the geometric feature extraction module of the feature extraction model; through the semantic-visual dual-guided module, the text semantic features, HOI action labels, and visual features are subjected to multi-head cross-attention calculation to achieve deep interaction between semantic and visual information and obtain visual features with fused semantics.

[0036] The interaction recognition module is used to construct a spatiotemporal graph centered on vision, taking the structured geometric features as different categories of entities, obtaining the hidden states of the temporal dimension through a bidirectional recurrent neural network (BiRNN), realizing information interaction between entities within and between categories through a message attention mechanism, and finally outputting the human interaction recognition result through decision-level fusion.

[0037] The beneficial effects of this invention are as follows:

[0038] 1. This invention covers the geometric relationships between people, objects, and the global scene through a multi-interaction graph set, and combines it with a graph convolutional network with an adaptive adjacency matrix to fully explore the topological relationships between skeletal joints and entities, thus solving the problem of incomplete geometric modeling in existing methods.

[0039] 2. This invention automatically generates image captions by calling the VLM API, improves the accuracy of image descriptions generated by VLM through Prompt constraints, solves the temporal discontinuity problem of text features by using a temporal alignment module, and achieves deep interaction between semantic and visual information by combining a semantic-visual dual-guidance module, thereby improving the alignment between modalities.

[0040] 3. This invention constructs a spatiotemporal graph centered on vision, which can integrate visual information, posture information, and textual semantic information; it captures temporal features through BiRNN, realizes dynamic information interaction between entities by using message attention mechanism, and combines feature-level and decision-level fusion strategies to adaptively weigh the contribution weights of different modalities, effectively avoids modal conflicts, and improves the robustness and recognition accuracy of the model in complex scenarios.

[0041] 4. This invention uses the feature map information corresponding to the bounding boxes of people and objects to describe the appearance and texture details of people and objects during the interaction process. Dimensionality reduction is performed through ROI Align and global average pooling to enhance the visual understanding of human interaction. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the process framework of the method in an embodiment of the present invention;

[0043] Figure 2 This is a flowchart illustrating the image preprocessing stage in an embodiment of the present invention;

[0044] Figure 3 This is a flowchart illustrating the feature extraction model of an embodiment of the present invention;

[0045] Figure 4 This is a schematic diagram of the framework for human interaction recognition in an embodiment of the present invention. Detailed Implementation

[0046] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0047] Example 1

[0048] like Figure 1 As shown, this embodiment provides a multimodal fusion method for human interaction recognition based on a large model, including the following steps:

[0049] S1) Obtain the target video and perform target detection, pose estimation and caption extraction on each frame of the target video to obtain the visual features, geometric features and textual semantic features of the target video;

[0050] In this embodiment, the extraction of visual features from the target video, such as... Figure 2 As shown, the specific steps include the following:

[0051] S11) Obtain feature maps for each frame of image using a pre-trained object detector. and the bounding boxes of all objects in that frame image. ;

[0052] The target detector used in this example is RT-DETRv2, which consists of a backbone network and a detection head.

[0053] S12) Using a pose estimator, obtain the skeleton point matrix of all human entities in each frame of the image. ;

[0054] Among them, the first The first frame of the image Personal skeletal matrix for The matrix stores The 2D coordinates of each skeletal point;

[0055] Calculate the skeletal point matrix for each person. Find the minimum bounding rectangle of the human body entity, and define the corresponding bounding box of the minimum bounding rectangle. ; Get the bounding boxes of all human entities Whether it's a human body bounding box or an object bounding box, the coordinates of the diagonal points of the bounding box are used. Indicates; among which, The maximum coordinates of the diagonal; The smallest coordinates of the diagonal;

[0056] S13) Using the bounding box of the human body entity bounding box of the object entity Perform feature map Perform ROIAlign and global average pooling; that is:

[0057] ;

[0058] ;

[0059] ;

[0060] ;

[0061] In the formula, This is a global average pooling process; Indicates ROI Align processing; Indicates the first The first frame of the image Individual visual characteristics of the human body; Indicates the first The first frame of the image Visual characteristics of an object; Represents the image in frame t. The bounding box of an object; and These are the human visual feature set and the object visual feature set, respectively.

[0062] In this embodiment, the extraction of geometric features from the target video, such as... Figure 2 As shown, the details are as follows:

[0063] Using the diagonal points of the bounding box as object keypoints, the human skeleton point matrix is... Combine human body key points and object key points to construct a multi-interaction graph set to obtain geometric features;

[0064] In this embodiment, a human-centered graph is constructed for each person and all objects. ,Right now:

[0065] ;

[0066] In the formula, The bounding box matrix representing the J-th object; Indicates the first An individual's skeletal point matrix;

[0067] Build an object relationship graph for all objects. ;Right now:

[0068] ;

[0069] Constructing a human connection diagram for all human bodies ;Right now:

[0070] ;

[0071] Build a global location graph for all entities ;Right now:

[0072] ;

[0073] In the formula, express The mean of all points;

[0074] , , and Together they constitute geometric features.

[0075] In all constructed graphs, in addition to the coordinates of each bone point and keypoint, velocity information is additionally introduced, represented by a one-step forward difference. Therefore, the... A point in the frame is represented as:

[0076] ;

[0077] In the formula, ( Indicates the first The horizontal and vertical coordinates of that point in the frame image; It is the first The forward difference of a point in the frame image, that is, the velocity of the horizontal and vertical coordinates corresponding to that point.

[0078] In this embodiment, the extraction of text semantic features, such as Figure 2 As shown, the details are as follows:

[0079] By calling the API of the pre-trained Visual Language Model (VLM) and using Prompt constraints, the caption of each frame of the video is obtained. Simultaneously, the text of all interactive action labels in the dataset is tokenized using BLIP and CLIP, and then text-encoded to obtain label knowledge. ;

[0080] The caption is processed through the CLIP text encoder to obtain the corresponding text encoding features. ;

[0081] The visual coding features of the same video segment are obtained frame by frame using the CLIP visual coding module. ;

[0082] Calculate text encoding features frame by frame and visual encoding features The cosine distance is used to replace the distance value in the dynamic time warping algorithm to obtain time-aligned text semantic features. .

[0083] S2) Construct a feature extraction model, which includes a geometric feature extraction module and a semantic-visual dual-guided module; use the geometric feature extraction module to extract structured geometric features from the multi-interaction graph set; through the semantic-visual dual-guided module, perform multi-head cross-attention calculation on the text semantic features, HOI action tags, and visual features to achieve deep interaction between semantic and visual information and obtain visual features with fused semantics.

[0084] In this embodiment, as Figure 3 As shown in (a), the geometric feature extraction module includes multiple weighted graph convolutional networks and linear layers; it adapts to different data distributions through an adaptive adjacency matrix, and obtains structured geometric features through graph convolution operations and feature mapping.

[0085] In this embodiment, each graph in the interaction graph set is assigned a weighted graph convolutional network. For each graph convolutional network in the geometric feature extraction module, an adaptive adjoint matrix is ​​used to adapt to different data distributions; specifically:

[0086] The query vector is calculated through two fully connected layers. and key vector Dot product attention is used to model the hidden relationships between nodes in the graph.

[0087] For the graph Its adaptive adjacency matrix is ​​expressed as:

[0088] ;

[0089] In the formula, An adaptive adjacency matrix; For activation functions; The diagrams are shown below. The corresponding query and key matrix; It is a fully connected layer; is the dimension of the adjacency matrix;

[0090] Then, through A learnable state transition matrix Performing graph convolution operation yields:

[0091] ;

[0092] In the formula, Indicates the first Individual-centered geometric features; This represents an N-layer graph convolution, using ReLU as the activation function.

[0093] Flatten the geometric features of all the obtained graphs and embed them into the same dimension in the MLP, thus merging all human-centered geometric features. By adding a learnable scalar weight to the corresponding visual features of the person, we obtain... ;Right now:

[0094] ;

[0095] In the formula, Represents human visual-geometric features; for The corresponding learnable scalar weights; It represents the stitching together of all human-centered geometric features; A set of human visual features;

[0096] Object relationship diagram Perform the same operation to obtain ,Right now:

[0097] ;

[0098] In the formula, Representing the visual-geometric features of an object; express The corresponding learnable scalar weights; Represents the geometric features of an object; A set of visual features of an object;

[0099] For human body connection diagram Global location map , and and By concatenating and mapping, we get:

[0100] ;

[0101] In the formula, Represents interactive geometric features; , These represent human body geometric features and global geometric features, respectively.

[0102] Interactive geometric features It is treated as an entity with the same visual features and used for subsequent fusion decoding.

[0103] In this embodiment, as Figure 3 As shown in (b), the semantic-visual dual-guided module includes two Transformer decoders for text semantic features. The semantic features are obtained by incorporating image captions into HOI action labels using a Transformer decoder. It includes HOI interaction information from global image alignment to specific sub-action spaces, thereby guiding the model to achieve robust recognition, as detailed below:

[0104] In the first Transformer decoder, for time Textual semantic features and label knowledge From tag knowledge As a query, text semantic features The sum is used for multi-head cross-attention calculation, the... layer The calculation process is expressed as follows:

[0105]

[0106] In the formula, Indicates the first Intermediate output of the layer; Indicates the first The final output of the layer; This represents residual connections and layer normalization; This indicates multi-headed cross-attention; This indicates a 2-layer MLP;

[0107] The output of the last layer is used as semantic features. For the semantic features of the second Transformer decoder Human visual feature set as a query vector As keys and values, the final output is a visual feature with fused semantics. This allows for guided visual interaction.

[0108] S3) Construct a spatiotemporal graph centered on the visual aspect, incorporating the structured geometric features, including interactive geometric features. Visual features that integrate geometric information As different types of entities, hidden states in the temporal dimension are obtained through a bidirectional recurrent neural network (BiRNN). Information interaction between entities within and between classes is achieved through a message attention mechanism. Finally, the results of human interaction recognition are output through decision-level fusion. Specifically, as follows... Figure 4 As shown;

[0109] In this embodiment, a bidirectional recurrent neural network (BiRNN) is used to extract time-related data. Human visual-geometric features at any moment Two-way hidden state The process of obtaining the hidden state is represented as:

[0110] ;

[0111] In the formula, Indicates a two-way hidden state ; express The positive RNN state at any given moment. This represents the inverse RNN state at time t; for exist Sampling at any given time; and They are The positive RNN state at time step and The reverse RNN state at time step.

[0112] For spatial edges, a message attention mechanism is designed to enable message passing between entities within and between classes, namely:

[0113] For inter-class messages: ;

[0114] For messages within a class: ;

[0115] in, Indicates the entity that sent the message. Indicates the message receiving entity. Represents the sending entity from different categories of entities. To the receiving entity The message; Represents sending entities of the same category. To the receiving entity The message; This indicates the interactive attention module; express Time Entity As a characteristic of the sending entity; express Time Entity As a characteristic of the receiving entity; express Time Entity As a hidden state of the receiving entity; , and Both indicate the category to which the entity belongs;

[0116] This is obtained by aggregating information from human categories and object categories, namely:

[0117]

[0118] In the formula, This represents the gumbel-sigmoid function. It is a fully connected layer that uses sigmoid as the activation function; , They represent aggregated information about the human body and aggregated information about objects, respectively. express Human visual-geometric features at any moment; express Visual-geometric features of objects at any given moment; express The human body must remain hidden at all times; express The hidden state of an object at any given moment; , They represent Inter-class and intra-class messages for human body entities at any given time; , They represent Intra-class and inter-class messages for objects at any given time;

[0119] The segment-level recognition results are output using ASSIGN's segment-level network. The segment-level network also includes a message passing process, which can be defined as follows:

[0120] For inter-segment class messages: ;

[0121] For messages within a segment-level class: ;

[0122] in, Indicates from the sending entity To the receiving entity Inter-segment messages, Indicates from the sending entity To the receiving entity Messages within a segment-level class; , Entities As the hidden state of the receiving and sending entities, the segment-level feature can be represented as:

[0123] ;

[0124] in, , These are segment-level features for the human body and objects, respectively.

[0125] Subsequently, the segment-level recognition result can be represented as:

[0126] ;

[0127] In the formula, express The segment-level two-way hidden state of the human body at any given moment; express The positive segmental RNN state at each moment. express The reverse segmental RNN state at time step; This represents the segment-level bidirectional hidden state of an object; the same applies to others.

[0128] When the spatiotemporal graph outputs visually centered human-object interaction sub-action recognition results, it is combined with the visual features fused with semantics. Decision-level fusion is achieved through weighted summation, and the final recognition result is output.

[0129] ;

[0130] In the formula, This indicates the result of human-object interaction sub-action recognition. This represents a fully connected layer with one layer, where Softmax(∙) is the Softmax activation function. It is a learnable semantically weighted scalar weight.

[0131] Example 2

[0132] This embodiment provides a multimodal fusion human interaction recognition system based on a large model, including:

[0133] The preliminary feature extraction module is used to perform target detection, pose estimation and caption extraction on each frame of the target video to obtain the visual features, geometric features and text features of the target video;

[0134] The feature extraction and fusion module calls the feature extraction model and performs structured geometric feature extraction through the geometric feature extraction module of the feature extraction model; through the semantic-visual dual-guided module, the text semantic features, HOI action labels, and visual features are subjected to multi-head cross-attention calculation to achieve deep interaction between semantic and visual information and obtain visual features with fused semantics.

[0135] The interaction recognition module is used to construct a spatiotemporal graph centered on vision, taking the structured geometric features as different categories of entities, obtaining the hidden states of the temporal dimension through a bidirectional recurrent neural network (BiRNN), realizing information interaction between entities within and between categories through a message attention mechanism, and finally outputting the human interaction recognition result through decision-level fusion.

[0136] The embodiments and descriptions above are merely illustrative of the principles and preferred embodiments of the present invention. Various changes and modifications may be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.

Claims

1. A multimodal fusion method for human interaction recognition based on a large model, characterized in that, Includes the following steps: S1) Acquire the target video and perform target detection, pose estimation and caption extraction on each frame of the target video to obtain the visual features, geometric features and text features of the target video; S2) Construct a feature extraction model, which includes a geometric feature extraction module and a semantic-visual dual-guided module; The geometric features are extracted using a geometric feature extraction module to extract structured geometric features. The semantic-visual dual-guided module performs multi-head cross-attention calculation on text semantic features, HOI action tags, and visual features to achieve deep interaction between semantic and visual information and obtain visual features with fused semantics. S3) Construct a spatiotemporal graph centered on vision, treat the structured geometric features as different categories of entities, obtain the hidden states of the temporal dimension through a bidirectional recurrent neural network (BiRNN), realize information interaction between entities within and between categories through a message attention mechanism, and finally output the human interaction recognition result through decision-level fusion.

2. The multimodal fusion human interaction recognition method based on a large model according to claim 1, characterized in that: Step S1), the extraction of visual features from the target video, specifically includes the following steps: S11) Obtain feature maps for each frame of image using a pre-trained object detector. and the bounding boxes of all objects in that frame image. ; S12) Using a pose estimator, obtain the skeleton point matrix of all human entities in each frame of the image. ; Calculate the skeletal point matrix for each person. Find the minimum bounding rectangle of the human body entity, and define the corresponding bounding box of the minimum bounding rectangle. ; Get the bounding boxes of all human entities ; S13) Using the bounding box of the human body entity bounding box of the object entity Perform feature map Perform ROI Align and global average pooling. Human visual feature set and object visual feature set.

3. The multimodal fusion human interaction recognition method based on a large model according to claim 2, characterized in that: In step S1), the extraction of geometric features of the target video is specifically as follows: Using the diagonal points of the bounding box as object keypoints, the human skeleton point matrix is... Combine human body key points and object key points to construct a multi-interaction graph set to obtain geometric features; Build human-centered graphs for every person and every object ; Build an object relationship graph for all objects. ; Constructing a human connection diagram for all human bodies ; Build a global location graph for all entities ; In all constructed graphs, in addition to the coordinates of each bone point and keypoint, velocity information is additionally introduced, represented by a one-step forward difference. , , and Together they constitute geometric features.

4. The multimodal fusion human interaction recognition method based on a large model according to claim 3, characterized in that: In step S1), the extraction of text semantic features is specifically as follows: By calling the API of the pre-trained Visual Language Model (VLM) and using Prompt constraints, the caption of each frame of the video is obtained. Simultaneously, the text of all interactive action labels in the dataset is tokenized using BLIP and CLIP, and then text-encoded to obtain label knowledge. The caption is processed through the CLIP text encoder to obtain its corresponding text encoding features. The CLIP visual coding module is used to obtain the corresponding visual coding features frame by frame from the same video segment. Text encoding features are calculated frame by frame. and visual encoding features The cosine distance is used to replace the distance value in the dynamic time warping algorithm to obtain time-aligned text semantic features. .

5. The multimodal fusion human interaction recognition method based on a large model according to claim 4, characterized in that: In step S2), the geometric feature extraction module includes multiple weight-independent graph convolutional networks and linear layers; it adapts to different data distributions through an adaptive adjacency matrix, and obtains structured geometric features through graph convolution operations and feature mapping. Each graph in the interaction graph set is assigned a weighted graph convolutional network. For each graph convolutional network in the geometric feature extraction module, an adaptive adjoint matrix is ​​used to adapt to different data distributions.

6. The multimodal fusion human interaction recognition method based on a large model according to claim 5, characterized in that: In step S2), for the graph First, calculate its adaptive adjacency matrix. ; Then, through A learnable state transition matrix, for the adaptive adjacency matrix and picture Perform graph convolution operation to obtain the first... Individual-centered geometric features; The geometric features of all the obtained graphs are flattened and embedded into the same dimension in the MLP. All human-centered geometric features are added to the corresponding human visual features through a learnable scalar weight to obtain human visual-geometric features. Object relationship diagram Perform the same operation to obtain the visual-geometric features of the object; For human body connection diagram Global location map , and and By splicing and mapping, interactive geometric features are obtained.

7. The multimodal fusion human interaction recognition method based on a large model according to claim 6, characterized in that: In step S2), the semantic-visual dual-guidance module includes two Transformer decoders for text semantic features. The semantic features are obtained by incorporating image captions into HOI action labels using a Transformer decoder. It contains HOI interaction information ranging from global image alignment to specific sub-action spaces.

8. The multimodal fusion human interaction recognition method based on a large model according to claim 7, characterized in that: In step S2), in the first Transformer decoder, for time... Textual semantic features and label knowledge From tag knowledge As a query, text semantic features The sum is used for multi-head cross-attention calculation; The output of the last layer is used as semantic features. ; For the second Transformer decoder, semantic features Human visual feature set as a query vector As keys and values, the final output is a visual feature with fused semantics. .

9. The multimodal fusion human interaction recognition method based on a large model according to claim 8, characterized in that: In step S3), the specific details are as follows: For the temporal aspect, a bidirectional recurrent neural network (BiRNN) is used for extraction. Human visual-geometric features at any moment Two-way hidden state ; For spatial edges, a message attention mechanism is designed to enable message passing between entities within and between classes. Information on human categories and object categories is aggregated to obtain aggregated human body information and aggregated object information; And it uses ASSIGN's segment-level network to output segment-level recognition results; When the spatiotemporal graph outputs visually centered human-object interaction sub-action recognition results, it is combined with the visual features fused with semantics. Decision-level fusion is achieved through weighted summation, and the final recognition result is output.

10. A multimodal fusion human interaction recognition system based on a large model, characterized in that, include: The preliminary feature extraction module is used to perform target detection, pose estimation and caption extraction on each frame of the target video to obtain the visual features, geometric features and text features of the target video; The feature extraction and fusion module calls the feature extraction model and performs structured geometric feature extraction through the geometric feature extraction module of the feature extraction model; through the semantic-visual dual-guided module, the text semantic features, HOI action labels, and visual features are subjected to multi-head cross-attention calculation to achieve deep interaction between semantic and visual information and obtain visual features with fused semantics. The interaction recognition module is used to construct a spatiotemporal graph centered on vision, taking the structured geometric features as different categories of entities, obtaining the hidden states of the temporal dimension through a bidirectional recurrent neural network (BiRNN), realizing information interaction between entities within and between categories through a message attention mechanism, and finally outputting the human interaction recognition result through decision-level fusion.

Images