A video action recognition method and system based on space-time interaction Transformer and object interaction prediction

Abstract

The application relates to the field of computer vision technology, and discloses a video action recognition method and system based on space-time interaction Transformer and object interaction prediction, which integrates space-time interaction Transformer and object interaction prediction (STIP) to construct a video action recognition system, can effectively recognize and analyze actions in a video, and has good effects in a multi-person interaction scene, and comprises the following specific steps: preprocessing an original video to obtain necessary video data including a video frame sequence and multi-modal features; encoding the preprocessed video data into a discrete token sequence; inputting the token sequence into a Transformer to perform M-layer self-attention calculation and extract space-time features; using an object interaction prediction module to judge and model the interaction relationship between objects in the video data; and classifying by using the fused multi-modal features to output the action category in the video.

Description

TECHNICAL FIELD

[0001] The application relates to the field of computer vision technology, and particularly relates to a video action recognition method and system based on spatio-temporal interaction Transformer and object interaction prediction. BACKGROUND

[0002] Action recognition technology mainly aims at automatic understanding of video content and is widely applied to video monitoring, human-computer interaction and other fields. Traditional methods rely on manual feature extraction and simple learning models, and it is difficult to effectively process complex dynamic information and rich scene changes in videos. Although significant progress has been made in deep learning-based methods in recent years, how to efficiently integrate information from different modalities and process complex human interactions in videos remains a major challenge. Current methods face the following challenges:

[0003] 1) Detection and recognition separation: most methods need to use external detectors to identify objects, which cannot realize end-to-end learning and need additional computational cost during inference.

[0004] 2) Background-independent noise interference: in the recognition process, there is no effective processing of object interference that may be irrelevant to human action, which affects the accuracy of recognition.

[0005] 3) Insufficient multi-modal information fusion: most existing methods still only focus on video features, and the fusion ability of other modalities such as audio and text is limited, which cannot fully utilize the complementary information between multi-modal data. This leads to insufficient understanding of fine-grained semantics and spatio-temporal correlation mining in complex scenes, thereby affecting the accuracy and robustness of action recognition. SUMMARY

[0006] The application aims to solve the problems of the prior art and provide a video action recognition method and system based on spatio-temporal interaction Transformer and object interaction prediction. The method constructs a video action recognition system by integrating spatio-temporal interaction Transformer and object interaction prediction (STIP), which can effectively recognize and analyze actions in videos, especially in multi-person interaction scenes.

[0007] The application is implemented by the following technical scheme: a video action recognition method based on spatio-temporal interaction Transformer and object interaction prediction, comprising the following specific steps:

[0008] 1) Input video embedding and query vector initialization:

[0009] The original video is preprocessed to obtain necessary video data including video frame sequence and multi-modal features;

[0010] 2) Video embedding encoding and Transformer processing:

[0011] The preprocessed video data is encoded into a discrete token sequence; this token sequence is then input into a Transformer for M-layer self-attention computation to extract spatiotemporal features.

[0012] 3) Object interactivity prediction:

[0013] The object interactivity prediction (STIP) module is used to determine and model the interaction relationships between objects in video data.

[0014] 4) Multimodal feature fusion and classification:

[0015] The processed feature vectors (including video, audio, and text features) are fused with features from other modalities; the fused multimodal features are then used for classification to output the action categories in the video.

[0016] To further improve the video action recognition method based on spatiotemporal interaction Transformer and object interactivity prediction described in this invention, the following configuration is specifically adopted: Step 1) includes the following specific steps:

[0017] 1.1) Obtaining the original video frame sequence:

[0018] Sparse sampling: Sparse sampling is performed on the original video to obtain a fixed number of t frames to form a frame sequence F, where F = {f1, f2, ..., f...} t}, as the input to the model, where f t This represents the raw image data of frame t.

[0019] Feature extraction: Visual features are extracted from the frame sequence F using a convolutional neural network (CNN) to generate a video feature representation V, where V = {v1, v2, ... v...} t}, where v t F represents the high-dimensional feature vector extracted from frame t; F represents the frame sequence of the video; and V represents the video feature representation, which is used for subsequent spatiotemporal coding.

[0020] 1.2) Refinement of Spatio-Temporal Patch Embedding:

[0021] Frame sequence partitioning: The frame sequence F is divided into spatiotemporal blocks of size t×h×w in both time and space dimensions. Each spatiotemporal block is unfolded into a one-dimensional vector and converted into an embedding representation through linear mapping to obtain the token sequence X. emb Then for the token sequence X emb Add time and space location coding PEtime and PE space to reserve the spatio-temporal position information; finally, the token sequence inputting the Transformer is obtained:

[0022] Token Video = X emb + PE time + PE space ;

[0023] wherein, in the time dimension: the I frame is divided by every i frame, and n time blocks are obtained; in the space dimension: the height and width of each frame are divided by h and w pixels, respectively, and n height blocks, width blocks are obtained, wherein H and W are the height and width of the frame; the total number of spatio-temporal blocks is N = n i × n h × n w .

[0024] 1.3) Query vector initialization:

[0025] Multimodal feature extraction: video feature representation V is extracted using CNN, audio feature A is extracted using VGGish, and text feature T is extracted using BERT;

[0026] Feature embedding and fusion: embedding each modal feature into a unified feature space; then, preliminary fusion is performed through weighted summation: Q = W v × V + W a × A + W t × T + b; wherein, W v , W a , W t are the weight matrices of each modal, and b is the bias term.

[0027] Further, in order to better implement the video action recognition method based on spatio-temporal interaction Transformer and object interaction prediction, the following setting mode is particularly adopted: the token sequence in step 2) is subjected to M-layer self-attention calculation in the Transformer, and the calculation process of each layer is:

[0028] lth layer Transformer calculation (l = 1, 2, …, M):

[0029] Input normalization:

[0030] wherein, X (0) = Token Videoo .

[0031] Multi-head self-attention mechanism:

[0032] Query, Key, Value computation:

[0033] where, is the weight matrix of the l-th layer;

[0034] Self-attention computation: Z (l) = MultiHeadAttention(Q (l) , K (l) , V (l) );

[0035] Residual connection: X' (l) = X (l-1) + Z (l) ;

[0036] Feed-forward network:

[0037] Normalization:

[0038] FFN computation:

[0039] Residual connection: X (l) = X' (l) + Y (l) ;

[0040] Output: After M layers of computation, the encoded spatio-temporal feature representation is obtained as: X (M) .

[0041] To better achieve the video action recognition method based on spatio-temporal interaction Transformer and object interaction prediction, the following setting mode is particularly adopted:

[0042] 3.1) Feature integration:

[0043] Fuse the video feature, audio feature and text feature at time step t:

[0044] F t = Concat(V t , A t , T t );

[0045] 3.2) Dynamic gating mechanism:

[0046] Adjust the feature weight: use the interaction score S t to weight the fused feature, highlight the object related to the action:

[0047] F e = S t * F t.

[0048] Further, in order to better achieve the video action recognition method based on spatio-temporal interaction Transformer and object interaction prediction, the following setting mode is particularly adopted: the step 4) is specifically:

[0049] 4.1) Temporal pooling and classification:

[0050] Temporal pooling: the adjusted feature F e is pooled in the time dimension to obtain a global feature vector:

[0051] F f = TemporalPooling(F e );

[0052] Classification: the global feature vector F f is input into a classifier to output the predicted probability of the action category;

[0053] 4.2) Model optimization:

[0054] Loss function: a cross-entropy loss function is used to evaluate the performance of the model:

[0055]

[0056] where y i is the true label, is the predicted probability;

[0057] Parameter update: through the backpropagation algorithm, the loss function L is minimized to update the model parameters.

[0058] A video action recognition system based on spatio-temporal interaction Transformer and object interaction prediction is used to implement the video action recognition method based on spatio-temporal interaction Transformer and object interaction prediction, comprising the following modules:

[0059] A multi-modal feature extraction module uses CNN, VGGish and BERT models to extract feature vectors from video, audio and text to obtain corresponding modal features;

[0060] A multi-scale visual Transformer module uses a multi-head pooling attention mechanism and a Transformer network structure to process the fused multi-modal features to capture complex spatio-temporal relationships.

[0061] An object interaction prediction (STIP) module highlights object features related to actions by using feature fusion, interaction score calculation and dynamic gating mechanisms to predict the interaction between different objects in the video.

[0062] An action classifier uses a full connection layer and a Softmax activation function to classify the extracted global features, output action categories, and obtain action prediction probabilities.

[0063] A global parameter adjustment module adjusts model parameters to improve recognition accuracy according to the classification result, calculates cross-entropy loss, and updates network weights using an optimization algorithm.

[0064] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0065] (1) The present application uses a query vector method, does not require an additional object detector, simplifies the model structure, and realizes end-to-end learning.

[0066] (2) The present application introduces an object interaction prediction (STIP) module, effectively suppresses the influence of objects unrelated to actions on recognition, and improves the accuracy of the model.

[0067] (3) The present application combines audio, text and video data, uses a space-time transformation mechanism, and enhances the action recognition ability of the model in complex scenes. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 The method flowchart of the present application.

[0069] Figure 2 The framework diagram of video action recognition based on a multi-scale visual transformer. DETAILED DESCRIPTION

[0070] The present application will be further described in detail below in conjunction with embodiments, but the embodiments of the present application are not limited thereto.

[0071] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0072] In addition, the terms "first", "second", etc. are used only for descriptive purposes and should not be construed as indicating or implying relative importance or an indicated number of technical features. Therefore, the features defined as "first", "second", etc. can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0073] Noun explanation:

[0074] Transformer: A deep learning model based on self-attention mechanism, good at processing sequence data, widely used in natural language processing and computer vision fields. Transformer processes sequence data through self-attention mechanism, can capture long-distance dependency relationship, realize efficient feature extraction and information integration.

[0075] STIP (Spatio Temporal Interaction Prediction): A module or method for video analysis, predicting and judging the interaction between objects or people through spatio-temporal information. STIP module is usually used in action recognition tasks to improve the accuracy of the model in complex interactive scenarios.

[0076] Token (Token): In natural language processing and deep learning, token usually refers to the smallest unit of text or symbol after segmentation (such as word, character, etc.), used to represent the unit of input data. In video analysis, token can refer to the embedding representation of each spatio-temporal block, used to input the model.

[0077] VGGish: A convolutional neural network (CNN) model improved from VGG model, specially used for audio feature extraction. VGGish model converts audio data into feature vectors, commonly used in multi-modal analysis tasks combining audio and video.

[0078] BERT (Bidirectional Encoder Representations from Transformers): A bidirectional encoder pre-training model based on Transformer, widely used in natural language processing. BERT can generate high-quality text representation vectors through bidirectional training on large-scale text corpus, used in sentiment analysis, text classification, question answering, etc.

[0079] Example 1:

[0080] The application designs a video action recognition method based on space-time interaction Transformer and object interaction prediction, constructs a video action recognition system by integrating space-time interaction Transformer and object interaction prediction (STIP), and can effectively recognize and analyze actions in a video, especially in a multi-person interaction scene, including the following specific steps:

[0081] 1) Input video embedding and query vector initialization:

[0082] The original video is preprocessed to obtain necessary video data including video frame sequences and multi-modal features;

[0083] 2) Video embedding encoding and Transformer processing:

[0084] The preprocessed video data is encoded into a discrete token sequence; the token sequence is input into the Transformer for M-layer self-attention calculation to extract space-time features;

[0085] 3) Object interaction prediction:

[0086] The object interaction prediction (STIP) module is used to judge and model the interaction relationship between objects in the video data; in this step, based on the feature information of the candidate objects obtained by the previous layer of Transformer, it is further judged whether these objects have an interaction relationship, so as to improve the accuracy of action recognition.

[0087] 4) Multi-modal feature fusion and classification:

[0088] The processed feature vector (including video, audio and text features, etc.) is fused with the features of other modalities; the fused multi-modal features are used for classification to output the action category in the video.

[0089] Embodiment 2:

[0090] This embodiment is further optimized on the basis of the above-mentioned embodiment, and the same as the foregoing technical solutions will not be repeated here, and further for better implementation of the application, a video action recognition method based on space-time interaction Transformer and object interaction prediction is described, especially using the following setting method: the step 1) includes the following specific steps:

[0091] 1.1) Original video frame sequence acquisition:

[0092] Sparse sampling: sparse sampling is performed on the original video to obtain a fixed number of t frames to form a frame sequence F, and F={f1, f2,..., ft}, which is used as the input of the model, wherein f t} is the input of the model, wherein f tdenotes the original image data of the t-th frame.

[0093] Feature extraction: extract visual features using a convolutional neural network (CNN) on the frame sequence F, generating a video feature representation V, and V = {v1, v2,.... vn}, where v t t denotes the high-dimensional feature vector extracted from the t-th frame; F represents the frame sequence of the video, V represents the video feature representation, which is used for subsequent spatio-temporal coding.

[0094] 1.2) Spatio-Temporal Patch Embedding refinement process:

[0095] Frame sequence division: divide the frame sequence F in the time and spatial dimensions into spatio-temporal patches of size i x h x w, expand each spatio-temporal patch into a one-dimensional vector, and convert it into an embedded representation through linear mapping to obtain a token sequence X emb ; Then add the position encodings PE emb and PE time of time and space to the token sequence X space to preserve spatio-temporal position information; finally, obtain the token sequence input to the Transformer:

[0096] Token Video = X emb + PE time + PE space ;

[0097] Where, in the time dimension: divide the I frame by every i frame to obtain time blocks; in the spatial dimension: divide the height and width of each frame by h and w pixels to obtain height blocks, width blocks, where H and W are the height and width of the frame; the total number of spatio-temporal blocks is N = n i x n h x n w ;

[0098] 1.3) Query vector initialization:

[0099] Multimodal feature extraction: extract video feature representation V using CNN (Convolutional Neural Network), extract audio feature A using VGGish, and extract text feature T using BERT;

[0100] Feature embedding and fusion: embed each modality feature into a unified feature space; then perform preliminary fusion through weighted summation: Q = W v x V + W a x A + W t x T + b; where W​v , W a , W t is the weight matrix of each modality, and b is the bias term.

[0101] Embodiment 3:

[0102] This embodiment is further optimized on the basis of any of the above embodiments, and the same as the foregoing technical solutions will not be repeated here. Further, in order to better achieve the video action recognition method based on space-time interaction Transformer and object interaction prediction, the following setting method is particularly adopted:

[0103] The first layer of Transformer calculation (l = 1, 2, …, M):

[0104] Input normalization:

[0105] wherein, X (0) = Token Videoo ;

[0106] Multi-head self-attention mechanism:

[0107] Query, key, value calculation:

[0108] wherein, is the weight matrix of the lth layer;

[0109] Self-attention calculation: Z (l) = MultiHeadAttention(Q (l) , K (l) , V (l) );

[0110] Residual connection: X' (l) = X (l-1) + Z (l) ;

[0111] Feedforward network:

[0112] Normalization:

[0113] FFN calculation:

[0114] Residual connection: X (l) = X' (l) + Y (l) ;

[0115] Output: After M layers of calculation, the encoded space-time feature representation is X(M) .

[0116] Embodiment 4:

[0117] This embodiment is further optimized on the basis of any of the above embodiments, and the same as the foregoing technical solutions will not be repeated here. Further, a video action recognition method based on space-time interaction Transformer and object interaction prediction is better implemented, in particular, the following setting method is adopted: the step 3) specifically comprises:

[0118] 3.1) Feature integration:

[0119] Fuse the video features, audio features and text features at time step t:

[0120] F t = Concat(V t ,A t ,T t );

[0121] 3.2) Dynamic gating mechanism:

[0122] Adjust the feature weight: use the interaction score S t to weight the fused features to highlight the objects related to the action:

[0123] F e =S t ×F t .

[0124] Embodiment 5:

[0125] This embodiment is further optimized on the basis of any of the above embodiments, and the same as the foregoing technical solutions will not be repeated here. Further, a video action recognition method based on space-time interaction Transformer and object interaction prediction is better implemented, in particular, the following setting method is adopted: the step 4) specifically comprises:

[0126] 4.1) Temporal pooling and classification:

[0127] Temporal pooling: pool the adjusted features F e in the time dimension to obtain a global feature vector:

[0128] F f = TemporalPooling(F e );

[0129] Classification: input the global feature vector F f into a classifier to output the prediction probability of the action category;

[0130] 4.2) Model optimization:

[0131] Loss function: Evaluate model performance using cross-entropy loss function:

[0132]

[0133] where y i is the true label, is the predicted probability;

[0134] Parameter update: Update model parameters by minimizing the loss function L through the backpropagation algorithm.

[0135] Example 6:

[0136] This embodiment is further optimized on the basis of any of the above embodiments, and the same as the foregoing technical solutions will not be repeated here. A video action recognition system based on spatiotemporal interaction Transformer and object interaction prediction is used to implement the video action recognition method based on spatiotemporal interaction Transformer and object interaction prediction, which includes the following modules:

[0137] A multi-modal feature extraction module uses CNN, VGGish, and BERT models to extract feature vectors from video, audio, and text to obtain corresponding modal features.

[0138] A multi-scale visual Transformer module uses a multi-head pooling attention mechanism and a Transformer network structure to process the fused multi-modal features and capture complex spatiotemporal relationships.

[0139] An object interaction prediction (STIP) module highlights object features related to actions by using feature fusion, interaction score calculation, and dynamic gating mechanisms to predict the interaction between different objects in the video.

[0140] An action classifier uses a fully connected layer and a Softmax activation function to classify the extracted global features, output the action class, and obtain the action prediction probability.

[0141] A global parameter adjustment module adjusts model parameters to improve recognition accuracy based on the classification results, calculates the cross-entropy loss, and updates the network weights using an optimization algorithm.

[0142] Example 7:

[0143] Referring to Figure 1 and Figure 2 , this embodiment provides a video action recognition method based on spatiotemporal interaction Transformer and object interaction prediction (STIP), which includes the following steps:

[0144] S1. Input video embedding and query vector initialization:

[0145] S1.1 Video Preprocessing and Feature Extraction:

[0146] Sparse sampling is performed on the original video to obtain a fixed number of t frames, forming a frame sequence:

[0147] F = {f1, f2, ..., f t};

[0148] Where t is the total number of frames after sampling. Next, a convolutional neural network (CNN) is used to extract video features from the frame sequence F, resulting in the video feature representation:

[0149] V = {v1, v2, ..., v} t};

[0150] Among them, each v t Let be the high-dimensional feature vector corresponding to frame t.

[0151] S1.2 Spatio-Temporal Patch Embedding:

[0152] The I-frame is divided into several spatiotemporal blocks along the temporal and spatial dimensions (height H and width W). Each block has a length of i (where i is the size of the temporal block) in the temporal dimension and is divided in units of h and w in the spatial dimension. For example, the size of each spatiotemporal block is i×h×w.

[0153] Therefore, we can conclude that:

[0154] n i = I / i (number of time dimension blocks),

[0155] n h =H / h (number of blocks in the spatial height direction),

[0156] n w = W / w (number of blocks in the spatial width direction).

[0157] The total number of spacetime blocks is N = n i ×n h ×n w .

[0158] For each spatiotemporal block, a corresponding embedding vector is generated through linear mapping, and temporal and spatial location encoding (P) is added. t With P s The token sequence of the input Transformer is obtained as follows:

[0159] X = X emb +P t +P s.

[0160] S1.3 Initialization of query vector Q (multimodal fusion):

[0161] In addition to video features V, features are also extracted from audio and text data:

[0162] Audio features A: extracted from audio using VGGish model;

[0163] Text features T: extracted from subtitle text using BERT model; (T here represents text features)

[0164] After mapping the three features (video V, audio A, text T) to a unified dimensional feature space, they are weightedly fused:

[0165] Q = W v * V + W a * A + W t * T + b

[0166] where W v , W a , W t are learnable weight matrices, and b is a bias term.

[0167] S2, encoded by self-attention mechanism:

[0168] The token sequence X of the video is input into the Transformer encoder. The Transformer uses the self-attention mechanism to capture the spatio-temporal dependencies between elements in the sequence by computing Q, K, V (this Q, K, V is the query, key, value in the attention mechanism, which is different from the query vector Q in step S1.3, and is a homonym different internal calculation symbol), generating an intermediate representation rich in spatio-temporal interaction information.

[0169] S3, after M-layer Transformer encoding, the query vector is decoded:

[0170] After processing by the M-layer self-attention encoder, the multimodal query vector Q combines the video spatio-temporal features to obtain a comprehensive representation. The encoded representation is input into the decoder to obtain a decoded vector containing rich spatio-temporal interaction information, providing input for subsequent interaction prediction and classification.

[0171] S4, use STIP (spatio-temporal interaction prediction) to obtain frame-level interaction representation:

[0172] The decoded query vector is passed into the STIP module to identify and model the interaction between the characters and objects in the video. The STIP module outputs a fine-grained interaction feature representation for each frame, highlighting the key objects and character relationships related to the target action.

[0173] S5, feature splicing and re-encoding:

[0174] The interaction features output by STIP are spliced with the tokens of the video sequence to form an enhanced feature sequence. The sequence is input into a self-attention layer for encoding, thereby integrating interaction information and video spatiotemporal features in a global range to obtain more robust action representations.

[0175] S6, action classification and optimization using fused interaction information and spatiotemporal representations:

[0176] S6.1 Temporal pooling and classification:

[0177] The fused features are pooled in the temporal dimension (TemporalPooling) to obtain a global feature vector F f :

[0178] F f = TemporalPooling(F e )

[0179] where F e is the feature representation after spatiotemporal interaction and STIP enhancement.

[0180] F f is input into a classifier composed of a fully connected layer and a Softmax activation function, and the output is the predicted probability distribution of each action class

[0181] S6.2 Model training and loss function optimization:

[0182] The cross-entropy loss function is used as the optimization objective to measure the difference between the predicted probability and the true label:

[0183]

[0184] where N is the total number of action classes, y i is the one-hot encoding of the true label, is the predicted probability of the model for the i-th action class. Through backpropagation and optimization algorithm (Adam), the model parameters are continuously adjusted to improve the accuracy of action recognition.

[0185] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification or equivalent change based on the technical essence of the present application to the above embodiment falls within the scope of protection of the present application.

Claims

1. A video action recognition method based on spatiotemporal interaction Transformer and object interactivity prediction, characterized in that: The specific steps include the following: 1) Input video embedding and query vector initialization: Preprocessing the original video to obtain necessary video data, including video frame sequences and multimodal features, includes the following specific steps: 1.1) Obtaining the original video frame sequence: Sparse sampling: Sparse sampling is performed on the original video to obtain a fixed number of t frames to form a frame sequence F, where F = {f1, f2, ..., f...} t }, as the input to the model; Feature extraction: Visual features are extracted from the frame sequence F using a convolutional neural network to generate a video feature representation V, where V = {v1, v2, ... v}. t }, where v t This represents the high-dimensional feature vector extracted from frame t. 1.2) Spatiotemporal block embedding refinement processing: Frame sequence partitioning: The frame sequence F is divided into spatiotemporal blocks of size i×h×w in both time and space dimensions. Each spatiotemporal block is unfolded into a one-dimensional vector and converted into an embedding representation through linear mapping to obtain the token sequence X. emb Then for the token sequence X emb Add time and space location coding PE time and PE space Finally, we obtain the token sequence input to the Transformer: Token Video =X emb +PE time +PE space 4 In the time dimension: I-frames are divided into i-frames, resulting in Each time block; in the spatial dimension: the height and width of each frame are divided into h and w pixels respectively, resulting in A height block, There are n width blocks, where H and W are the height and width of the frame; the total number of spatiotemporal blocks is N = n i ×n h ×n w ; 1.3) Query vector initialization: Multimodal feature extraction: CNN is used to extract video feature representation V, VGGish is used to extract audio feature A, and BERT is used to extract text feature T; Feature embedding and fusion: Features from various modalities are embedded into a unified feature space; then, preliminary fusion is performed through weighted summation: Q = W v ×V+W a ×A+W t ×T+b; where W v W a W t Let b be the weight matrix for each mode, and b be the bias term. 2) Video embedding encoding and Transformer processing: The preprocessed video data is encoded into a discrete token sequence; the token sequence is then input into a Transformer to perform M-layer self-attention calculations and extract spatiotemporal features. 3) Object interactivity prediction: The object interactivity prediction module is used to determine and model the interaction relationships between objects in video data, specifically including: 3.1) Feature integration: The video features, audio features, and text features at time step t are fused together: F t =Concat(V t ,A t ,T t ); 3.2) Dynamic gating mechanism: Adjusting feature weights: utilizing interaction score S t The merged features are weighted to highlight objects relevant to the action: F e =S t ×F t ; 4) Multimodal feature fusion and classification: The processed feature vectors are fused with features from other modalities; the fused multimodal features are then used for classification to output the action categories in the video.

2. The video action recognition method based on spatiotemporal interaction Transformer and object interactivity prediction according to claim 1, characterized in that: In step 2), the token sequence undergoes M-layer self-attention computation in the Transformer. The computation process for each layer is as follows: The l-th layer Transformer is calculated (l = 1, 2, ..., M): Input normalization: Among them, X (0) =Token Videoo ; Multi-head self-attention mechanism: Query, key-value calculation: in, Here is the weight matrix for the l-th layer; Self-attention calculation: Z (l) =MultiHeadAttention(Q (l) ,K (l) V (l) ); Residual connection: X' (l) =X (l-1) +Z (l) ; Feedforward networks: Normalization: FFN calculation: Residual connection: X (l) =X' (l) +Y (l) ; Output: After M layers of computation, the encoded spatiotemporal feature representation is obtained as: X (M) .

3. The video action recognition method based on spatiotemporal interaction Transformer and object interactivity prediction according to claim 1, characterized in that: Step 4) specifically involves: 4.1) Time Pooling and Classification: Temporal pooling: For the adjusted feature F e Pooling is performed along the time dimension to obtain the global feature vector: F f =TemporalPooling(F e ); Classification: The global feature vector F f Input the classifier and output the predicted probability of the action category; 4.2) Model Optimization: Loss function: Use the cross-entropy loss function to evaluate model performance. Among them, y i For real labels, To predict probabilities; Parameter update: The model parameters are updated by minimizing the loss function L through the backpropagation algorithm.

4. A video action recognition system based on spatiotemporal interaction Transformer and object interactivity prediction, characterized in that: A video action recognition method based on spatiotemporal interaction Transformer and object interactivity prediction as described in claims 1-3 includes the following modules: The multimodal feature extraction module uses CNN, VGGish and BERT models to extract feature vectors from video, audio and text to obtain the corresponding modal features; The multi-scale visual Transformer module utilizes a multi-head pooling attention mechanism and a Transformer network structure to process fused multimodal features and capture complex spatiotemporal relationships. The object interactivity prediction module uses feature fusion, interactivity score calculation, and dynamic gating mechanism to highlight action-related object features and predict the interactivity between different objects in a video. An action classifier uses fully connected layers and a softmax activation function to classify extracted global features, output action categories, and obtain action prediction probabilities. The global parameter adjustment module adjusts model parameters to improve recognition accuracy based on classification results; it calculates cross-entropy loss and uses optimization algorithms to update network weights.

Images