Video action recognition method and system based on time-space interaction Transform and object interactivity prediction

Abstract

The invention relates to the field of computer vision technology and the like, and discloses a video action recognition method and system based on space-time interaction Transform and object interactivity prediction.The method constructs the video action recognition system by integrating the space-time interaction Transform and the object interactivity prediction.The action in a video can be effectively recognized and analyzed, and the video action recognition efficiency is improved. The method has a good effect especially in a multi-user interaction scene, and comprises the following specific steps: preprocessing an original video, and obtaining necessary video data comprising a video frame sequence and multi-modal features; encoding the preprocessed video data into a discrete token sequence; the token sequence is input into Transform, M-layer self-attention calculation is carried out, and spatial-temporal features are extracted; an object interactivity prediction module is used for judging and modeling the interaction relationship between objects in the video data; and carrying out classification by utilizing the fused multi-modal features, and outputting action categories in the video.

Description

Technical Field

[0001] The present invention relates to fields such as computer vision technology, and in particular to a video action recognition method and system based on spatiotemporal interactive Transformer and object interactivity prediction. Background Art

[0002] Action recognition technology is mainly aimed at automatic understanding of video content and is widely used in video surveillance, human-computer interaction and other fields. Traditional methods rely on manual feature extraction and simple learning models, which are difficult to effectively handle complex dynamic information and rich scene changes in videos. Although deep learning-based methods have made significant progress in recent years, how to efficiently integrate information from different modalities and handle complex character interactions in videos remains a major challenge. Current methods face the following challenges:

[0003] 1) Separation of detection and recognition: Most methods require the use of external detectors to identify objects, which cannot achieve end-to-end learning and requires additional computational costs during reasoning.

[0004] 2) Background irrelevant noise interference: During the recognition process, the interference of objects that may be irrelevant to the person’s movements is not effectively processed, thus affecting the accuracy of recognition.

[0005] 3) Insufficient fusion of multimodal information: Most existing methods still only focus on video features, and have limited fusion capabilities for other modal features such as audio and text, and cannot fully utilize the complementary information between multimodal data. This leads to insufficient fine-grained semantic understanding and spatiotemporal association mining in complex scenes, thus affecting the accuracy and robustness of action recognition. Summary of the invention

[0006] The purpose of the present invention is to solve the shortcomings of the prior art and provide a video action recognition method and system based on spatiotemporal interactive Transformer and object interactivity prediction. The method constructs a video action recognition system by integrating spatiotemporal interactive Transformer and object interactivity prediction (STIP), which can effectively recognize and analyze actions in videos, especially in multi-person interaction scenarios.

[0007] The present invention is implemented by the following technical solution: a video action recognition method based on spatiotemporal interactive Transformer and object interactivity prediction, comprising the following specific steps:

[0008] 1) Input video embedding and query vector initialization:

[0009] Preprocess the original video to obtain necessary video data including video frame sequence and multimodal features;

[0010] 2) Video Embedding Coding and Transformer Processing:

[0011] Encode the preprocessed video data into a discrete token sequence; input the token sequence into the Transformer, perform M-layer self-attention calculations, and extract spatiotemporal features;

[0012] 3) Object interactivity prediction:

[0013] Use the STIP module to determine and model the interaction between objects in video data.

[0014] 4) Multimodal feature fusion and classification:

[0015] The processed feature vector (including video, audio and text features, etc.) is fused with features of other modalities; the fused multimodal features are used for classification to output the action categories in the video.

[0016] In order to better implement the video action recognition method based on spatiotemporal interactive Transformer and object interactivity prediction described in the present invention, the following setting is particularly adopted: the step 1) includes the following specific steps:

[0017] 1.1) Original video frame sequence acquisition:

[0018] Sparse sampling: Sparse sampling is performed on the original video to obtain a fixed number of t frames to form a frame sequence F, where F = {f1, f2, ..... f t ,}, as the input of the model, where f t Represents the original image data of the tth frame.

[0019] Feature extraction: Use convolutional neural network (CNN) to extract visual features from the frame sequence F and generate video feature representation V, where V = {v1, v2, ..... v T ,}, where v t represents the high-dimensional feature vector extracted from the tth frame; F represents the frame sequence of the video, and V represents the video feature representation, which is used for subsequent spatiotemporal coding.

[0020] 1.2) Spatio-Temporal Patch Embedding Refinement Processing:

[0021] Frame sequence division: Divide the frame sequence F into spatiotemporal blocks of size t×h×w in time and space dimensions, expand each spatiotemporal block into a one-dimensional vector, and convert it into an embedded representation through linear mapping to obtain the token sequence X emb ; Then the token sequence X emb Add time and space position encoding PEtime and PE space , to preserve the spatiotemporal position information; finally, the token sequence input to the Transformer is obtained:

[0022] Token Video =X emb +PE time +PE space ;

[0023] Among them, in the time dimension: divide T frames into every t frames, and get In the spatial dimension, the height and width of each frame are divided into h and w pixels respectively, and we get Height blocks, width blocks, where H and W are the height and width of the frame; the total number of spatiotemporal blocks is N = n t ×n h ×n w ;

[0024] 1.3) Query vector initialization:

[0025] Multimodal feature extraction: Use CNN to extract video feature representation V, use VGGish to extract audio features A, and use BERT to extract text features T;

[0026] Feature embedding and fusion: embed each modal feature into a unified feature space; then perform preliminary fusion through weighted summation: Q = W v ×V+W a ×A+W t ×T+b; where W v , W a , W t is the weight matrix of each mode, and b is the bias term.

[0027] In order to better implement the video action recognition method based on spatiotemporal interactive Transformer and object interactivity prediction described in the present invention, the following setting is particularly adopted: in step 2), the token sequence performs M-layer self-attention calculation in the Transformer, and the calculation process of each layer is:

[0028] Transformer calculation at layer l (l = 1, 2, ..., M):

[0029] Input Normalization:

[0030] Among them, X (0) =Token Videoo ;

[0031] Multi-head self-attention mechanism:

[0032] Query, key, and value calculation:

[0033] in, is the weight matrix of the lth layer;

[0034] Self-attention calculation: Z (l) =MultiHeadAttention(Q (l) ,K (l) ,V (l) );

[0035] Residual connection: X ′(l) =X (l-1) +Z (l) ;

[0036] Feedforward Network:

[0037] Normalization:

[0038] FFN calculation:

[0039] Residual connection: X (l) =X ′(l) +Y (l) ;

[0040] Output: After M layers of calculation, the encoded spatiotemporal features are expressed as: X (M) .

[0041] In order to better implement the video action recognition method based on spatiotemporal interactive Transformer and object interactivity prediction described in the present invention, the following setting is particularly adopted: the step 3) specifically includes:

[0042] 3.1) Feature Integration:

[0043] Fusion of video features, audio features and text features at time step t:

[0044] F t =Concat(V t ,A t ,T t );

[0045] 3.2) Dynamic gating mechanism:

[0046] Adjusting feature weights: Using the interactivity score I t Weight the fused features to highlight objects related to the action:

[0047] F e =I t ×F t .

[0048] In order to better implement the video action recognition method based on spatiotemporal interactive Transformer and object interactivity prediction described in the present invention, the following setting is particularly adopted: the step 4) is specifically:

[0049] 4.1) Temporal pooling and classification:

[0050] Time pooling: After adjustment, the feature F e Pooling is performed in the time dimension to obtain the global feature vector:

[0051] F f = TemporalPooling(F e );

[0052] Classification: The global feature vector F f Input classifier, output predicted probability of action category;

[0053] 4.2) Model optimization:

[0054] Loss function: The cross entropy loss function is used to evaluate the model performance:

[0055]

[0056] Among them, y i is the true label, is the predicted probability;

[0057] Parameter update: Through the back-propagation algorithm, minimize the loss function L and update the model parameters.

[0058] A video action recognition system based on spatiotemporal interactive Transformer and object interactivity prediction, used to implement the video action recognition method based on spatiotemporal interactive Transformer and object interactivity prediction, comprising the following modules:

[0059] The multimodal feature extraction module uses CNN, VGGish and BERT models to extract feature vectors from video, audio and text to obtain corresponding modal features;

[0060] The multi-scale visual Transformer module uses the multi-head pooling attention mechanism and Transformer network structure to process the fused multimodal features and capture complex spatiotemporal relationships.

[0061] The STIP module highlights the action-related object features and predicts the interactivity between different objects in the video through feature fusion, interactivity score calculation and dynamic gating mechanism.

[0062] The action classifier uses a fully connected layer and a Softmax activation function to classify the extracted global features, output the action category, and obtain the action prediction probability.

[0063] The global parameter adjustment module adjusts the model parameters to improve the recognition accuracy according to the classification results; calculates the cross entropy loss and uses the optimization algorithm to update the network weights.

[0064] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0065] (1) The present invention adopts the query vector method, does not require an additional object detector, simplifies the model structure, and realizes end-to-end learning.

[0066] (2) The present invention introduces the STIP module, which effectively suppresses the influence of objects unrelated to the action on the recognition and improves the accuracy of the model.

[0067] (3) The present invention combines audio, text and video data and utilizes the spatiotemporal transformation mechanism to enhance the model's action recognition capability in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 The figure is a flow chart of the method of the present invention.

[0069] Figure 2 Framework diagram for video action recognition based on multi-scale visual transformer. DETAILED DESCRIPTION

[0070] The present invention is further described in detail below in conjunction with examples, but the embodiments of the present invention are not limited thereto.

[0071] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0072] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0073] Glossary:

[0074] Transformer: A deep learning model based on the self-attention mechanism, which is good at processing sequence data and is widely used in natural language processing and computer vision. Transformer processes sequence data through the self-attention mechanism, which can capture long-distance dependencies and achieve efficient feature extraction and information integration.

[0075] STIP (Spatio Temporal Interaction Prediction): A module or method for video analysis that predicts and determines the interactivity between objects or people based on spatiotemporal information. The STIP module is often used in action recognition tasks to improve the accuracy of the model in complex interaction scenarios.

[0076] Token: In natural language processing and deep learning, a token usually refers to the smallest text unit or symbol (such as a word, character, etc.) after segmentation, which is used to represent the unit of input data. In video analysis, a token can refer to the embedded representation of each spatiotemporal block, which is used to input the model.

[0077] VGGish: A convolutional neural network (CNN) model that is improved from the VGG model and is specifically used for audio feature extraction. The VGGish model converts audio data into feature vectors and is often used for multimodal analysis tasks that combine audio and video.

[0078] BERT (Bidirectional Encoder Representations from Transformers): A Transformer-based bidirectional encoder pre-training model widely used in natural language processing. BERT can generate high-quality text representation vectors for tasks such as sentiment analysis, text classification, and question answering by performing bidirectional training on large-scale text corpora.

[0079] Embodiment 1:

[0080] The present invention designs a video action recognition method based on spatiotemporal interactive Transformer and object interactivity prediction. By integrating spatiotemporal interactive Transformer and object interactivity prediction (STIP) to build a video action recognition system, the method can effectively recognize and analyze actions in the video, especially in multi-person interaction scenes, and includes the following specific steps:

[0081] 1) Input video embedding and query vector initialization:

[0082] Preprocess the original video to obtain necessary video data including video frame sequence and multimodal features;

[0083] 2) Video Embedding Coding and Transformer Processing:

[0084] Encode the preprocessed video data into a discrete token sequence; input the token sequence into the Transformer, perform M-layer self-attention calculations, and extract spatiotemporal features;

[0085] 3) Object interactivity prediction:

[0086] The object interactivity prediction (STIP) module is used to judge and model the interaction relationship between objects in the video data. In this step, based on the feature information of the candidate objects obtained by the previous layer of Transformer, it is further judged whether these objects have an interaction relationship to improve the accuracy of action recognition.

[0087] 4) Multimodal feature fusion and classification:

[0088] The processed feature vector (including video, audio and text features, etc.) is fused with features of other modalities; the fused multimodal features are used for classification to output the action categories in the video.

[0089] Embodiment 2:

[0090] This embodiment is further optimized on the basis of the above embodiment, and the same points as the above technical solution are not repeated here. In order to better realize the video action recognition method based on spatiotemporal interactive Transformer and object interactivity prediction described in the present invention, the following setting is particularly adopted: the step 1) includes the following specific steps:

[0091] 1.1) Original video frame sequence acquisition:

[0092] Sparse sampling: Sparse sampling is performed on the original video to obtain a fixed number of t frames to form a frame sequence F, where F = {f1, f2, ..... f t ,}, as the input of the model, where f tRepresents the original image data of the tth frame.

[0093] Feature extraction: Use convolutional neural network (CNN) to extract visual features from the frame sequence F and generate video feature representation V, where V = {v1, v2, ..... v T ,}, where v t represents the high-dimensional feature vector extracted from the tth frame; F represents the frame sequence of the video, and V represents the video feature representation, which is used for subsequent spatiotemporal coding.

[0094] 1.2) Spatio-Temporal Patch Embedding Refinement Processing:

[0095] Frame sequence division: Divide the frame sequence F into spatiotemporal blocks of size t×h×w in time and space dimensions, expand each spatiotemporal block into a one-dimensional vector, and convert it into an embedded representation through linear mapping to obtain the token sequence X emb ; Then the token sequence X emb Add time and space position encoding PE time and PE space , to preserve the spatiotemporal position information; finally, the token sequence input to the Transformer is obtained:

[0096] Token Video =X emb +PE time +PE space ;

[0097] Among them, in the time dimension: divide T frames into every t frames, and get In the spatial dimension, the height and width of each frame are divided into h and w pixels respectively, and we get Height blocks, width blocks, where H and W are the height and width of the frame; the total number of spatiotemporal blocks is N = n t ×n h ×n w ;

[0098] 1.3) Query vector initialization:

[0099] Multimodal feature extraction: Use CNN (convolutional neural network) to extract video feature representation V, use VGGish to extract audio features A, and use BERT to extract text features T;

[0100] Feature embedding and fusion: embed each modal feature into a unified feature space; then perform preliminary fusion through weighted summation: Q = W v ×V+W a ×A+W t ×T+b; where Wv , W a , W t is the weight matrix of each mode, and b is the bias term.

[0101] Embodiment 3:

[0102] This embodiment is further optimized on the basis of any of the above embodiments, and the same points as the above technical solutions are not repeated here. In order to better realize the video action recognition method based on spatiotemporal interactive Transformer and object interactivity prediction described in the present invention, the following setting is particularly adopted: in step 2), the token sequence performs M-layer self-attention calculation in the Transformer, and the calculation process of each layer is:

[0103] Transformer calculation at layer l (l = 1, 2, ..., M):

[0104] Input Normalization:

[0105] Among them, X (0) =Token Videoo ;

[0106] Multi-head self-attention mechanism:

[0107] Query, key, and value calculation:

[0108] in, is the weight matrix of the lth layer;

[0109] Self-attention calculation: Z (l) =MultiHeadAttention(Q (l) ,K (l) ,V (l) );

[0110] Residual connection: X ′(l) =X (l-1) +Z (l) ;

[0111] Feedforward Network:

[0112] Normalization:

[0113] FFN calculation:

[0114] Residual connection: X (l) =X ′(l) +Y (l) ;

[0115] Output: After M layers of calculation, the encoded spatiotemporal features are expressed as: X(M) .

[0116] Embodiment 4:

[0117] This embodiment is further optimized on the basis of any of the above embodiments, and the similarities with the above technical solutions are not repeated here. In order to better implement the video action recognition method based on spatiotemporal interactive Transformer and object interactivity prediction described in the present invention, the following setting is particularly adopted: the step 3) specifically includes:

[0118] 3.1) Feature Integration:

[0119] Fusion of video features, audio features and text features at time step t:

[0120] F t =Concat(V t ,A t ,T t );

[0121] 3.2) Dynamic gating mechanism:

[0122] Adjusting feature weights: Using the interactivity score I t Weight the fused features to highlight objects related to the action:

[0123] F e =I t ×F t .

[0124] Embodiment 5:

[0125] This embodiment is further optimized on the basis of any of the above embodiments, and the same points as the above technical solutions are not repeated here. In order to better realize the video action recognition method based on spatiotemporal interactive Transformer and object interactivity prediction described in the present invention, the following setting is particularly adopted: the step 4) is specifically:

[0126] 4.1) Temporal pooling and classification:

[0127] Time pooling: After adjustment, the feature F e Pooling is performed in the time dimension to obtain the global feature vector:

[0128] F f = TemporalPooling(F e );

[0129] Classification: The global feature vector F f Input classifier, output predicted probability of action category;

[0130] 4.2) Model optimization:

[0131] Loss function: The cross entropy loss function is used to evaluate the model performance:

[0132]

[0133] Among them, y i is the true label, is the predicted probability;

[0134] Parameter update: Through the back-propagation algorithm, minimize the loss function L and update the model parameters.

[0135] Embodiment 6:

[0136] This embodiment is further optimized on the basis of any of the above embodiments, and the similarities with the above technical solutions are not repeated here. A video action recognition system based on spatiotemporal interactive Transformer and object interactivity prediction is used to implement the video action recognition method based on spatiotemporal interactive Transformer and object interactivity prediction, including the following modules:

[0137] The multimodal feature extraction module uses CNN, VGGish and BERT models to extract feature vectors from video, audio and text to obtain corresponding modal features;

[0138] The multi-scale visual Transformer module uses the multi-head pooling attention mechanism and Transformer network structure to process the fused multimodal features and capture complex spatiotemporal relationships.

[0139] The STIP module highlights the action-related object features and predicts the interactivity between different objects in the video through feature fusion, interactivity score calculation and dynamic gating mechanism.

[0140] The action classifier uses a fully connected layer and a Softmax activation function to classify the extracted global features, output the action category, and obtain the action prediction probability.

[0141] The global parameter adjustment module adjusts the model parameters to improve the recognition accuracy according to the classification results; calculates the cross entropy loss and uses the optimization algorithm to update the network weights.

[0142] Embodiment 7:

[0143] See also Figure 1 and Figure 2 This embodiment provides a video action recognition method based on spatiotemporal interactive Transformer and object interactivity prediction (STIP), comprising the following steps:

[0144] S1, input video embedding and query vector initialization:

[0145] S1.1 Video preprocessing and feature extraction:

[0146] Sparsely sample the original video to obtain a fixed number of T frames to form a frame sequence:

[0147] F={f1,f2,...,f T};

[0148] Among them, T is the total number of frames after sampling. Then, the convolutional neural network (CNN) is used to extract video features from the frame sequence F to obtain the video feature representation:

[0149] V={v1,v2,...,v T};

[0150] Among them, each v t is the high-dimensional feature vector corresponding to the t-th frame.

[0151] S1.2 Spatio-Temporal Patch Embedding:

[0152] The T frame is divided into several spatiotemporal blocks along the time dimension and the space dimension (height H and width W), each block has a length of τ in the time dimension (τ is the time block size), and is divided into units of h and w in space. For example, the size of each spatiotemporal block is τ×h×w.

[0153] From this we can get:

[0154] nt = T / τ (number of time dimension blocks),

[0155] nh = H / h (number of blocks in the spatial height direction),

[0156] nw=W / w (number of blocks in the spatial width direction).

[0157] The total number of space-time blocks is N = n t ×n h ×n w .

[0158] For each spatiotemporal block, a corresponding embedding vector is generated by linear mapping, and the temporal and spatial position encoding (P t With P s )Get the token sequence of input Transformer:

[0159] X=X emb +P t +P s .

[0160] S1.3 Initialization of query vector Q (multimodal fusion):

[0161] In addition to the video features V, features are also extracted from audio and text data:

[0162] Audio feature A: extracted from audio using the VGGish model;

[0163] Text features U: extracted from the subtitle text using the BERT model (U is used here to represent text features to avoid confusion with the number of frames T).

[0164] The three features (video V, audio A, text U) are mapped to a feature space of uniform dimension and then weightedly fused:

[0165] Q=W v *V+W a *A+W t *U+b

[0166] Among them, W v , W a , W t is the learnable weight matrix and b is the bias term.

[0167] S2, encoded by self-attention mechanism:

[0168] The token sequence X of the video is input into the Transformer encoder. Transformer uses the self-attention mechanism to capture the spatiotemporal dependencies between elements in the sequence by calculating Q, K, and V (these Q, K, and V are the query, key, and value in the attention mechanism, which are different from the query vector Q in step S1.3 and are internal calculation symbols with the same name but different meanings) to generate an intermediate representation rich in spatiotemporal interaction information.

[0169] S3. Decode the query vector after encoding it through M layers of Transformers:

[0170] After being processed by the M-layer self-attention encoder, the multimodal query vector Q is combined with the spatiotemporal features of the video to obtain a comprehensive representation. The encoded representation is input into the decoder to obtain a decoding vector containing rich spatiotemporal interaction information, which provides input for subsequent interactivity prediction and classification.

[0171] S4. Use STIP (Spatial-temporal Interactivity Prediction) to get frame-level interaction representation:

[0172] The decoded query vector is passed to the STIP module to identify and model the interaction between people and objects in the video. The STIP module outputs a fine-grained interaction feature representation for each frame, highlighting the key objects and people relationships related to the target action.

[0173] S5. Feature concatenation and re-encoding:

[0174] The interaction features output by STIP are concatenated with the tokens of the video sequence to form an enhanced feature sequence. This sequence is again input into the self-attention layer for encoding, thereby integrating the interaction information and the spatiotemporal features of the video globally to obtain a more robust action representation.

[0175] S6. Use the fused interaction information and spatiotemporal representation to classify and optimize actions:

[0176] S6.1 Temporal pooling and classification:

[0177] The fused features are pooled in the time dimension (TemporalPooling) to obtain the global feature vector F_f:

[0178] F f = TemporalPooling(F e )

[0179] Among them, F e It is the feature representation after spatiotemporal interaction and STIP enhancement.

[0180] F f Input the classifier composed of the fully connected layer and the Softmax activation function, and output the predicted probability distribution of various actions

[0181] S6.2 Model training and loss function optimization:

[0182] The cross entropy loss function is used as the optimization objective to measure the difference between the predicted probability and the true label:

[0183]

[0184] Where N is the total number of action categories, y i is the one-hot encoding of the true label, is the predicted probability of the model for the i-th action. Through back propagation and optimization algorithm (Adam), the model parameters are continuously adjusted to improve the accuracy of action recognition.

[0185] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A video action recognition method based on spatiotemporal interactive Transformer and object interactivity prediction, characterized by: The specific steps include: 1) Input video embedding and query vector initialization: Preprocess the original video to obtain necessary video data including video frame sequence and multimodal features; 2) Video Embedding Coding and Transformer Processing: Encode the preprocessed video data into a discrete token sequence; input the token sequence into the Transformer, perform M-layer self-attention calculations, and extract spatiotemporal features; 3) Object interactivity prediction: Use the object interactivity prediction module to judge and model the interaction relationship between objects in the video data; 4) Multimodal feature fusion and classification: The processed feature vector is fused with features of other modalities; the fused multimodal features are used for classification to output the action categories in the video.

2. According to claim 1, a video action recognition method based on spatiotemporal interactive Transformer and object interactivity prediction is characterized by: The step 1) comprises the following specific steps: 1.1) Original video frame sequence acquisition: Sparse sampling: Sparse sampling is performed on the original video to obtain a fixed number of t frames to form a frame sequence F, where F = {f1, f2, ..... f t ,}, as the input of the model; Feature extraction: Use convolutional neural network to extract visual features from the frame sequence F and generate video feature representation V, where V={v1,v2,.....v T ,}, where v t represents the high-dimensional feature vector extracted from the t-th frame; 1.2) Spatiotemporal block embedding refinement processing: Frame sequence division: Divide the frame sequence F into spatiotemporal blocks of size t×h×w in time and space dimensions, expand each spatiotemporal block into a one-dimensional vector, and convert it into an embedded representation through linear mapping to obtain the token sequence X emb ; Then the token sequence X emb Add time and space position encoding PE time and PE space ; Finally, we get the token sequence that is input into the Transformer: Token Video =X emb +PE time +PE space 4 Among them, in the time dimension: divide T frames into every t frames, and get In the spatial dimension, the height and width of each frame are divided into h and w pixels respectively, and we get Height blocks, width blocks, where H and W are the height and width of the frame; the total number of spatiotemporal blocks is N = n t ×n h ×n w ; 1.3) Query vector initialization: Multimodal feature extraction: Use CNN to extract video feature representation V, use VGGish to extract audio features A, and use BERT to extract text features T; Feature embedding and fusion: embed each modal feature into a unified feature space; then perform preliminary fusion through weighted summation: Q = W v ×V+W a ×A+W t ×T+b; where W v , W a , W t is the weight matrix of each mode, and b is the bias term.

3. The video action recognition method based on spatiotemporal interactive Transformer and object interactivity prediction according to claim 2, characterized in that: In step 2), the token sequence is subjected to M-layer self-attention calculation in the Transformer, and the calculation process of each layer is: Transformer calculation at layer l (l = 1, 2, ..., M): Input normalization: Among them, X (0) =Token Videoo ; Multi-head self-attention mechanism: Query, key, and value calculation: in, is the weight matrix of the lth layer; Self-attention calculation: Z (l) =MultiHeadAttention(Q (l) ,K (l) ,V (l) ); Residual connection: X′ (l) =X (l-1) +Z (l) ; Feedforward Network: Normalization: FFN calculation: Residual connection: X (l) =X′ (l) +Y (l) ; Output: After M layers of calculation, the encoded spatiotemporal features are expressed as: X (M) .

4. The video action recognition method based on spatiotemporal interactive Transformer and object interactivity prediction according to claim 3, characterized in that: The step 3) specifically includes: 3.1) Feature Integration: Fusion of video features, audio features and text features at time step t: F t =Concat(V t ,A t ,T t ); 3.2) Dynamic gating mechanism: Adjusting feature weights: Using the interactivity score I t Weight the fused features to highlight objects related to the action: F e =I t ×F t 。 5. The video action recognition method based on spatiotemporal interactive Transformer and object interactivity prediction according to claim 4, characterized in that: The step 4) is specifically as follows: 4.1) Temporal pooling and classification: Time pooling: After adjustment, the feature F e Pooling is performed in the time dimension to obtain the global feature vector: F f =TemporalPooling(F e ); Classification: The global feature vector F f Input classifier, output predicted probability of action category; 4.2) Model optimization: Loss function: The cross entropy loss function is used to evaluate the model performance: Among them, y i is the true label, is the predicted probability; Parameter update: Through the back-propagation algorithm, minimize the loss function L and update the model parameters.

6. A video action recognition system based on spatiotemporal interactive Transformer and object interactivity prediction, characterized by: The method for video action recognition based on spatiotemporal interactive Transformer and object interactivity prediction as described in claims 1 to 5 comprises the following modules: The multimodal feature extraction module uses CNN, VGGish and BERT models to extract feature vectors from video, audio and text to obtain corresponding modal features; The multi-scale visual Transformer module uses the multi-head pooling attention mechanism and Transformer network structure to process the fused multimodal features and capture complex spatiotemporal relationships. The object interactivity prediction module highlights the object features related to the action through feature fusion, interactivity score calculation and dynamic gating mechanism, and predicts the interactivity between different objects in the video. The action classifier uses a fully connected layer and a Softmax activation function to classify the extracted global features, output the action category, and obtain the action prediction probability. The global parameter adjustment module adjusts the model parameters to improve the recognition accuracy according to the classification results; calculates the cross entropy loss and uses the optimization algorithm to update the network weights.

Images