A video action recognition method based on multi-view spatio-temporal feature collaborative modeling

Abstract

The application discloses a video action recognition method based on multi-view spatiotemporal feature collaborative modeling, relates to the technical field of video recognition, and introduces a guide refinement network composed of a guide token generation module and a query guide view refinement module on the basis of multi-view spatiotemporal features extracted by a shared-parameter spatiotemporal encoder, and combines cross-view consistency constraints, multi-view attention fusion and a class prototype alignment mechanism to realize collaborative modeling, selective enhancement, alignment and aggregation of multi-view video action spatiotemporal features, and improve the accuracy and robustness of multi-view video action recognition.

Description

Technical Field

[0001] This invention relates to the field of video recognition technology, and specifically to a video action recognition method based on multi-view spatiotemporal feature collaborative modeling. Background Technology

[0002] In recent years, with the widespread adoption of video capture equipment and the development of deep learning, fields such as intelligent security, sports analysis, and rehabilitation assessment have generated massive amounts of video data, creating an urgent need for efficient and accurate video action recognition technology to effectively support applications such as security early warning, tactical analysis, and rehabilitation training. Currently, most video action recognition methods use RGB video captured by a single-view camera for modeling. Single-view methods can achieve good results in environments with relatively standard viewpoints and minimal occlusion. However, due to the fixed camera position, its field of view often cannot cover the entire spatial range of human movement. Once key body parts are affected by prolonged lighting and background interference, visual cues related to the action in the video become blurred or even missing, significantly reducing recognition accuracy and robustness.

[0003] To alleviate the limitations of single-viewpoint methods, multi-view video action recognition has emerged as a research direction in recent years. Compared to single-viewpoint methods, multi-view action recognition utilizes multiple cameras deployed in a scene to simultaneously capture human motion videos from different directions. Theoretically, this allows for more comprehensive acquisition of motion information and better addresses perspective changes and occlusion issues in real-world scenarios. However, while multi-viewpoint methods bring richer information, they also introduce more complex modeling challenges.

[0004] Existing multi-view action recognition methods can be broadly categorized as follows: One approach employs a paradigm of independent encoding and fusion of each viewpoint. Features are extracted from the video of each viewpoint, and then simply fused (e.g., stitching or averaging) at the feature or decision level. While structurally simple, this method is relatively coarse in its mining of complementary information from multiple views at the feature level, easily incorporating noise, background bias, and effective information from some views, weakening the overall discriminative power of the action representation. Another approach explicitly decouples viewpoint factors and action factors in the feature space to obtain viewpoint-invariant action representations. This approach mitigates the distributional differences between different views to some extent, better reflecting action correlations. However, overemphasizing viewpoint invariance can erase the viewpoint-specific details needed to distinguish similar actions, making it difficult for the model to fully utilize the complementary cues contained in each viewpoint, resulting in limited ability to distinguish fine-grained actions. A third approach is based on a contrastive learning framework, whose basic idea is to mine the correlations and differences between multi-view action representations by constructing reasonable positive and negative sample pairs. Such methods typically use synchronized segments of the same action instance under different perspectives as positive samples and different actions or different instances as negative samples. They also combine strategies such as hard example reweighting or perspective generation to improve robustness and generalization ability to perspective changes.

[0005] On the other hand, in real-world multi-camera deployment environments, the amount of information from each viewpoint often varies significantly. Some views are affected by factors such as long-term occlusion, while others can provide clearer action outlines and motion cues. Many existing multi-view fusion methods treat all views equally during the fusion stage, failing to adaptively distinguish between key and secondary views. They are easily dominated by noise from low-quality views, making it difficult to fully leverage the advantages of multi-view fusion, especially in small-sample scenarios with limited data.

[0006] In summary, existing RGB-based multi-view video action recognition technologies still have shortcomings in mining complementary information from multiple perspectives, suppressing view-related noise, and taking into account view-specific details. They are not yet able to stably obtain recognition results that combine accuracy and robustness under complex view changes and occlusion conditions. Summary of the Invention

[0007] In order to overcome the shortcomings of the above technologies, this invention provides a method to improve the accuracy of multi-view video action recognition.

[0008] The technical solution adopted by this invention to overcome its technical problems is:

[0009] A video action recognition method based on multi-view spatiotemporal feature collaborative modeling includes:

[0010] S1. Obtain from the dataset Each video segment and its corresponding annotation information will be... Each video clip is organized into a multi-view instance set according to the camera ID. , , For the first A multi-perspective example , For the number of multi-view instances, , For the first A multi-perspective example The Middle Video clips from various perspectives , ;

[0011] S2. The first A multi-perspective example The Middle Video clips from various perspectives Preprocessing is performed to obtain preprocessed video clips. ;

[0012] S3. Transfer the preprocessed video clips The input is fed into a spatiotemporal encoder with shared parameters, and the output is a video spatiotemporal feature sequence. ;

[0013] S4. Construct a guide token generation module to generate video spatiotemporal feature sequences. The input is fed into the bootstrap token generation module, and the output is a bootstrap token sequence. ;

[0014] S5. Construct a query guidance perspective refinement module to refine the video spatiotemporal feature sequence. With bootstrap token sequence The input is fed into the query guidance perspective refinement module, and the output is a refined feature representation. ;

[0015] S6. Construct a multi-view fusion module to represent refined features. The input is fed into the multi-view fusion module, and the output is the aggregated representation of the current instance. ;

[0016] S7. Aggregate representation The input is fed into the action classifier, and the output is the predicted score for each action category.

[0017] Furthermore, the dataset in step S1 is either the NTU RGB+D 60 dataset or the NTU RGB+D 120 dataset.

[0018] The annotation information corresponding to the video segment in step S1 includes configuration identifier, camera identifier, performer identifier, repetition count identifier, and action category identifier, which will be obtained from the dataset. Based on the annotation information, the video clips were captured using the same capture configuration, the same performer, the same number of repetitions, the same action category, and were filmed by different cameras. The video clip was inserted into the first... A multi-perspective example middle.

[0019] Furthermore, step S2 includes the following steps:

[0020] S2-1. Use the linspace function from the NumPy library to start from the... A multi-perspective example The Middle Video clips from various perspectives A fixed-length video segment consisting of 16 frames sampled evenly over time is obtained. Each frame's pixels are divided by 255, and the pixels are then normalized to the [0,1] interval to obtain the pixel-normalized video segment. ;

[0021] S2-2. Use the transforms.Normalize() function from the PyTorchVideo library to transform fixed-length video segments. Each frame of the image is pixel-normalized to obtain a normalized video clip consisting of 16 sampled frames. ;

[0022] S2-3. Use the RandomShortSideScale() function in the PyTorchVideo library to standardize video clips. Perform random scaling to obtain scaled video clips. ;

[0023] S2-4. Use the RandomCrop() function in the PyTorch library to crop the video clip. Perform a random cropping operation to obtain a video clip with a resolution of 224×224 pixels. ;

[0024] S2-5. Use the RandomHorizontalFlip() function in the PyTorch library to flip video clips. Perform a random horizontal flip operation to obtain the preprocessed video clip. .

[0025] Furthermore, the aforementioned spatiotemporal encoder with shared parameters is constructed using the R3D-18 model.

[0026] Furthermore, step S4 includes the following steps:

[0027] S4-1. The guide token generation module consists of a time feature mapping submodule and a location encoding submodule;

[0028] S4-2. The temporal feature mapping submodule of the guide token generation module consists of a fully connected layer, a ReLU activation function, a layer normalization layer, and a Dropout layer, which sequentially generate the video spatiotemporal feature sequence. The input is fed into the time feature mapping submodule, and the output is the intermediate guiding feature sequence. ;

[0029] S4-3. The location encoding submodule of the guide token generation module includes a parameter matrix. The parameter matrix The number of rows and the intermediate guiding feature sequence The number of frames is the same, and the number of columns is the same as the intermediate guiding feature sequence. The feature dimensions are the same, and the parameter matrix is... Each element in the sequence is a random number between 0 and 1, which will guide the intermediate feature sequence. Feature vectors and parameter matrices for each frame The time position vectors of each corresponding row are summed element-wise along the feature dimension to obtain the guide token sequence. .

[0030] Furthermore, step S5 includes the following steps:

[0031] S5-1. The query guidance perspective refinement module consists of a first residual structure, a second residual structure, and a time mean pooling unit;

[0032] S5-2. The first residual structure of the query guidance perspective refinement module consists of a multi-head attention mechanism and a layer normalization layer;

[0033] S5-3. Guide token sequence As a query vector, the video spatiotemporal feature sequence The key and value vectors are input into the multi-head attention mechanism of the first residual structure, and the output is the attention output feature. ;

[0034] S5-4. Outputting attention features With bootstrap token sequence After the addition operation along the feature dimension, the result is fed into the layer normalization layer of the first residual structure, and the output is the first intermediate feature sequence. ;

[0035] S5-5. The second residual structure of the query-guided perspective refinement module consists of a linear transformation layer, a GELU activation function, a Dropout layer, a fully connected layer, and a layer normalization layer;

[0036] S5-6. The first intermediate feature sequence The inputs are sequentially fed into the linear transformation layer, GELU activation function, Dropout layer, and fully connected layer of the second residual structure, and the output is the second intermediate feature sequence. ;

[0037] S5-7. The first intermediate feature sequence With the second intermediate feature sequence After the addition operation along the feature dimension, the result is fed into the layer normalization layer of the second residual structure to obtain the first... Temporal refined feature sequences from multiple perspectives ;

[0038] S5-8. The time-average pooling unit of the query guidance perspective refinement module consists of an average pooling layer, which refines the time-series feature sequences. The input is fed into the time-averaged pooling unit, and the output is the first... Refined feature representation from each perspective .

[0039] Furthermore, step S6 includes the following steps:

[0040] S6-1. The multi-view fusion module consists of a linear transformation layer, a ReLU activation function, a Dropout layer, a fully connected layer, and a Softmax function in a feedforward neural network.

[0041] S6-2. Representing refined features The inputs are sequentially fed into the linear transform layer, ReLU activation function, and Dropout layer of the feedforward neural network of the multi-view fusion module, and the output is the first... Intermediate perspective features of each viewpoint ;

[0042] S6-3. Features of the intermediate view The inputs are sequentially fed into the fully connected layer and the Softmax function of the multi-view fusion module, and the output yields the attention weights for each viewpoint. ;

[0043] S6-4. Through formula The aggregate representation of the current instance is calculated. In the formula, Weight for attention With refined feature representation Element-wise multiplication along the feature dimension.

[0044] Furthermore, in step S7, the action classifier is a fully connected layer.

[0045] Furthermore, it also includes through formulas The total loss was calculated. In the formula, , , All are weighted parameters. Loss for action classification, For cross-view consistency comparison loss, Using the prototype comparison loss, the total loss is optimized through a gradient descent algorithm. The parameters of the spatiotemporal encoder with shared parameters, the guide token generation module, the query guide perspective refinement module, the multi-view fusion module, and the action classifier are jointly optimized.

[0046] The beneficial effects of this invention are as follows: Based on the multi-view video spatiotemporal features extracted by a spatiotemporal encoder with shared parameters, a guided refinement network consisting of a guided token generation module and a query guided view refinement module is introduced. Combined with a multi-view fusion module and cross-view consistency contrast loss and category prototype contrast loss applied during the training phase, collaborative modeling, selective enhancement, alignment, and aggregation of multi-view video action spatiotemporal features are achieved. This scheme can fully utilize complementary information from multiple views under complex conditions such as viewpoint changes, occlusion, and illumination changes, significantly improving the accuracy and robustness of multi-view video action recognition, reducing dependence on a single high-quality viewpoint, and facilitating deployment and application in practical video surveillance and behavior analysis scenarios. Attached Figure Description

[0047] Figure 1 This is a flowchart of the method of the present invention.

[0048] Figure 2 This is a diagram of the architecture of the present invention. Detailed Implementation

[0049] The following is in conjunction with the appendix Figure 1 Appendix Figure 2 The present invention will be further described below.

[0050] A video action recognition method based on multi-view spatiotemporal feature collaborative modeling includes:

[0051] S1. Obtain from the dataset Each video segment and its corresponding annotation information will be... Each video clip is organized into a multi-view instance set according to the camera ID. , , For the first A multi-perspective example , For the number of multi-view instances, , For the first A multi-perspective example The Middle Video clips from various perspectives , .

[0052] S2. The first A multi-perspective example The Middle Video clips from various perspectives Preprocessing is performed to obtain preprocessed video clips. .

[0053] S3. Transfer the preprocessed video clips The input is fed into a spatiotemporal encoder with shared parameters, and the output is a video spatiotemporal feature sequence. A shared-parameter spatiotemporal encoder is used to perform joint feature extraction on the input video clips in both spatial and temporal dimensions.

[0054] S4. Construct a guide token generation module to generate video spatiotemporal feature sequences. The input is fed into the bootstrap token generation module, and the output is a bootstrap token sequence. The guide token generation module generates a guide token sequence based on the input video spatiotemporal feature sequence, providing guidance information for the subsequent query guide perspective refinement module to refine spatiotemporal features at each perspective.

[0055] S5. Construct a query guidance perspective refinement module to refine the video spatiotemporal feature sequence. With bootstrap token sequence The input is fed into the query guidance perspective refinement module, and the output is a refined feature representation. .

[0056] S6. Construct a multi-view fusion module to represent refined features. The input is fed into the multi-view fusion module, and the output is the aggregated representation of the current instance. .

[0057] S7. Aggregate representation The input is fed into the action classifier, and the output is the predicted score for each action category.

[0058] This video action recognition method based on multi-view spatiotemporal feature collaborative modeling can improve overall recognition performance by efficiently integrating effective features from multiple perspectives and supplementing key information from missing perspectives while maintaining the essential semantics of action representation.

[0059] In one embodiment of the present invention, the dataset in step S1 is an NTU RGB+D 60 dataset or an NTU RGB+D 120 dataset.

[0060] In one embodiment of the present invention, the annotation information corresponding to the video segment in step S1 includes a configuration identifier, a camera identifier, a performer identifier, a repetition count identifier, and an action category identifier, which will be obtained from the dataset. Based on the annotation information, the video clips were captured using the same capture configuration, the same performer, the same number of repetitions, the same action category, and were filmed by different cameras. The video clip was inserted into the first... A multi-perspective example middle.

[0061] In one embodiment of the present invention, step S2 includes the following steps:

[0062] S2-1. Use the linspace function from the NumPy library to start from the... A multi-perspective example The Middle Video clips from various perspectives A fixed-length video segment consisting of 16 frames sampled evenly over time is obtained. Each frame's pixels are divided by 255, and the pixels are then normalized to the [0,1] interval to obtain the pixel-normalized video segment. .

[0063] S2-2. Use the transforms.Normalize() function from the PyTorchVideo library to transform fixed-length video segments. Each frame of the image is pixel-normalized to obtain a normalized video clip consisting of 16 sampled frames. .

[0064] S2-3. Use the RandomShortSideScale() function in the PyTorchVideo library to standardize video clips. Perform random scaling to obtain scaled video clips. .

[0065] S2-4. Use the RandomCrop() function in the PyTorch library to crop the video clip. Perform a random cropping operation to obtain a video clip with a resolution of 224×224 pixels. .

[0066] S2-5. Use the RandomHorizontalFlip() function in the PyTorch library to flip video clips. Perform a random horizontal flip operation to obtain the preprocessed video clip. .

[0067] In one embodiment of the present invention, the spatiotemporal encoder with shared parameters described above is composed of an R3D-18 model.

[0068] In one embodiment of the present invention, step S4 includes the following steps:

[0069] S4-1. The guide token generation module consists of a time feature mapping submodule and a location encoding submodule.

[0070] S4-2. The temporal feature mapping submodule of the guide token generation module consists of a fully connected layer, a ReLU activation function, a layer normalization layer, and a Dropout layer, which sequentially generate the video spatiotemporal feature sequence. The input is fed into the time feature mapping submodule, and the output is the intermediate guiding feature sequence. .

[0071] S4-3. The location encoding submodule of the guide token generation module includes a parameter matrix. The parameter matrix The number of rows and the intermediate guiding feature sequence The number of frames is the same, and the number of columns is the same as the intermediate guiding feature sequence. The feature dimensions are the same, and the parameter matrix is... Each element in the sequence is a random number between 0 and 1, which will guide the intermediate feature sequence. Feature vectors and parameter matrices for each frame The time position vectors of each corresponding row are summed element-wise along the feature dimension to obtain the guide token sequence. .

[0072] In one embodiment of the present invention, step S5 includes the following steps:

[0073] S5-1. The query guidance perspective refinement module consists of a first residual structure, a second residual structure, and a time mean pooling unit.

[0074] S5-2. The first residual structure of the query guidance perspective refinement module consists of a multi-head attention mechanism and a layer normalization layer.

[0075] S5-3. Guide token sequence As a query vector, the video spatiotemporal feature sequence The key and value vectors are input into the multi-head attention mechanism of the first residual structure, and the output is the attention output feature. .

[0076] S5-4. Outputting attention features With bootstrap token sequence After the addition operation along the feature dimension, the result is fed into the layer normalization layer of the first residual structure, and the output is the first intermediate feature sequence. .

[0077] S5-5. The second residual structure of the query-guided perspective refinement module consists of a linear transformation layer, a GELU activation function, a Dropout layer, a fully connected layer, and a layer normalization layer.

[0078] S5-6. The first intermediate feature sequence The inputs are sequentially fed into the linear transformation layer, GELU activation function, Dropout layer, and fully connected layer of the second residual structure, and the output is the second intermediate feature sequence. .

[0079] S5-7. The first intermediate feature sequence With the second intermediate feature sequence After the addition operation along the feature dimension, the result is fed into the layer normalization layer of the second residual structure to obtain the first... Temporal refined feature sequences from multiple perspectives .

[0080] S5-8. The time-average pooling unit of the query guidance perspective refinement module consists of an average pooling layer, which refines the time-series feature sequences. The input is fed into the time-averaged pooling unit, and the output is the first... Refined feature representation from each perspective .

[0081] In one embodiment of the present invention, step S6 includes the following steps:

[0082] S6-1. The multi-view fusion module consists of a linear transformation layer, a ReLU activation function, a Dropout layer, a fully connected layer, and a Softmax function in a feedforward neural network.

[0083] S6-2. Representing refined features The inputs are sequentially fed into the linear transform layer, ReLU activation function, and Dropout layer of the feedforward neural network of the multi-view fusion module, and the output is the first... Intermediate perspective features of each viewpoint .

[0084] S6-3. Features of the intermediate view The inputs are sequentially fed into the fully connected layer and the Softmax function of the multi-view fusion module, and the output yields the attention weights for each viewpoint. .

[0085] S6-4. Through formula The aggregate representation of the current instance is calculated. In the formula, Weight for attention With refined feature representation Element-wise multiplication along the feature dimension.

[0086] In one embodiment of the present invention, the action classifier in step S7 is a fully connected layer, which is used to map the instance-level aggregate representation to the action category space.

[0087] In one embodiment of the invention, it further includes using a formula The total loss was calculated. In the formula, , , All are weighted parameters. Loss for action classification, For cross-view consistency comparison loss, Using the prototype comparison loss, the total loss is optimized through a gradient descent algorithm. The parameters of the spatiotemporal encoder, guiding token generation module, query guidance perspective refinement module, multi-view fusion module, and action classifier with shared parameters are jointly optimized. By optimizing the query guidance perspective refinement module and applying cross-view consistency contrast constraints, the refined feature representations of the same multi-view instance under different perspectives can be brought closer together, while the refined feature representations of different action instances can be distanced, thereby achieving cross-view semantic consistency alignment and mitigating semantic shifts caused by perspective changes. By applying a category prototype contrast loss between the instance-level aggregated representation output by the multi-view fusion module and the learnable category prototype, and simultaneously updating the parameters of the multi-view fusion module and the category prototype, the distance between the instance aggregated representation and its own category prototype can be brought closer together, while the distance between it and other category prototypes can be distanced, thereby enhancing intra-class compactness and inter-class separability.

[0088] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video action recognition method based on multi-view spatiotemporal feature collaborative modeling, characterized in that, include: S1. Obtain from the dataset Each video segment and its corresponding annotation information will be... Each video clip is organized into a multi-view instance set according to the camera ID. , , For the first A multi-perspective example , For the number of multi-view instances, , For the first A multi-perspective example The Middle Video clips from various perspectives , ; S2. The first A multi-perspective example The Middle Video clips from various perspectives Preprocessing is performed to obtain preprocessed video clips. ; S3. Transfer the preprocessed video clips The input is fed into a spatiotemporal encoder with shared parameters, and the output is a video spatiotemporal feature sequence. ; S4. Construct a guide token generation module to generate video spatiotemporal feature sequences. The input is fed into the bootstrap token generation module, and the output is a bootstrap token sequence. ; S5. Construct a query guidance perspective refinement module to refine the video spatiotemporal feature sequence. With bootstrap token sequence The input is fed into the query guidance perspective refinement module, and the output is a refined feature representation. ; S6. Construct a multi-view fusion module to represent refined features. The input is fed into the multi-view fusion module, and the output is the aggregated representation of the current instance. ; S7. Aggregate representation The input is fed into the action classifier, and the output is the predicted score for each action category.

2. The video action recognition method based on multi-view spatiotemporal feature collaborative modeling according to claim 1, characterized in that: The dataset in step S1 is either the NTU RGB+D 60 dataset or the NTU RGB+D 120 dataset.

3. The video action recognition method based on multi-view spatiotemporal feature collaborative modeling according to claim 2, characterized in that: The annotation information corresponding to the video segment in step S1 includes configuration identifier, camera identifier, performer identifier, repetition count identifier, and action category identifier, which will be obtained from the dataset. Based on the annotation information, the video clips were captured using the same capture configuration, the same performer, the same number of repetitions, the same action category, and were filmed by different cameras. The video clip was inserted into the first... A multi-perspective example middle.

4. The video action recognition method based on multi-view spatiotemporal feature collaborative modeling according to claim 1, characterized in that, Step S2 includes the following steps: S2-1. Use the linspace function from the NumPy library to start from the... A multi-perspective example The Middle Video clips from various perspectives A fixed-length video segment consisting of 16 frames sampled evenly over time is obtained. Each frame's pixels are divided by 255, and the pixels are then normalized to the [0,1] interval to obtain the pixel-normalized video segment. ; S2-2. Use the transforms.Normalize() function from the PyTorchVideo library to transform fixed-length video segments. Each frame of the image is pixel-normalized to obtain a normalized video clip consisting of 16 sampled frames. ; S2-3. Use the RandomShortSideScale() function in the PyTorchVideo library to standardize video clips. Perform random scaling to obtain scaled video clips. ; S2-4. Use the RandomCrop() function in the PyTorch library to crop the video clip. Perform a random cropping operation to obtain a video clip with a resolution of 224×224 pixels. ; S2-5. Use the RandomHorizontalFlip() function from the PyTorch library to flip video clips. Perform a random horizontal flip operation to obtain the preprocessed video clip. .

5. The video action recognition method based on multi-view spatiotemporal feature collaborative modeling according to claim 1, characterized in that: The shared parameter spatiotemporal encoder is composed of an R3D-18 model.

6. The video action recognition method based on multi-view spatiotemporal feature collaborative modeling according to claim 1, characterized in that, Step S4 includes the following steps: S4-1. The guide token generation module consists of a time feature mapping submodule and a location encoding submodule; S4-2. The temporal feature mapping submodule of the guide token generation module consists of a fully connected layer, a ReLU activation function, a layer normalization layer, and a Dropout layer, which sequentially generate the video spatiotemporal feature sequence. The input is fed into the time feature mapping submodule, and the output is the intermediate guiding feature sequence. ; S4-3. The location encoding submodule of the guide token generation module includes a parameter matrix. The parameter matrix The number of rows and the intermediate guiding feature sequence The number of frames is the same, and the number of columns is the same as the intermediate guiding feature sequence. The feature dimensions are the same, and the parameter matrix is... Each element in the sequence is a random number between 0 and 1, which will guide the intermediate feature sequence. Feature vectors and parameter matrices for each frame The time position vectors of each corresponding row are summed element-wise along the feature dimension to obtain the guide token sequence. .

7. The video action recognition method based on multi-view spatiotemporal feature collaborative modeling according to claim 1, characterized in that, Step S5 includes the following steps: S5-1. The query guidance perspective refinement module consists of a first residual structure, a second residual structure, and a time mean pooling unit; S5-2. The first residual structure of the query guidance perspective refinement module consists of a multi-head attention mechanism and a layer normalization layer; S5-3. Guide token sequence As a query vector, the video spatiotemporal feature sequence The key and value vectors are input into the multi-head attention mechanism of the first residual structure, and the output is the attention output feature. ; S5-4. Outputting attention features With bootstrap token sequence After the addition operation along the feature dimension, the result is fed into the layer normalization layer of the first residual structure, and the output is the first intermediate feature sequence. ; S5-5. The second residual structure of the query-guided perspective refinement module consists of a linear transformation layer, a GELU activation function, a Dropout layer, a fully connected layer, and a layer normalization layer; S5-6. The first intermediate feature sequence The inputs are sequentially fed into the linear transformation layer, GELU activation function, Dropout layer, and fully connected layer of the second residual structure, and the output is the second intermediate feature sequence. ; S5-7. The first intermediate feature sequence With the second intermediate feature sequence After the addition operation along the feature dimension, the result is fed into the layer normalization layer of the second residual structure to obtain the first... Temporal refined feature sequences from multiple perspectives ; S5-8. The time-average pooling unit of the query guidance perspective refinement module consists of an average pooling layer, which refines the time-series feature sequences. The input is fed into the time-averaged pooling unit, and the output is the first... Refined feature representation from each perspective .

8. The video action recognition method based on multi-view spatiotemporal feature collaborative modeling according to claim 1, characterized in that, Step S6 includes the following steps: S6-1. The multi-view fusion module consists of a linear transformation layer, a ReLU activation function, a Dropout layer, a fully connected layer, and a Softmax function in a feedforward neural network. S6-2. Representing refined features The inputs are sequentially fed into the linear transform layer, ReLU activation function, and Dropout layer of the feedforward neural network of the multi-view fusion module, and the output is the first... Intermediate perspective features of each viewpoint ; S6-3. Features of the intermediate view The inputs are sequentially fed into the fully connected layer and the Softmax function of the multi-view fusion module, and the output yields the attention weights for each viewpoint. ; S6-4. Through formula The aggregate representation of the current instance is calculated. In the formula, Weight for attention With refined feature representation Element-wise multiplication along the feature dimension.

9. The video action recognition method based on multi-view spatiotemporal feature collaborative modeling according to claim 1, characterized in that: In step S7, the action classifier is a fully connected layer.

10. The video action recognition method based on multi-view spatiotemporal feature collaborative modeling according to claim 1, characterized in that: It also includes through formulas The total loss was calculated. In the formula, , , All are weighted parameters. Loss for action classification, For cross-view consistency comparison loss, Using the prototype comparison loss, the total loss is optimized through a gradient descent algorithm. The parameters of the spatiotemporal encoder with shared parameters, the guide token generation module, the query guide perspective refinement module, the multi-view fusion module, and the action classifier are jointly optimized.

Images