Video action recognition method based on multi-view spatio-temporal feature collaborative modeling

Abstract

A video action recognition method based on multi-view spatiotemporal feature collaborative modeling relates to the technical field of video recognition, and introduces a guide refining network composed of a guide token generation module and a query guide view refining module on the basis of multi-view spatiotemporal features extracted by a spatiotemporal encoder sharing parameters. And in combination with cross-view-angle consistency constraint, multi-view-angle attention fusion and a category prototype alignment mechanism, collaborative modeling and selective enhancement, alignment and aggregation of multi-view-angle video action spatial-temporal features are realized, and the accuracy and robustness of multi-view-angle video action recognition are improved.

Description

Technical Field

[0001] This invention relates to the field of video recognition technology, and specifically to a video action recognition method based on multi-view spatiotemporal feature collaborative modeling. Background Technology

[0002] In recent years, with the widespread adoption of video capture equipment and the development of deep learning, fields such as intelligent security, sports analysis, and rehabilitation assessment have generated massive amounts of video data, creating an urgent need for efficient and accurate video action recognition technology to effectively support applications such as security early warning, tactical analysis, and rehabilitation training. Currently, most video action recognition methods use RGB video captured by a single-view camera for modeling. Single-view methods can achieve good results in environments with relatively standard viewpoints and minimal occlusion. However, due to the fixed camera position, its field of view often cannot cover the entire spatial range of human movement. Once key body parts are affected by prolonged lighting and background interference, visual cues related to the action in the video become blurred or even missing, significantly reducing recognition accuracy and robustness.

[0003] To alleviate the limitations of single-viewpoint methods, multi-view video action recognition has emerged as a research direction in recent years. Compared to single-viewpoint methods, multi-view action recognition utilizes multiple cameras deployed in a scene to simultaneously capture human motion videos from different directions. Theoretically, this allows for more comprehensive acquisition of motion information and better addresses perspective changes and occlusion issues in real-world scenarios. However, while multi-viewpoint methods bring richer information, they also introduce more complex modeling challenges.

[0004] Existing multi-view action recognition methods can be broadly categorized as follows: One approach employs a paradigm of independent encoding and fusion of each viewpoint. Features are extracted from the video of each viewpoint, and then simply fused (e.g., stitching or averaging) at the feature or decision level. While structurally simple, this method is relatively coarse in its mining of complementary information from multiple views at the feature level, easily incorporating noise, background bias, and effective information from some views, weakening the overall discriminative power of the action representation. Another approach explicitly decouples viewpoint factors and action factors in the feature space to obtain viewpoint-invariant action representations. This approach mitigates the distributional differences between different views to some extent, better reflecting action correlations. However, overemphasizing viewpoint invariance can erase the viewpoint-specific details needed to distinguish similar actions, making it difficult for the model to fully utilize the complementary cues contained in each viewpoint, resulting in limited ability to distinguish fine-grained actions. A third approach is based on a contrastive learning framework, whose basic idea is to mine the correlations and differences between multi-view action representations by constructing reasonable positive and negative sample pairs. Such methods typically use synchronized segments of the same action instance under different perspectives as positive samples and different actions or different instances as negative samples. They also combine strategies such as hard example reweighting or perspective generation to improve robustness and generalization ability to perspective changes.

[0005] On the other hand, in real-world multi-camera deployment environments, the amount of information from each viewpoint often varies significantly. Some views are affected by factors such as long-term occlusion, while others can provide clearer action outlines and motion cues. Many existing multi-view fusion methods treat all views equally during the fusion stage, failing to adaptively distinguish between key and secondary views. They are easily dominated by noise from low-quality views, making it difficult to fully leverage the advantages of multi-view fusion, especially in small-sample scenarios with limited data.

[0006] In summary, existing RGB-based multi-view video action recognition technologies still have shortcomings in mining complementary information from multiple perspectives, suppressing view-related noise, and taking into account view-specific details. They are not yet able to stably obtain recognition results that combine accuracy and robustness under complex view changes and occlusion conditions. Summary of the Invention

[0007] In order to overcome the shortcomings of the above technologies, this invention provides a method to improve the accuracy of multi-view video action recognition.

[0008] The technical solution adopted by this invention to overcome its technical problems is: A video action recognition method based on multi-view spatiotemporal feature collaborative modeling includes: S1. Obtain from the dataset Each video segment and its corresponding annotation information will be... Each video clip is organized into a multi-view instance set according to the camera ID. , , For the first A multi-perspective example , For the number of multi-view instances, , For the first A multi-perspective example The Middle Video clips from various perspectives , ; S2. The first A multi-perspective example The Middle Video clips from various perspectives Preprocessing is performed to obtain preprocessed video clips. ; S3. Transfer the preprocessed video clips The input is fed into a spatiotemporal encoder with shared parameters, and the output is a video spatiotemporal feature sequence. ; S4. Construct a guide token generation module to generate video spatiotemporal feature sequences. The input is fed into the bootstrap token generation module, and the output is a bootstrap token sequence. ; S5. Construct a query guidance perspective refinement module to refine the video spatiotemporal feature sequence. With bootstrap token sequence The input is fed into the query guidance perspective refinement module, and the output is a refined feature representation. ; S6. Construct a multi-view fusion module to represent refined features. The input is fed into the multi-view fusion module, and the output is the aggregated representation of the current instance. ; S7. Aggregate representation The input is fed into the action classifier, and the output is the predicted score for each action category.

[0009] Furthermore, the dataset in step S1 is either the NTU RGB+D 60 dataset or the NTU RGB+D 120 dataset.

[0010] The annotation information corresponding to the video segment in step S1 includes configuration identifier, camera identifier, performer identifier, repetition count identifier, and action category identifier, which will be obtained from the dataset. Based on the annotation information, the video clips were captured using the same capture configuration, the same performer, the same number of repetitions, the same action category, and were filmed by different cameras. The video clip was inserted into the first... A multi-perspective example middle.

[0011] Furthermore, step S2 includes the following steps: S2-1. Use the linspace function from the NumPy library to start from the... A multi-perspective example The Middle Video clips from various perspectives A fixed-length video segment consisting of 16 frames sampled evenly over time is obtained. Each frame's pixels are divided by 255, and the pixels are then normalized to the [0,1] interval to obtain the pixel-normalized video segment. ; S2-2. Use the transforms.Normalize() function from the PyTorchVideo library to transform fixed-length video segments. Each frame of the image is pixel-normalized to obtain a normalized video clip consisting of 16 sampled frames. ; S2-3. Use the RandomShortSideScale() function in the PyTorchVideo library to standardize video clips. Perform random scaling to obtain scaled video clips. ; S2-4. Use the RandomCrop() function in the PyTorch library to crop the video clip. Perform a random cropping operation to obtain a video clip with a resolution of 224×224 pixels. ; S2-5. Use the RandomHorizontalFlip() function in the PyTorch library to flip video clips. Perform a random horizontal flip operation to obtain the preprocessed video clip. .

[0012] Furthermore, the aforementioned spatiotemporal encoder with shared parameters is constructed using the R3D-18 model.

[0013] Furthermore, step S4 includes the following steps: S4-1. The guide token generation module consists of a time feature mapping submodule and a location encoding submodule; S4-2. The temporal feature mapping submodule of the guide token generation module consists of a fully connected layer, a ReLU activation function, a layer normalization layer, and a Dropout layer, which sequentially generate the video spatiotemporal feature sequence. The input is fed into the time feature mapping submodule, and the output is the intermediate guiding feature sequence. ; S4-3. The location encoding submodule of the guide token generation module includes a parameter matrix. The parameter matrix The number of rows and the intermediate guiding feature sequence The number of frames is the same, and the number of columns is the same as the intermediate guiding feature sequence. The feature dimensions are the same, and the parameter matrix is... Each element in the sequence is a random number between 0 and 1, which will guide the intermediate feature sequence. Feature vectors and parameter matrices for each frame The time position vectors of each corresponding row are summed element-wise along the feature dimension to obtain the guide token sequence. .

[0014] Furthermore, step S5 includes the following steps: S5-1. The query guidance perspective refinement module consists of a first residual structure, a second residual structure, and a time mean pooling unit; S5-2. The first residual structure of the query guidance perspective refinement module consists of a multi-head attention mechanism and a layer normalization layer; S5-3. Guide token sequence As a query vector, the video spatiotemporal feature sequence The key and value vectors are input into the multi-head attention mechanism of the first residual structure, and the output is the attention output feature. ; S5-4. Outputting attention features With bootstrap token sequence After the addition operation along the feature dimension, the result is fed into the layer normalization layer of the first residual structure, and the output is the first intermediate feature sequence. ; S5-5. The second residual structure of the query-guided perspective refinement module consists of a linear transformation layer, a GELU activation function, a Dropout layer, a fully connected layer, and a layer normalization layer; S5-6. The first intermediate feature sequence The inputs are sequentially fed into the linear transformation layer, GELU activation function, Dropout layer, and fully connected layer of the second residual structure, and the output is the second intermediate feature sequence. ; S5-7. The first intermediate feature sequence With the second intermediate feature sequence After the addition operation along the feature dimension, the result is fed into the layer normalization layer of the second residual structure to obtain the first... Temporal refined feature sequences from multiple perspectives ; S5-8. The time-average pooling unit of the query guidance perspective refinement module consists of an average pooling layer, which refines the time-series feature sequences. The input is fed into the time-averaged pooling unit, and the output is the first... Refined feature representation from each perspective .

[0015] Furthermore, step S6 includes the following steps: S6-1. The multi-view fusion module consists of a linear transformation layer, a ReLU activation function, a Dropout layer, a fully connected layer, and a Softmax function in a feedforward neural network. S6-2. Representing refined features The inputs are sequentially fed into the linear transform layer, ReLU activation function, and Dropout layer of the feedforward neural network of the multi-view fusion module, and the output is the first... Intermediate perspective features of each viewpoint ; S6-3. Features of the intermediate view The inputs are sequentially fed into the fully connected layer and the Softmax function of the multi-view fusion module, and the output yields the attention weights for each viewpoint. ; S6-4. Through formula The aggregate representation of the current instance is calculated. In the formula, Weight for attention With refined feature representation Element-wise multiplication along the feature dimension.

[0016] Furthermore, in step S7, the action classifier is a fully connected layer.

[0017] Furthermore, it also includes through formulas The total loss was calculated. In the formula, , , All are weighted parameters. Loss for action classification, For cross-view consistency comparison loss, Using the prototype comparison loss, the total loss is optimized through a gradient descent algorithm. The parameters of the spatiotemporal encoder with shared parameters, the guide token generation module, the query guide perspective refinement module, the multi-view fusion module, and the action classifier are jointly optimized.

[0018] The beneficial effects of this invention are as follows: Based on the multi-view video spatiotemporal features extracted by a spatiotemporal encoder with shared parameters, a guided refinement network consisting of a guided token generation module and a query guided view refinement module is introduced. Combined with a multi-view fusion module and cross-view consistency contrast loss and category prototype contrast loss applied during the training phase, collaborative modeling, selective enhancement, alignment, and aggregation of multi-view video action spatiotemporal features are achieved. This scheme can fully utilize complementary information from multiple views under complex conditions such as viewpoint changes, occlusion, and illumination changes, significantly improving the accuracy and robustness of multi-view video action recognition, reducing dependence on a single high-quality viewpoint, and facilitating deployment and application in practical video surveillance and behavior analysis scenarios. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention. Figure 2 This is a diagram of the architecture of the present invention. Detailed Implementation

[0020] The following is in conjunction with the appendix Figure 1 Appendix Figure 2 The present invention will be further described below.

[0021] A video action recognition method based on multi-view spatiotemporal feature collaborative modeling includes: S1. Obtain from the dataset Each video segment and its corresponding annotation information will be... Each video clip is organized into a multi-view instance set according to the camera ID. , , For the first A multi-perspective example , For the number of multi-view instances, , For the first A multi-perspective example The Middle Video clips from various perspectives , .

[0022] S2. The first A multi-perspective example The Middle Video clips from various perspectives Preprocessing is performed to obtain preprocessed video clips. .

[0023] S3. Transfer the preprocessed video clips The input is fed into a spatiotemporal encoder with shared parameters, and the output is a video spatiotemporal feature sequence. A shared-parameter spatiotemporal encoder is used to perform joint feature extraction on the input video clips in both spatial and temporal dimensions.

[0024] S4. Construct a guide token generation module to generate video spatiotemporal feature sequences. The input is fed into the bootstrap token generation module, and the output is a bootstrap token sequence. The guide token generation module generates a guide token sequence based on the input video spatiotemporal feature sequence, providing guidance information for the subsequent query guidance perspective refinement module to refine spatiotemporal features at each perspective.

[0025] S5. Construct a query guidance perspective refinement module to refine the video spatiotemporal feature sequence. With bootstrap token sequence The input is fed into the query guidance perspective refinement module, and the output is a refined feature representation. .

[0026] S6. Construct a multi-view fusion module to represent refined features. The input is fed into the multi-view fusion module, and the output is the aggregated representation of the current instance. .

[0027] S7. Aggregate representation The input is fed into the action classifier, and the output is the predicted score for each action category.

[0028] This video action recognition method based on multi-view spatiotemporal feature collaborative modeling can improve overall recognition performance by efficiently integrating effective features from multiple perspectives and supplementing key information from missing perspectives while maintaining the essential semantics of action representation.

[0029] In one embodiment of the present invention, the dataset in step S1 is an NTU RGB+D 60 dataset or an NTU RGB+D 120 dataset.

[0030] In one embodiment of the present invention, the annotation information corresponding to the video segment in step S1 includes a configuration identifier, a camera identifier, a performer identifier, a repetition count identifier, and an action category identifier, which will be obtained from the dataset. Based on the annotation information, the video clips were captured using the same capture configuration, the same performer, the same number of repetitions, the same action category, and were filmed by different cameras. The video clip was inserted into the first... A multi-perspective example middle.

[0031] In one embodiment of the present invention, step S2 includes the following steps: S2-1. Use the linspace function from the NumPy library to start from the... A multi-perspective example The Middle Video clips from various perspectives A fixed-length video segment consisting of 16 frames sampled evenly over time is obtained. Each frame's pixels are divided by 255, and the pixels are then normalized to the [0,1] interval to obtain the pixel-normalized video segment. .

[0032] S2-2. Use the transforms.Normalize() function from the PyTorchVideo library to transform fixed-length video segments. Each frame of the image is pixel-normalized to obtain a normalized video clip consisting of 16 sampled frames. .

[0033] S2-3. Use the RandomShortSideScale() function in the PyTorchVideo library to standardize video clips. Perform random scaling to obtain scaled video clips. .

[0034] S2-4. Use the RandomCrop() function in the PyTorch library to crop the video clip. Perform a random cropping operation to obtain a video clip with a resolution of 224×224 pixels. .

[0035] S2-5. Use the RandomHorizontalFlip() function in the PyTorch library to flip video clips. Perform a random horizontal flip operation to obtain the preprocessed video clip. .

[0036] In one embodiment of the present invention, the spatiotemporal encoder with shared parameters described above is composed of an R3D-18 model.

[0037] In one embodiment of the present invention, step S4 includes the following steps: S4-1. The guide token generation module consists of a time feature mapping submodule and a location encoding submodule.

[0038] S4-2. The temporal feature mapping submodule of the guide token generation module consists of a fully connected layer, a ReLU activation function, a layer normalization layer, and a Dropout layer, which sequentially generate the video spatiotemporal feature sequence. The input is fed into the time feature mapping submodule, and the output is the intermediate guiding feature sequence. .

[0039] S4-3. The location encoding submodule of the guide token generation module includes a parameter matrix. The parameter matrix The number of rows and the intermediate guiding feature sequence The number of frames is the same, and the number of columns is the same as the intermediate guiding feature sequence. The feature dimensions are the same, and the parameter matrix is... Each element in the sequence is a random number between 0 and 1, which will guide the intermediate feature sequence. Feature vectors and parameter matrices for each frame The time position vectors of each corresponding row are summed element-wise along the feature dimension to obtain the guide token sequence. .

[0040] In one embodiment of the present invention, step S5 includes the following steps: S5-1. The query guidance perspective refinement module consists of a first residual structure, a second residual structure, and a time mean pooling unit.

[0041] S5-2. The first residual structure of the query guidance perspective refinement module consists of a multi-head attention mechanism and a layer normalization layer.

[0042] S5-3. Guide token sequence As a query vector, the video spatiotemporal feature sequence The key and value vectors are input into the multi-head attention mechanism of the first residual structure, and the output is the attention output feature. .

[0043] S5-4. Outputting attention features With bootstrap token sequence After the addition operation along the feature dimension, the result is fed into the layer normalization layer of the first residual structure, and the output is the first intermediate feature sequence. .

[0044] S5-5. The second residual structure of the query-guided perspective refinement module consists of a linear transformation layer, a GELU activation function, a Dropout layer, a fully connected layer, and a layer normalization layer.

[0045] S5-6. The first intermediate feature sequence The inputs are sequentially fed into the linear transformation layer, GELU activation function, Dropout layer, and fully connected layer of the second residual structure, and the output is the second intermediate feature sequence. .

[0046] S5-7. The first intermediate feature sequence With the second intermediate feature sequence After the addition operation along the feature dimension, the result is fed into the layer normalization layer of the second residual structure to obtain the first... Temporal refined feature sequences from multiple perspectives .

[0047] S5-8. The time-average pooling unit of the query guidance perspective refinement module consists of an average pooling layer, which refines the time-series feature sequences. The input is fed into the time-averaged pooling unit, and the output is the first... Refined feature representation from each perspective .

[0048] In one embodiment of the present invention, step S6 includes the following steps: S6-1. The multi-view fusion module consists of a linear transformation layer, a ReLU activation function, a Dropout layer, a fully connected layer, and a Softmax function in a feedforward neural network.

[0049] S6-2. Representing refined features The inputs are sequentially fed into the linear transform layer, ReLU activation function, and Dropout layer of the feedforward neural network of the multi-view fusion module, and the output is the first... Intermediate perspective features of each viewpoint .

[0050] S6-3. Features of the intermediate view The inputs are sequentially fed into the fully connected layer and the Softmax function of the multi-view fusion module, and the output yields the attention weights for each viewpoint. .

[0051] S6-4. Through formula The aggregate representation of the current instance is calculated. In the formula, Weight for attention With refined feature representation Element-wise multiplication along the feature dimension.

[0052] In one embodiment of the present invention, the action classifier in step S7 is a fully connected layer, which is used to map the instance-level aggregate representation to the action category space.

[0053] In one embodiment of the invention, it further includes using a formula The total loss was calculated. In the formula, , , All are weighted parameters. Loss for action classification, For cross-view consistency comparison loss, Using the prototype comparison loss, the total loss is optimized through a gradient descent algorithm. The parameters of the spatiotemporal encoder, guiding token generation module, query guidance perspective refinement module, multi-view fusion module, and action classifier with shared parameters are jointly optimized. By optimizing the query guidance perspective refinement module and applying cross-view consistency contrast constraints, the refined feature representations of the same multi-view instance under different perspectives can be brought closer together, while the refined feature representations of different action instances can be distanced, thereby achieving cross-view semantic consistency alignment and mitigating semantic shifts caused by perspective changes. By applying a category prototype contrast loss between the instance-level aggregated representation output by the multi-view fusion module and the learnable category prototype, and simultaneously updating the parameters of the multi-view fusion module and the category prototype, the distance between the instance aggregated representation and its own category prototype can be brought closer together, while the distance between it and other category prototypes can be distanced, thereby enhancing intra-class compactness and inter-class separability.

[0054] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video action recognition method based on multi-view spatio-temporal feature collaborative modeling, characterized in that, Comprise: S1. obtaining a plurality of video clips and corresponding label information of each video clip from a dataset, constructing a multi-view instance set according to the camera ID, for the i-th multi-view instance, wherein i is the number of multi-view instances, for the i-th multi-view instance, for the j-th view of the i-th multi-view instance, ;​​​​​​​​​ S2. pre-process the video segment in the first view of the first multi-view instance to obtain a pre-processed video segment ;​​​​ S3. The pre-processed video clip is input into the shared parameter spatio-temporal encoder, and the video spatio-temporal feature sequence is output ; S4. Constructing a guiding token generation module, inputting the video spatiotemporal feature sequence into the guiding token generation module, and outputting a guiding token sequence ; S5. Constructing a query-guided view angle refinement module, inputting the video spatiotemporal feature sequence and the guide token sequence into the query-guided view angle refinement module, and outputting a refined feature representation ; S6. Constructing a multi-view fusion module, and inputting the refined feature representation to the multi-view fusion module, and outputting an aggregated representation of the current instance ; S7. The aggregated representations are input into an action classifier, which outputs a predicted score for each action class. 2.The video action recognition method based on multi-view spatio-temporal feature collaborative modeling according to claim 1, characterized in that: The data set in step S1 is NTU RGB+D 60 dataset or NTU RGB+D 120 dataset. 3.The video action recognition method based on multi-view spatio-temporal feature collaborative modeling according to claim 2, characterized in that: The annotation information corresponding to the video clip in step S1 includes configuration identification, camera identification, performer identification, repetition number identification, and action category identification. The video clip is obtained from the data set According to the annotation information, the same collection configuration, the same performer, the same repetition number, the same action category, and different camera shooting video clips are placed in the first multi-view instance . 4.The method of claim 1, wherein, Step S2 comprises the following steps: S2-1. Use the linspace function in the NumPy library to generate 1000 evenly spaced values from the 0th to the 999th multiple view instances th view in the video clip th view in the video clip th view in the video clip, divide each pixel in the video clip by 255 to normalize the pixels to the interval [0, 1], and obtain the pixel-normalized video clip ; S2-2. Pixel normalization is performed on each frame of the fixed-length video clip using the transforms.Normalize() function in the PyTorchVideo library, resulting in a standardized video clip composed of 16 evenly sampled frames ;​ S2-3. Use the RandomShortSideScale() function in the PyTorchVideo library to scale the normalized video clip Perform a random scaling operation to obtain a scaled video clip ; S2-4. Use the RandomCrop() function in the PyTorch library to crop the scaled video segment A random crop operation is performed to obtain a video segment with a resolution of 224x224 pixels ; S2-5. Use the RandomHorizontalFlip() function in the PyTorch library to perform a random horizontal flip operation on the video segment to obtain the preprocessed video segment .

5. The video action recognition method based on multi-view spatio-temporal feature collaborative modeling according to claim 1, characterized in that: The spatio-temporal encoder of the shared parameter is composed of an R3D-18 model. 6.The method of claim 1, wherein, Step S4 comprises the following steps: S4-1. The guide token generation module is composed of a time feature mapping submodule and a position encoding submodule. S4-2. The time feature mapping submodule of the guidance token generation module is sequentially composed of a fully connected layer, a ReLU activation function, a layer normalization layer, and a Dropout layer. The video spatiotemporal feature sequence is input into the time feature mapping submodule, and an intermediate guidance feature sequence is output. ​ S4-3. The position encoding submodule of the guide token generation module includes a parameter matrix , the number of rows of the parameter matrix is the same as the number of frames of the intermediate guide feature sequence , the number of columns of the parameter matrix is the same as the feature dimension of the intermediate guide feature sequence , each element in the parameter matrix is a random number between 0 and 1, and the feature vector of each frame in the intermediate guide feature sequence is added element by element with the time position vector of each corresponding row in the parameter matrix in the feature dimension to obtain a guide token sequence 7.The method of claim 1, wherein, Step S5 comprises the following steps: S5-1. The query guide view angle refinement module is composed of a first residual structure, a second residual structure, and a time mean pooling unit. S5-2. The first residual structure of the query guide view angle refinement module is composed of a multi-head attention mechanism and a layer normalization layer. S5-3. The sequence of guide tokens is input into the first residual structure The video spatio-temporal feature sequence is input into the first residual structure as a key vector and a value vector The attention output feature is output by the multi-head attention mechanism of the first residual structure ; S5-4. Pay attention to the output characteristics With the guidance token sequence After the addition operation in the feature dimension, the layer normalization layer input to the first residual structure, and the output obtains the first intermediate feature sequence ; S5-5. The second residual structure of the query guide view angle refinement module is composed of a linear transformation layer, a GELU activation function, a Dropout layer, a fully connected layer, and a layer normalization layer. S5-6. The first intermediate feature sequence is sequentially input into a linear transformation layer, a GELU activation function, a Dropout layer, and a fully connected layer of the second residual structure, and a second intermediate feature sequence is output ; S5-7. The first intermediate feature sequence is added with the second intermediate feature sequence to obtain the time sequence refined feature sequence of the first view after the layer normalization layer input to the second residual structure ; S5-8. The time average pooling unit of the query-guided view angle refinement module is composed of an average pooling layer, which inputs the time refined feature sequence into the time average pooling unit and outputs the refined feature representation of the view angle . 8.The method of claim 1, wherein, Step S6 comprises the following steps: S6-1. The multi-view fusion module is composed of a linear transformation layer of a feedforward neural network, a ReLU activation function, a Dropout layer, a fully connected layer, and a Softmax function. S6-2. Refining the feature representation The linear transformation layer, the ReLU activation function and the Dropout layer of the feedforward neural network of the multi-view fusion module are sequentially input, and the intermediate view feature of the first view is output ;​ S6-3. The intermediate view features are sequentially input into a full connection layer and a Softmax function of the multi-view fusion module, and output to obtain the attention weight of each view ; S6-4. Compute the aggregate representation of the current instance by the formula S6-4. Compute the aggregate representation of the current instance by the formula S6-4. Compute the aggregate representation of the current instance by the formula S6-4. Compute the aggregate representation of the current instance by the formula S6-4. Compute the aggregate representation of the current instance by the formula S6-4. Compute the aggregate representation of the current instance by the formula 9.The video action recognition method based on multi-view spatio-temporal feature collaborative modeling of claim 1, characterized in that: The action classifier in step S7 is a fully connected layer. 10.The method of claim 1, wherein the method further comprises: determining a plurality of motion vectors of the plurality of video frames; and determining a plurality of motion vectors of the plurality of video frames. Further comprising calculating the total loss by formula , wherein, , , are weight parameters, is the action classification loss, is the cross-view consistency contrast loss, is the prototype contrast loss, and the total loss is used to jointly optimize the parameters of the spatio-temporal encoder, the guided token generation module, the query-guided view refinement module, the multi-view fusion module, and the action classifier by using a gradient descent optimization algorithm. ​

Images