Deep video behavior recognition method and system based on multi-view feature interaction fusion

Abstract

The application provides a deep video behavior recognition method and system based on multi-view feature interaction and fusion, relates to the field of privacy calculation, and utilizes depth projection and order pooling technology to construct a multi-view depth dynamic roll from an original depth video; extracts a space-time feature of the multi-view depth dynamic roll to obtain a multi-view feature map; learns long-distance space-time correlation inside the multi-view feature map to generate a space-time aggregation feature representation of each individual view; interacts and fuses the space-time aggregation feature representations between different views to generate a global multi-view feature representation of a video behavior; and inputs the global multi-view feature representation into a full connection layer to complete behavior recognition; the application uses 3D CNN feature maps of different views as inputs of a multi-view feature interaction and fusion framework based on a Transformer to learn feature correlation inside the views and between the views, so that a global feature capable of representing a 3D motion mode of a human body behavior is generated, and the performance of deep video behavior recognition is improved.

Description

Technical Field

[0001] This invention belongs to the field of privacy computing, and in particular relates to a method and system for deep video behavior recognition based on multi-view feature interaction fusion. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Video-based human behavior recognition is a popular and valuable research topic in the field of computer vision, due to its wide range of practical applications, such as human-computer interaction, intelligent video surveillance, and autonomous driving. Recently, with the emergence of low-cost and reliable depth sensors (such as Microsoft Kinect), behavior recognition based on depth video has attracted increasing interest. From traditional methods based on handcrafted features to today's deep learning methods, depth data is less sensitive to color and lighting changes compared to RGB video, and can provide rich and advanced visual cues closely related to human behavior, such as body contours and 3D scene structure information. In addition, depth data modalities can effectively protect human privacy, which is very important in some real-world applications with high confidentiality requirements.

[0004] Spatiotemporal feature extraction is the most critical and core problem in video action recognition tasks. In the past few years, deep learning has automatically learned discriminative spatiotemporal features in videos in an end-to-end manner, greatly improving recognition performance and achieving remarkable success in the field of video action recognition. Due to the ease of obtaining RGB action videos from the Internet, data-driven deep learning methods initially focused on action recognition tasks in RGB videos. Later, the release of large-scale RGBD benchmark datasets (NTU RGB+D60&120 datasets) greatly promoted the development of deep learning-based deep video action recognition. Most deep learning-based video action recognition methods rely on convolutional neural networks (CNNs), where convolution operations can extract multi-level local feature information from video sequences. From the perspective of spatiotemporal feature modeling, commonly used typical deep networks can usually be divided into 2D CNNs and 3D CNNs. 2D CNNs learn spatial features from a single video frame and cannot directly model the temporal structure of the video, so additional temporal information needs to be captured to achieve effective video action recognition, such as optical flow or LSTM networks. 3D CNNs, by introducing 1D temporal convolution, can simultaneously model spatial and temporal features, overcoming the limitations of 2D CNNs. Despite the weaknesses of CNNs, they have achieved superior recognition performance. Although convolutional neural networks have made great progress in video action recognition tasks, they have difficulty modeling long-distance or global spatiotemporal information, which limits their ability to further improve action recognition performance.

[0005] Recently, the Transformer has attracted considerable attention due to its significant success in natural language processing. Because of its remarkable advantage in capturing long-range dependencies in data, researchers have extended it to various computer vision tasks, such as image classification, object detection, and video action recognition. For action recognition tasks, most existing methods use 2D convolutional features of image patches, image sequences, or skeleton data as input labels to the Transformer to model long-term spatiotemporal contextual information. However, these existing Transformer methods are specific to RGB and skeleton data, and there is a lack of research on deep data. Furthermore, in existing CNN+Transformer hybrid networks, 2D CNN feature maps are typically used as input to the Transformer to model global information, but 3D CNN convolutional features are rarely used. Moreover, existing Transformer-based action recognition methods typically model long-range information from both spatial and temporal dimensions, and few works focus on exploring the Transformer structure's ability to handle the interaction and fusion of features from different perspectives.

[0006] Therefore, existing Transformer-based methods only use RGB video data from a single viewpoint, which limits the generalization performance of the Transformer model. How to extend the Transformer architecture to deep video data and perform deep video behavior recognition with multi-view feature learning is a topic worthy of research. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, this invention provides a deep video behavior recognition method and system based on multi-view feature interaction fusion. It uses 3D CNN feature maps from different viewpoints as input to the Transformer to learn the feature correlations within and between viewpoints, thereby generating global features that can characterize the 3D motion patterns of human behavior, thus improving the performance of deep video behavior recognition.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] The first aspect of this invention provides a deep video behavior recognition method based on multi-view feature interaction fusion;

[0010] Deep video behavior recognition methods based on multi-view feature interaction fusion include:

[0011] Multi-view depth dynamic volumes are constructed from raw depth videos using depth projection and ordered pooling techniques.

[0012] Spatiotemporal feature extraction is performed on the multi-view deep dynamic volume to obtain multi-view feature maps;

[0013] Based on multi-view feature maps, long-distance spatiotemporal correlations in the feature maps of each individual view are learned to generate spatiotemporal aggregated feature representations for each view.

[0014] Interacting and fusing spatiotemporal aggregated feature representations from different perspectives to generate a global multi-perspective feature representation of video behavior;

[0015] The global multi-view feature representation is input into the fully connected layer to complete behavior recognition.

[0016] Furthermore, the construction of the multi-view depth dynamic volume specifically involves:

[0017] The original depth video is projected onto three orthogonal planes to generate a sequence of depth projection maps from three different viewpoints.

[0018] Sequential pooling is performed on the depth projection map sequences from three different perspectives to obtain a multi-view dynamic depth volume.

[0019] Furthermore, the sequential pooling process is performed on the depth projection map sequences from the three viewpoints, specifically as follows:

[0020] A sparse sampling strategy is used to uniformly divide the depth projection map sequence under a single viewpoint into multiple non-overlapping short segments.

[0021] Sequential pooling is applied to aggregate short-term spatiotemporal information from each short segment to generate the corresponding dynamic image;

[0022] All the short, moving images together form a deep dynamic roll from a single perspective;

[0023] A multi-view depth dynamic volume is formed by depth dynamic volumes from all perspectives.

[0024] Furthermore, a 3D CNN model is used to extract spatiotemporal features from the multi-view deep dynamic volume.

[0025] Furthermore, based on multi-view feature maps, the long-range spatiotemporal correlations in the feature maps of each individual view are learned, generating a spatiotemporal aggregated feature representation for each view, specifically:

[0026] Dimensional processing is performed on the multi-view feature maps to obtain the input sequence of the in-view self-attention encoder module;

[0027] The in-view self-attention encoder module consists of a multi-layer Transformer encoder architecture. It learns the spatiotemporal dependencies in the feature maps of each view through a self-attention mechanism, and obtains a spatiotemporal aggregated feature representation.

[0028] Furthermore, the self-attention mechanism is extended to a mutual attention mechanism through the cross-view feature fusion module, enabling interaction and fusion between spatiotemporal aggregated feature representations from different perspectives, thereby generating a global multi-view feature representation of video behavior.

[0029] Furthermore, the cross-view feature fusion module includes cross-view feature interaction and multi-view feature fusion;

[0030] The cross-view feature interaction is performed between every two views, multi-head mutual attention between views is calculated, and then the interaction feature sequence between any two views is obtained through a multilayer perceptron.

[0031] The multi-view feature fusion involves averaging the interaction feature sequences of any two views to obtain a cross-view feature representation. The cross-view feature representations of every two views are then connected to fuse multi-view action information, resulting in the final global multi-view feature representation.

[0032] The second aspect of the present invention provides a deep video behavior recognition system based on multi-view feature interaction fusion.

[0033] The deep video behavior recognition system based on multi-view feature interaction fusion includes a multi-view deep dynamic convolution construction module, a convolutional feature extraction module, an intra-view self-attention encoder module, a cross-view feature fusion module, and a behavior recognition module.

[0034] The multi-view depth dynamic volume construction module is configured to construct a multi-view depth dynamic volume from the original depth video using depth projection and ordered pooling techniques.

[0035] The convolutional feature extraction module is configured to perform spatiotemporal feature extraction on multi-view depth dynamic convolutions to obtain multi-view feature maps.

[0036] The in-view self-attention encoder module is configured to: learn the long-distance spatiotemporal correlation in the feature maps of each individual view based on the multi-view feature maps, and generate a spatiotemporal aggregated feature representation for each view.

[0037] The cross-perspective feature fusion module is configured to interact and fuse spatiotemporal aggregated feature representations from different perspectives to generate a global multi-perspective feature representation of video behavior.

[0038] The behavior recognition module is configured to input global multi-view feature representations into the fully connected layer to complete behavior recognition.

[0039] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the deep video behavior recognition method based on multi-view feature interaction fusion as described in the first aspect of the present invention.

[0040] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the deep video behavior recognition method based on multi-view feature interaction fusion as described in the first aspect of the present invention.

[0041] The above one or more technical solutions have the following beneficial effects:

[0042] This invention proposes a multi-view depth dynamic roll to express depth video behavior, which can effectively describe the three-dimensional spatiotemporal motion pattern of human behavior from different angles.

[0043] This invention extends the Transformer encoder architecture to deep data modalities and proposes an algorithm framework based on Transformer multi-view feature interaction fusion. The proposed method models the feature correlation within the viewpoint and the feature interaction between the viewpoints on the 3DCNN feature maps of three views to obtain a more discriminative and richer comprehensive global video feature representation.

[0044] Extensive experiments conducted on two challenging large-scale RGBD behavior datasets demonstrate that the proposed behavior recognition method based on Transformer multi-view feature interaction fusion fully learns complementary feature information from different viewpoints, achieving superior performance in deep video behavior recognition.

[0045] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0046] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0047] Figure 1 This is the first embodiment of a deep video behavior recognition framework based on Transformer multi-view feature interaction fusion.

[0048] Figure 2 This is a schematic diagram of a multi-view depth dynamic volume with two types of behavior in the NTU RGB+D 60 dataset of the first embodiment.

[0049] Figure 3 This is a flowchart of the ISEM process for the first embodiment.

[0050] Figure 4 This is a flowchart of the CFFM process for the first embodiment.

[0051] Figure 5 This is a system structure diagram of the second embodiment. Detailed Implementation

[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0053] Example 1

[0054] The Transformer architecture has attracted increasing attention and achieved remarkable success in the field of video action recognition in recent years. However, almost all existing Transformer-based methods only use RGB video data from a single viewpoint, which limits the generalization performance of Transformer models. Depth video action recognition is an important research direction because depth data is not only invariant to changes in lighting and color, but also provides reliable three-dimensional geometric information of body contours.

[0055] This embodiment extends the Transformer architecture to deep video behavior recognition and proposes a Transformer-based multi-view deep feature interaction fusion framework, which mainly consists of two parts: an Intra-view Self-attention Encoder Module (ISEM) and a Cross-view Feature Fusion Module (CFFM). Specifically, the original deep video is first projected onto three orthogonal planes, and sequential pooling is used to construct a multi-view deep dynamic convolution to describe the 3D spatiotemporal evolution of human behavior. Then, the multi-view deep dynamic convolution is input into a 3D CNN for spatiotemporal feature modeling. Based on the deep convolutional feature maps of the three views, ISEM learns the long-distance spatiotemporal correlation in each view, while CFFM performs cross-view feature interaction and integrates the cross-view features to obtain a global multi-view feature representation of video behavior, which is finally used for behavior recognition.

[0056] This embodiment discloses a deep video behavior recognition method based on multi-view feature interaction fusion, such as... Figure 1 As shown, it includes:

[0057] Step S1: Using depth projection and sequential pooling techniques, construct a multi-view depth dynamic volume from the original depth video to describe the 3D spatiotemporal motion dynamics of human body movements from different angles.

[0058] Human behavior will exhibit different characteristics when observed from different angles. Multi-view action descriptions can provide complementary feature information, which helps to improve the robustness and accuracy of video behavior recognition. Since depth video can provide 3D data structure information, this embodiment projects the depth video onto three orthogonal view planes to obtain a richer and more comprehensive multi-view action description. In order to significantly capture the 3D spatiotemporal motion changes of video behavior, sequential pooling is further applied to the depth projection map sequence under each view to construct a dynamic volume representation.

[0059] Depth video projection

[0060] Based on the Depth Motion Maps (DMM) method, the depth video is projected onto three orthogonal 2D Cartesian planes to generate a sequence of depth projection maps from three perspectives. The XY plane represents the forward direction, the YZ plane represents the lateral direction, and the XZ plane represents the upward direction. This multi-view depth projection map sequence can provide more behavioral feature information from different perspectives, and is represented as follows: in, Let N represent the i-th depth image from the v-th projection viewpoint, and N represent the total number of frames in the depth video.

[0061] Dynamic volume construction

[0062] To effectively capture the salient spatiotemporal motion of human behavior in each projection viewpoint, a dynamic convolution representation is constructed based on a sequence of multi-view depth projection maps using the RankPooling method; specifically, taking the forward viewpoint as an example:

[0063] First, a sparse sampling strategy is used to generate the depth projection map sequence. Divide the data into T non-overlapping short segments, where f represents the forward viewpoint;

[0064] Then, sequential pooling is applied to aggregate short-term spatiotemporal information from each segment to generate the corresponding dynamic image. The sequential pooling method learns a linear sorting function whose parameters can encode the temporal order of video frames, which can usually be converted into a 2D image representation to describe the spatial motion dynamics over a period of time.

[0065] All the short, moving images together form a forward-viewing depth dynamics volume representation.

[0066] Similarly, lateral and tilt depth dynamics representations can be obtained, and depth dynamics representations from all viewpoints constitute a multi-view depth dynamics representation.

[0067] Multi-view depth dynamics describes the 3D spatiotemporal evolution of human behavior and can provide highly discriminative feature information to improve the performance of behavior recognition. Figure 2A selection of representative images from a multi-view depth dynamics volume with two classes of behavior in the NTU RGB+D 60 dataset are presented.

[0068] Step S2: Use a 3D CNN model to extract spatiotemporal features from the multi-view deep dynamic convolution, and extract the output feature map of the last convolutional layer as the multi-view feature map;

[0069] Convolutional Neural Networks (CNNs) have a deep structure and are adept at using convolution operations to progressively extract features with high-level semantic information from shallow to deep. Their hierarchical local feature patterns are more discriminative for behavior recognition. Therefore, based on the above characteristics of the CNN architecture, and considering the superior performance of 3D CNNs in simultaneously learning spatial and temporal feature information, this embodiment uses multi-view deep dynamic convolution inputs into the 3D CNN model for spatiotemporal feature extraction.

[0070] Specifically, Res3D-50 was chosen as the 3D backbone network due to its good balance between accuracy and efficiency. More specifically, the depth dynamic convolution under each projection viewpoint was fed into the Res3D-50 network, and the output convolutional feature map of the 4th layer was extracted and represented as a ConvF function. v ∈R T×C×H×w Where v represents the projection viewpoint (front / side / top), T is the time length, C is the number of channels, and H×W is the spatial size of the feature map; the three viewpoint convolutional feature maps ConvF v It is used as the input feature embedding for the Transformer multi-view feature interaction and fusion part in the next step.

[0071] Step S3: Based on the multi-view feature maps, learn the long-distance spatiotemporal correlation in the feature maps under each individual view, and generate the spatiotemporal aggregated feature representation for each view.

[0072] To learn the spatiotemporal correlations within the convolutional feature maps of each viewpoint, this embodiment designs an Intra-view Self-attention Encoder Module (ISEM). This module fully utilizes the self-attention mechanism in the Transformer encoder to learn the spatiotemporal dependencies in the feature maps of each viewpoint, obtaining a spatiotemporal aggregated feature representation. The specific structure of ISEM is detailed below using the forward projection viewpoint as an example. Figure 3 As shown.

[0073] The multi-view feature map output by step S2 is represented as ConvF f ∈R T×C×H×WWhere f represents forward, T = 1, C = 2048, H = W = 4, the specific steps are as follows:

[0074] Step S301: Perform dimensionality processing on the multi-view feature maps to obtain the input sequence of the Transformer encoder;

[0075] First, the time dimension T=1 is compressed; then, the convolutional feature map is unfolded along the spatial dimension and mapped to a spatial embedding X through a linear transformation. f ∈R HW×d Here, d represents the dimension of the embedding vector, which was set to 512 in subsequent experiments. Similar to BERT, in the embedding vector sequence X... f A learnable marker X was also added in front. cls ∈R d The final input sequence for ISEM is generated as follows:

[0076]

[0077] Considering that convolutional features have multi-channel and multi-level characteristics, and their receptive field can reflect the feature information at the video spatial location, this embodiment does not use the position embedding vector.

[0078] Step S302: The Transformer encoder uses a self-attention mechanism to learn the spatiotemporal dependencies in the feature maps of each viewpoint, and obtains a spatiotemporal aggregated feature representation.

[0079] In a Transformer encoder architecture with an input sequence X of depth L1, each layer l consists of multi-head self-attention (MSA), layer normalization (LN), and a multilayer perceptron (MLP). MSA uses a multi-head mechanism to extend the model's ability to focus on different locations in the input data; for each head h... i The self-attention computation first uses different linear transformations to map the input X to a query matrix. Key matrix Sum matrix The i-th head h i The self-attention calculation process is as follows:

[0080]

[0081] The outputs of all the heads are then concatenated and the output of MSA is obtained through a linear mapping:

[0082] MSA(X) = Concat[h i(X)]W O (3)

[0083] in, It is a parameter matrix; in the experiment, the number of multiple heads h = 8, d k =d v =d / h=64; After adding a residual connection to the MSA output, input it into the MLP to obtain the output of the l-th layer of the Transformer encoder:

[0084] Y l =MSA(LN(X) l ))+X l (4)

[0085] X l+1 =MLP(LN(Y) l ))+Y l (5)

[0086] The MLP consists of two linear layers separated by the GELU nonlinear activation function; after information transmission through the L1 layer, the final state at the first learnable labeled position output by the Transformer encoder is taken as the spatiotemporal aggregate feature representation of the forward view.

[0087] Similarly, spatiotemporal aggregated feature representations from lateral and top-down perspectives can be obtained; these three spatiotemporal aggregated feature representations, as the outputs of the ISEM, are respectively represented as F. f ∈R d F s ∈R d and F t ∈R d , where f, s, and t represent forward / lateral / top orientation. ISEM uses MSA to model the spatiotemporal correlation in the convolutional feature map, generating a highly discriminative spatiotemporal feature representation for each viewpoint.

[0088] Step S4: Interact and fuse the spatiotemporal aggregated feature representations from different perspectives to generate a global multi-view feature representation of video behavior;

[0089] The spatiotemporal aggregation feature representation of three independent perspectives provides spatiotemporal motion information of video behavior in different directions. In order to further learn the intrinsic correlation of the features of the three perspectives, this embodiment designs a cross-view feature fusion module (CFFM). By extending the self-attention mechanism to a mutual attention mechanism, CFFM can effectively interact the features of different perspectives and finally generate a global multi-view feature representation of video behavior. Figure 4 The implementation process of CFFM is shown below in detail.

[0090] CFFM consists of two parts: cross-view feature interaction and multi-view feature fusion.

[0091] The cross-view feature interaction consists of L2 encoder layers, each of which comprises multi-head cross-attention (MCA), layer normalization (LN), and a multilayer perceptron (MLP); the spatiotemporal aggregated feature representation based on the three views output by ISEM (F f F s and F t The cross-perspective feature interaction part is implemented as follows:

[0092] First, feature interaction is performed between every two projection views; taking the forward and lateral directions as examples, similar to the calculation process of MSA, for the i-th head of MCA, the input sequence [F f Fs] is mapped to and d k =d v =64.

[0093] Then, the query vector from the forward view. Key vectors from a lateral perspective Sum value vector The input is fed into the MCA to compute the forward-to-lateral mutual attention output; similarly, the query vector from the lateral view... Key vectors in the forward view Sum value vector The input is fed into the MCA to calculate the lateral mutual attention output relative to the forward direction; the specific calculation process is as follows:

[0094]

[0095]

[0096] Among them, h i (F f ) is the forward-to-lateral mutual attention output, h i (F s ) is the lateral mutual attention output relative to the forward direction.

[0097] The output of all heads [h] i (F f ),h i (F s The layers are still concatenated and a linear mapping is used to obtain the output of the MCA; the output of the MCA is used as the input of the MLP to generate the output of the l-th encoder layer.

[0098] [Y f ,Y s ] l =MCA(LN([F f ,F s ] l ))+[F f ,F s ] l (8)

[0099] [F f ,F s ] l+1 =MLP(LN([Y f ,Y s ] l ))+[Y f ,Y s ] l (9)

[0100] After passing through the L2 layer for cross-view feature interaction, the output sequence of interaction features between the forward and lateral views is obtained. Using the above method, the interaction features between any two viewpoints can be obtained.

[0101] In the multi-view feature fusion section, the average value (Avg) of the interaction feature sequences of any two views in the cross-view feature interaction output is taken to obtain the cross-view feature representation. The cross-view feature representations of every two views are concatenated to fuse multi-view action information, resulting in the global multi-view feature representation F of the depth video behavior. fusion The entire process is represented as:

[0102]

[0103] CFFM utilizes the MCA mechanism to explore feature interactions between different perspectives, and further generates highly discriminative deep video feature representations by fusing multi-perspective interaction information.

[0104] Step S5: Input the global multi-view feature representation into the fully connected layer to complete the behavior recognition.

[0105] The global multi-view feature representation of deep video behavior F fusion The input is fed into a fully connected layer (FC) to complete behavior recognition and classification.

[0106] Example 2

[0107] This embodiment discloses a deep video behavior recognition system based on multi-view feature interaction fusion;

[0108] like Figure 5As shown, the deep video behavior recognition system based on multi-view feature interaction fusion includes a multi-view deep dynamic convolution construction module, a convolutional feature extraction module, an intra-view self-attention encoder module, a cross-view feature fusion module, and a behavior recognition module.

[0109] The multi-view depth dynamic volume construction module is configured to construct a multi-view depth dynamic volume from the original depth video using depth projection and ordered pooling techniques.

[0110] The convolutional feature extraction module is configured to perform spatiotemporal feature extraction on multi-view depth dynamic convolutions to obtain multi-view feature maps.

[0111] The in-view self-attention encoder module is configured to: learn the long-distance spatiotemporal correlation in the feature maps of each individual view based on the multi-view feature maps, and generate a spatiotemporal aggregated feature representation for each view.

[0112] The cross-perspective feature fusion module is configured to interact and fuse spatiotemporal aggregated feature representations from different perspectives to generate a global multi-perspective feature representation of video behavior.

[0113] The behavior recognition module is configured to input global multi-view feature representations into the fully connected layer to complete behavior recognition.

[0114] Example 3

[0115] The purpose of this embodiment is to provide a computer-readable storage medium.

[0116] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the deep video behavior recognition method based on multi-view feature interaction fusion as described in Embodiment 1 of this disclosure.

[0117] Example 4

[0118] The purpose of this embodiment is to provide an electronic device.

[0119] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the deep video behavior recognition method based on multi-view feature interaction fusion as described in Embodiment 1 of this disclosure.

[0120] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A deep video behavior recognition method based on multi-view feature interaction fusion, characterized in that, include: Multi-view depth dynamic volumes are constructed from raw depth videos using depth projection and ordered pooling techniques. Spatiotemporal feature extraction is performed on the multi-view deep dynamic volume to obtain multi-view feature maps; Based on multi-view feature maps, long-distance spatiotemporal correlations in the feature maps of each individual view are learned to generate spatiotemporal aggregated feature representations for each view. Interacting and fusing spatiotemporal aggregated feature representations from different perspectives to generate a global multi-perspective feature representation of video behavior; By extending the self-attention mechanism to a mutual attention mechanism through the cross-view feature fusion module, interaction and fusion between spatiotemporal aggregated feature representations from different perspectives are carried out to generate a global multi-view feature representation of video behavior. The cross-view feature fusion module includes cross-view feature interaction and multi-view feature fusion; The cross-view feature interaction is performed between every two views, multi-head mutual attention between views is calculated, and then the interaction feature sequence between any two views is obtained through a multilayer perceptron. The multi-view feature fusion involves averaging the interaction feature sequences of any two views to obtain a cross-view feature representation. The cross-view feature representations of every two views are then connected to fuse multi-view action information, resulting in the final global multi-view feature representation. The global multi-view feature representation is input into the fully connected layer to complete behavior recognition.

2. The deep video behavior recognition method based on multi-view feature interaction fusion as described in claim 1, characterized in that, The construction of the multi-view depth dynamic volume specifically involves: The original depth video is projected onto three orthogonal planes to generate a sequence of depth projection maps from three different viewpoints. Sequential pooling is performed on the depth projection map sequences from three different perspectives to obtain a multi-view dynamic depth volume.

3. The deep video behavior recognition method based on multi-view feature interaction fusion as described in claim 2, characterized in that, The sequential pooling process for the depth projection map sequences from the three viewpoints is as follows: A sparse sampling strategy is used to uniformly divide the depth projection map sequence under a single viewpoint into multiple non-overlapping short segments. Sequential pooling is applied to aggregate short-term spatiotemporal information from each short segment to generate the corresponding dynamic image; All the short, moving images together form a deep dynamic roll from a single perspective; A multi-view depth dynamic volume is formed by depth dynamic volumes from all perspectives.

4. The deep video behavior recognition method based on multi-view feature interaction fusion as described in claim 1, characterized in that, A 3D CNN model is used to extract spatiotemporal features from multi-view deep dynamic volumes.

5. The deep video behavior recognition method based on multi-view feature interaction fusion as described in claim 1, characterized in that, The method based on multi-view feature maps learns the long-distance spatiotemporal correlation in the feature maps of each individual viewpoint, generating a spatiotemporal aggregated feature representation for each viewpoint, specifically as follows: Dimensional processing is performed on the multi-view feature maps to obtain the input sequence of the in-view self-attention encoder module; The in-view self-attention encoder module consists of a multi-layer Transformer encoder architecture. It learns the spatiotemporal dependencies in the feature maps of each view through a self-attention mechanism, and obtains a spatiotemporal aggregated feature representation.

6. A deep video behavior recognition system based on multi-view feature interaction fusion, characterized in that, It includes a multi-view deep dynamic convolution construction module, a convolutional feature extraction module, an intra-view self-attention encoder module, a cross-view feature fusion module, and an action recognition module; The multi-view depth dynamic volume construction module is configured to: construct a multi-view depth dynamic volume from the original depth video using depth projection and sequential pooling techniques. The convolutional feature extraction module is configured to: extract spatiotemporal features from a multi-view depth dynamic convolution to obtain a multi-view feature map; The in-view self-attention encoder module is configured to: learn the long-distance spatiotemporal correlation in the feature map of each individual view based on the multi-view feature map, and generate a spatiotemporal aggregated feature representation for each view. The cross-view feature fusion module is configured to: interact and fuse spatiotemporal aggregated feature representations from different perspectives to generate a global multi-view feature representation of video behavior; By extending the self-attention mechanism to a mutual attention mechanism through the cross-view feature fusion module, interaction and fusion between spatiotemporal aggregated feature representations from different perspectives are carried out to generate a global multi-view feature representation of video behavior. The cross-view feature fusion module includes cross-view feature interaction and multi-view feature fusion; The cross-view feature interaction is performed between every two views, multi-head mutual attention between views is calculated, and then the interaction feature sequence between any two views is obtained through a multilayer perceptron. The multi-view feature fusion involves averaging the interaction feature sequences of any two views to obtain a cross-view feature representation. The cross-view feature representations of every two views are then connected to fuse multi-view action information, resulting in the final global multi-view feature representation. The behavior recognition module is configured to input global multi-view feature representations into a fully connected layer to complete behavior recognition.

7. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the deep video behavior recognition method based on multi-view feature interaction fusion as described in any one of claims 1-5.

8. An electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the deep video behavior recognition method based on multi-view feature interaction fusion as described in any one of claims 1-5.

Images