Video Behavior Recognition Method and System Based on Dual-Perspective Spatiotemporal Interaction Network

Abstract

The present invention discloses a video behavior recognition method and system based on a dual-view spatio-temporal interaction network; the method includes: projecting the depth video sequence of the video human behavior in the forward view and the lateral view to obtain the depth projection map sequences in the forward view and the lateral view; preprocessing the depth projection map sequences in the forward view and the lateral view to obtain the depth dynamic image sequences in the forward view and the lateral view; inputting the depth dynamic image sequences in the forward view and the lateral view into the dual-view spatio-temporal interaction network to obtain the video behavior recognition result. Depth data is invariant to changes in illumination and color and can provide three-dimensional structure information of the human body. Based on the depth video, the present invention can obtain multi-view feature descriptions of human behaviors, which is beneficial to constructing a more comprehensive and robust video behavior feature expression model.

Description

Technical Field

[0001] The present invention relates to the technical field of video behavior recognition, and particularly to a video behavior recognition method and system based on a dual-view spatio-temporal interaction network. Background Art

[0002] In recent years, video-based human behavior recognition has received continuous attention due to its wide application prospects in fields such as human-computer interaction, video surveillance, and elderly care. Video data is high-dimensional, containing both spatial structure information and temporal motion information. Therefore, for the video behavior recognition task, effective modeling of spatio-temporal features is the key to improving recognition performance. Traditional behavior recognition methods extract spatio-temporal information of video behaviors by manually designing complex feature descriptors, such as spatio-temporal interest points, depth motion projection maps, spatio-temporal depth cubes, etc. In recent years, with the explosive progress of deep learning, deep learning-based behavior recognition methods automatically learn spatio-temporal features of behaviors in videos with an end-to-end training framework, significantly improving the performance of behavior recognition.

[0003] From the perspective of spatio-temporal feature modeling, current deep learning-based behavior recognition methods are mainly divided into two categories: two-dimensional CNN-based methods and three-dimensional CNN-based methods. Two-dimensional CNN has strong learning ability for local spatial features of images, but it cannot directly model the video time structure, and additional time information usually needs to be supplemented when applied to the video behavior recognition task. Two-stream CNN is the most typical network framework in two-dimensional CNN, where RGB images and stacked optical flow images are respectively input into a spatial stream network and a temporal stream network to extract appearance and motion information for behavior recognition. The Temporal Segment Network (TSN) is an upgraded version of the two-stream network. Based on the two-stream CNN architecture, it operates on a series of short segments sparsely sampled from the entire video and can effectively model the long-term structure of the video. The Recurrent Neural Network (RNN) has good modeling ability for the time dependence of long-term videos. The network model of two-dimensional CNN + RNN uses CNN for spatial feature learning and uses RNN to achieve time dynamics modeling.

[0004] To overcome the limitation that two-dimensional CNN-based methods cannot capture both spatial and temporal information of videos simultaneously, three-dimensional CNN emerged. Tran et al. proposed C3D, which can encode the spatio-temporal features of behavior videos simultaneously by expanding two-dimensional convolutional kernels into three-dimensional convolutional kernels. Subsequently, Res3D was proposed to further improve the recognition performance. I3D is one of the current best three-dimensional network models. By expanding a two-dimensional network into a three-dimensional network, it can utilize pre-trained two-dimensional models to improve the accuracy of video behavior recognition. Although three-dimensional CNN has achieved superior recognition performance, its large number of model parameters brings a heavy computational burden. Some recent research works have designed a temporal correlation module and embedded it into a two-dimensional CNN architecture to rival the performance of three-dimensional CNN while reducing the computational complexity. The above-mentioned behavior recognition methods based on deep networks mainly use RGB video data as input and only explore the spatio-temporal feature expression patterns of video human behaviors from a single perspective, which limits the accuracy and robustness of behavior recognition. Summary of the Invention

[0005] To solve the deficiencies of the prior art, the present invention provides a video behavior recognition method and system based on a dual-view spatio-temporal interaction network; depth data is invariant to changes in illumination and color and can provide three-dimensional structure information of the human body. Based on depth videos, the present invention can obtain multi-view feature descriptions of human behaviors, which is conducive to constructing a more comprehensive and robust video behavior feature expression pattern.

[0006] On the one hand, a video behavior recognition method based on a dual-view spatio-temporal interaction network is provided, including:

[0007] Projecting the depth video sequence of video human behaviors in the forward view and the side view to obtain depth projection map sequences in the forward view and the side view;

[0008] Preprocessing the depth projection map sequences in the forward view and the side view to obtain depth dynamic image sequences in the forward view and the side view;

[0009] Inputting the depth dynamic image sequences in the forward view and the side view into the dual-view spatio-temporal interaction network to obtain video behavior recognition results;

[0010] Among them, the dual-view spatio-temporal interaction network replaces all standard residual blocks in the second, third, fourth, and fifth stages of the two-dimensional Resnet-50 network with dual-view spatio-temporal interaction blocks; the dual-view spatio-temporal interaction block is obtained by embedding a dual-view spatio-temporal interaction unit after the first two-dimensional convolutional layer and before the second two-dimensional convolutional layer of the standard residual block of the two-dimensional Resnet-50 network;

[0011] The dual-view spatio-temporal interaction unit first performs cross-view channel feature exchange and then performs intra-view temporal feature displacement to achieve in-depth interaction of spatio-temporal information in the forward view and the lateral view. Among them, the interacted forward and lateral convolutional feature maps output by the last dual-view spatio-temporal interaction block are both subjected to average pooling operations along the spatial and temporal dimensions respectively to generate feature representations of the forward view and the lateral view. The feature representations of the forward view and the lateral view are concatenated to obtain a video global feature representation, and the video global feature representation is input into a fully connected layer to obtain a behavior recognition result.

[0012] On the other hand, a video behavior recognition system based on a dual-view spatio-temporal interaction network is provided, including:

[0013] A projection module configured to project the depth video sequence of the video human behavior in the forward view and the lateral view to obtain depth projection map sequences of the forward view and the lateral view;

[0014] A preprocessing module configured to preprocess the depth projection map sequences of the forward view and the lateral view to obtain depth dynamic image sequences of the forward view and the lateral view;

[0015] An identification module configured to input the depth dynamic image sequences of the forward view and the lateral view into the dual-view spatio-temporal interaction network to obtain a video behavior recognition result;

[0016] Among them, in the dual-view spatio-temporal interaction network, all standard residual blocks in the second, third, fourth, and fifth stages of the two-dimensional Resnet-50 network are replaced with dual-view spatio-temporal interaction blocks. The dual-view spatio-temporal interaction block is obtained by embedding a dual-view spatio-temporal interaction unit after the first two-dimensional convolutional layer and before the second two-dimensional convolutional layer of the standard residual block of the two-dimensional Resnet-50 network;

[0017] The dual-view spatio-temporal interaction unit first performs cross-view channel feature exchange and then performs intra-view temporal feature displacement to achieve in-depth interaction of spatio-temporal information in the forward view and the lateral view. Among them, the interacted forward and lateral convolutional feature maps output by the last dual-view spatio-temporal interaction block are both subjected to average pooling operations along the spatial and temporal dimensions respectively to generate feature representations of the forward view and the lateral view. The feature representations of the forward view and the lateral view are concatenated to obtain a video global feature representation, and the video global feature representation is input into a fully connected layer to obtain a behavior recognition result.

[0018] On yet another aspect, an electronic device is further provided, including:

[0019] A memory for non-temporarily storing computer-readable instructions; and a processor for running the computer-readable instructions,

[0020] Wherein, when the computer-readable instructions are run by the processor, the method described in the first aspect above is executed.

[0021] In another aspect, a storage medium is further provided, which non-temporarily stores computer-readable instructions. When the non-temporary computer-readable instructions are executed by a computer, the instructions for executing the method described in the first aspect are executed.

[0022] In another aspect, a computer program product is further provided, including a computer program, and the computer program is used to implement the method described in the first aspect above when running on one or more processors.

[0023] The above technical solutions have the following advantages or beneficial effects:

[0024] (1) Focusing on depth video behavior recognition, a dual-view spatio-temporal interaction unit is proposed. By interacting the spatio-temporal features of the forward view and the lateral view along the channel dimension and applying temporal displacement, the interaction and perception of information from different views are completed. This module does not introduce additional parameters and can be flexibly embedded into a two-dimensional network model to achieve efficient modeling of spatio-temporal features of video behaviors.

[0025] (2) Embedding the dual-view spatio-temporal interaction unit into a two-dimensional Resnet50 network, a dual-view spatio-temporal interaction network is proposed. This network takes the forward and lateral depth dynamic image sequences as inputs and can realize end-to-end simultaneous training of a dual-path network architecture through dual-view spatio-temporal feature interaction.

[0026] (3) Evaluating the effectiveness of the proposed method on two challenging RGB-D datasets. Through a large number of experiments, it shows that the method of the present invention achieves advanced recognition performance. Description of the Drawings

[0027] The specification drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0028] Figure 1 It is the behavior recognition framework based on the dual-view spatio-temporal interaction network in the first embodiment;

[0029] Figure 2 It is the representation of the forward and lateral depth dynamic image sequences of four types of behaviors in the NTU RGB+D 60 dataset in the first embodiment;

[0030] Figure 3(a) is the schematic diagram of the channel exchange sub-unit in the first embodiment;

[0031] Figure 3(b) is the schematic diagram of the average fusion-channel exchange sub-unit in the first embodiment;

[0032] Figure 3(c) is the schematic diagram of the correlation score perspective interaction subunit in the first embodiment; Detailed implementation manners

[0033] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0034] In the case of no conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0035] All data acquisition in this embodiment is based on compliance with laws and regulations and user consent, and is a legal application of the data.

[0036] Capturing strong discriminative spatio-temporal feature information is the key to improving the performance of video behavior recognition. Currently, most methods mainly achieve behavior recognition in a single perspective by embedding a spatio-temporal feature modeling module into the backbone network. However, video behavior has a three-dimensional structure, and multi-perspective information helps to obtain more comprehensive behavior clues to improve the robustness and accuracy of behavior recognition. Depth data is insensitive to illumination and color changes, and has significant advantages especially in providing reliable three-dimensional geometric information of human behavior.

[0037] The present invention focuses on the task of depth video behavior recognition, and proposes a dual-view spatio-temporal interaction network, which enhances the performance of behavior recognition by effectively fusing spatio-temporal interaction features from different forward and lateral perspectives. Specifically, the core of the dual-view spatio-temporal interaction network is the dual-view spatio-temporal interaction unit (Dual-view Spatio-Temporal Interactive Module, DSTIM). Based on the convolutional feature maps of two perspectives, spatio-temporal information exchange is first performed along the channel dimension to achieve inter-perspective channel feature interaction, and then temporal displacement is applied along the time dimension within each perspective to model motion information. The proposed DSTIM can be flexibly embedded into a two-dimensional depth network architecture to generate a dual-view spatio-temporal interaction network. This network takes depth dynamic image sequences from two different perspectives as inputs, and through spatio-temporal information interaction and fusion between perspectives, can achieve an end-to-end behavior recognition training framework. A large number of experiments conducted on two large-scale RGBD datasets show that the proposed method can significantly improve the performance of video behavior recognition.

[0038] The first embodiment

[0039] This embodiment provides a video behavior recognition method based on a dual-view spatio-temporal interaction network, including:

[0040] S101: Project the depth video sequence of the video human behavior in the front view and the side view to obtain the depth projection map sequences of the front view and the side view;

[0041] S102: Preprocess the depth projection map sequences of the front view and the side view to obtain the depth dynamic image sequences of the front view and the side view;

[0042] S103: Input the depth dynamic image sequences of the front view and the side view into the dual-view spatio-temporal interaction network to obtain the video behavior recognition result;

[0043] Among them, the dual-view spatio-temporal interaction network replaces all the standard residual blocks in the second, third, fourth, and fifth stages of the two-dimensional Resnet-50 network with dual-view spatio-temporal interaction blocks; the dual-view spatio-temporal interaction block is obtained by embedding a dual-view spatio-temporal interaction unit after the first two-dimensional convolutional layer and before the second two-dimensional convolutional layer of the standard residual block of the two-dimensional Resnet-50 network;

[0044] The dual-view spatio-temporal interaction unit first performs cross-view channel feature exchange, and then performs intra-view temporal feature displacement to achieve in-depth interaction of spatio-temporal information under the front view and the side view; the interacted front and side convolutional feature maps output by the last dual-view spatio-temporal interaction block are respectively subjected to average pooling operations along the spatial and temporal dimensions to generate feature representations of the front view and the side view; the feature representations of the front view and the side view are concatenated to obtain the video global feature representation, and the video global feature representation is input into the fully connected layer to obtain the behavior recognition result.

[0045] Further, in S101: the depth video sequence of the video human behavior is collected by a Kinect camera. [[ID=!7]]

[0046] Further, in S101: Project the depth video sequence of the video human behavior in the front view and the side view to obtain the depth projection map sequences of the front view and the side view, which specifically includes:

[0047] In view of the 3D attribute of the depth video data, the depth sequence is projected in the front and side directions to obtain the depth projection map sequence. Among them, the front view is the X-Y plane, and the side view is the Y-Z plane. The depth projection map sequences under the two views are represented as where is the t-th image under the projection view v, and N is the sequence length.

[0048] Further, in S102: Preprocess the depth projection map sequences of the front view and the side view to obtain the depth dynamic image sequences of the front view and the side view, which specifically includes:

[0049] S102-1: Uniformly segment both the forward-view and side-view depth projection map sequences.

[0050] S102-2: Perform sorting pooling on the uniformly segmented results to obtain the forward and side-view depth dynamic image sequences.

[0051] Furthermore, the S102-1: Uniformly segment both the forward-view and side-view depth projection map sequences, specifically including:

[0052] Based on the forward and side-view depth projection map sequences, use a uniform non-overlapping segmentation method to divide them into a set of short segment sequences. The number of short segments is 16.

[0053] Furthermore, the S102-2: Perform sorting pooling on the uniformly segmented results to obtain the forward and side-view depth dynamic image sequences, specifically including:

[0054] Apply sorting pooling to each short-time segment to aggregate local spatio-temporal motion changes and generate a depth dynamic image.

[0055] Sorting pooling, as a temporal pooling method, can not only capture the temporal changes of the video well but also be easily implemented. Sorting pooling uses a pairwise linear sorter to learn a linear function whose parameters can encode the temporal order of video frames.

[0056] Where represents the j-th image in the t-th short segment of the depth projection sequence, and L is the length of the short segment. The time-varying average vector operation is used to capture the temporal information between consecutive frames in the t-th segment, as shown in Equation (1).

[0057] <00^00133>The smoothed vector sequence can still retain the temporal information between the L consecutive images in the t-th segment. The linear sorting function is defined as α is the parameter vector of the sorting function, which can retain the relative temporal information between video frames, that is, if then the sorting function value satisfies Use structural risk minimization to define the objective function of sorting pooling, as shown in Equation (2).

[0059]

[0060] ε

[0059] , , , , ,

[0058] , * , , , , , , , ij , ,

[0060] , , ,

[0057] is a slack variable, and α * is the optimal parameter vector that satisfies the above objective function and is subsequently converted into a two-dimensional matrix representation to generate the dynamic image.

[0061] Using the parameters of the linear sorting function can effectively encode the temporal relationship within the video. The dynamic images of 16 short segments are stacked along the time dimension to form a forward / lateral depth dynamic image sequence. Figure 2 The forward and lateral dynamic image sequence representations of four types of behaviors in the NTU RGB+D 60 dataset are given. It can be seen that different perspectives can capture the dynamic behavior movements in different dimensions, providing more comprehensive and rich video behavior description information.

[0062] Video behaviors are high-dimensional, and different perspectives can provide complementary behavior feature descriptions, which helps to improve the comprehensiveness and robustness of feature expression. In order to capture more discriminative spatio-temporal motion information in depth videos, the present invention projects the depth video sequence in multiple dimensions, and then uses sorting pooling to construct a dual-view depth dynamic image sequence. Since the downward perspective contains too much ground noise and greatly interferes with obtaining recognizable behavior motion change information, the present invention mainly considers using the forward and lateral perspectives.

[0063] Further, in step S103: input the depth dynamic image sequences of the forward perspective and the lateral perspective into a dual-view spatio-temporal interaction network to obtain the video behavior recognition result. Among them, the training process of the dual-view spatio-temporal interaction network includes:

[0064] Construct a training set and a test set, both of which are depth videos with known video behavior recognition results; construct the first, second, and third dual-view spatio-temporal interaction networks;

[0065] The first, second, and third dual-view spatio-temporal interaction networks include the same parts and different parts;

[0066] Among them, the same parts refer to: the first, second, and third dual-view spatio-temporal interaction networks all include: replacing all the standard residual blocks in the second, third, fourth, and fifth stages of the two-dimensional Resnet-50 network with dual-view spatio-temporal interaction blocks; the dual-view spatio-temporal interaction block is obtained by embedding a dual-view spatio-temporal interaction unit after the first two-dimensional convolutional layer and before the second two-dimensional convolutional layer of the standard residual block of the two-dimensional Resnet-50 network;

[0067] Among them, the different parts refer to: the dual-view spatio-temporal interaction unit of the first dual-view spatio-temporal interaction network includes: a channel exchange subunit and a temporal displacement subunit connected in sequence;

[0068] The dual-view spatio-temporal interaction unit of the second dual-view spatio-temporal interaction network includes: an average fusion-channel exchange subunit and a temporal displacement subunit connected in sequence;

[0069] The dual-view spatio-temporal interaction unit of the third dual-view spatio-temporal interaction network includes: a correlation score view interaction subunit and a temporal displacement subunit connected in sequence;

[0070] Input the training set into the first, second, and third dual-view spatio-temporal interaction networks respectively, and train the three networks separately. When the loss function value of each network no longer decreases, or the number of iterations reaches the set number, stop training to obtain the trained first, second, and third dual-view spatio-temporal interaction networks;

[0071] Input the test set into the first, second, and third dual-view spatio-temporal interaction networks, and select the dual-view spatio-temporal interaction network with the highest test accuracy as the dual-view spatio-temporal interaction network.

[0072] Furthermore, the dual-view spatio-temporal interaction network is obtained by improving the original Resnet50 network.

[0073] Furthermore, the first, second, and third dual-view spatio-temporal interaction networks all include: the first-stage Conv1, the second-stage Conv2_x, the third-stage Conv3_x, the fourth-stage Conv4_x, the fifth-stage Conv5_x, the global average pooling layer, and the fully connected layer;

[0074] The first-stage Conv1 is a two-dimensional convolutional layer;

[0075] The second-stage Conv2_x includes: a max pooling layer and three dual-view spatio-temporal interaction blocks connected in sequence;

[0076] The third-stage Conv3_x includes: four dual-view spatio-temporal interaction blocks connected in sequence;

[0077] The fourth-stage Conv4_x includes: six dual-view spatio-temporal interaction blocks connected in sequence;

[0078] The fifth-stage Conv5_x includes: three dual-view spatio-temporal interaction blocks connected in sequence.

[0079] Furthermore, the dual-view spatio-temporal interaction block includes: a 1×1 two-dimensional convolutional layer, a dual-view spatio-temporal interaction unit, a 3×3 two-dimensional convolutional layer, a 1×1 two-dimensional convolutional layer, and an adder connected in sequence; the input end of the first 1×1 two-dimensional convolutional layer is connected to the input end of the adder; the input end of the first 1×1 two-dimensional convolutional layer is the input end of the dual-view spatio-temporal interaction block; the output end of the adder is the output end of the dual-view spatio-temporal interaction block.

[0080] Furthermore, the standard residual block includes: a 1×1 two-dimensional convolutional layer, a 3×3 two-dimensional convolutional layer, a 1×1 two-dimensional convolutional layer, and an adder connected in sequence; the input end of the first 1×1 two-dimensional convolutional layer is connected to the input end of the adder; the input end of the first 1×1 two-dimensional convolutional layer is the input end of the standard residual block; the output end of the adder is the output end of the standard residual block.

[0081] It should be understood that the two-dimensional convolutional layer is used to implement feature extraction, and the dual-view spatio-temporal interaction unit is used to implement feature interaction and fusion.

[0082] It should be understood that the present invention constructs three dual-view spatio-temporal interaction networks, and by comparing the recognition accuracies of different networks in experiments, the network with the highest accuracy is selected for application.

[0083] Exemplarily, the known video behavior recognition results are, for example: human behavior actions such as raising hands, tearing paper, drinking water, running, etc.

[0084] After comparison with the test set, in this embodiment, the dual-view spatio-temporal interaction unit includes: a channel exchange sub-unit and a temporal displacement sub-unit connected in sequence.

[0085] Furthermore, the channel exchange sub-unit includes: assuming that the forward and lateral convolutional feature maps are respectively represented as ConvF ∈ R T×C×H×W and ConvS ∈ R T×C×H×W , channel exchange is to swap the feature channels of a set ratio P along the channel dimension C of the feature maps ConvF and ConvS, and the remaining channel features remain unchanged.

[0086] It should be understood that channel exchange realizes the interactive perception of forward and lateral feature information and does not introduce any additional parameters. The specific implementation process of channel exchange is shown in Fig. 3(a).

[0087] Furthermore, the average fusion-channel exchange sub-unit includes:

[0088] First, the forward and lateral convolutional feature maps are fused by average operation;

[0089] The generated fused feature map ConvAve ∈ R T×C×H×W is respectively subjected to channel feature exchange with the forward feature map ConvF and the lateral feature map ConvS:

[0090] ConvAve is evenly divided into two parts along the channel dimension. The first 1 / 2 channels are taken to replace the first 1 / 2 feature channels of ConvF, and the last 1 / 2 channels are taken to replace the last 1 / 2 feature channels of ConvS. The remaining channels of ConvF and ConvS remain unchanged with the original feature information.

[0091] It should be understood that the average fusion-channel exchange can also ensure that the forward and lateral features after interaction contain complementary information of each other without introducing additional parameters. The specific implementation process is shown in Fig. 3(b).

[0092] Furthermore, the correlation score perspective interaction subunit includes:

[0093] The interaction and fusion of the two-way perspective are realized by calculating the correlation between the forward feature and the lateral feature at each moment: for each moment, first calculate the correlation score matrix S of the forward spatial feature and the lateral spatial feature of each channel along the channel dimension;

[0094]

[0095] where, and respectively represent the spatial feature vectors of the k-th channel at the t-th moment in the forward and lateral directions, t ∈ {1, 2... T}, k ∈ {1, 2... C}, the symbol ⊙ represents the vector dot product operation, and S ∈ R T×C represents the correlation score matrix;

[0096] The correlation score matrix S is multiplied by the lateral convolutional feature map ConvS along the spatial dimension, and then added to the forward convolutional feature map ConvF to obtain the feature information after the forward and lateral interaction;

[0097] Similarly, S is multiplied by the forward convolutional feature map ConvF along the spatial dimension, and then added to the lateral convolutional feature map ConvS to obtain the feature information after the lateral and forward interaction. The specific implementation process is shown in Fig. 3(c).

[0098] In the experiment, the present invention compares the performance of the above three information interaction strategies. The results show that the channel exchange strategy can obtain better recognition accuracy. Therefore, the channel exchange strategy is adopted in the subsequent experiments to realize the information interaction of the two-way perspective.

[0099] After the forward perspective feature map and the lateral perspective feature map perform information interaction between perspectives, it can ensure that the two-way perspective contains more comprehensive and rich feature information. In order to model the temporal features of video behaviors, the present invention further uses the temporal displacement subunit to capture the motion information within the perspective.

[0100] Furthermore, the temporal displacement subunit includes:

[0101] The interacted forward feature map is moved forward and backward along the time dimension by a certain number of feature channels;

[0102] The interacted lateral feature map is moved forward and backward along the time dimension by a certain number of feature channels.

[0103] It should be understood that the temporal displacement subunit enables the current frame to perceive both forward motion information and backward motion information, enhancing the temporal expression ability of features without introducing additional computational parameters. Through the channel feature exchange and temporal feature displacement of the forward and lateral perspectives, the dual-view spatio-temporal interaction unit realizes the deep interaction of spatio-temporal feature information under different perspectives.

[0104] The dual-view spatio-temporal interaction unit uses channel exchange and temporal displacement to realize the perception and interaction of complementary feature information in the forward and lateral perspectives, without introducing additional computational parameters, and can be flexibly embedded into the two-dimensional convolutional network architecture to enhance the spatio-temporal feature expression ability of the network. Considering that the two-dimensional Resnet-50 network has a good balance between efficiency and accuracy, the dual-view spatio-temporal interaction unit is embedded after the first 1x1 two-dimensional convolutional layer in the standard residual block in the present invention to generate a dual-view spatio-temporal interaction block. The dual-view spatio-temporal interaction block replaces all the standard residual blocks in the second, third, fourth, and fifth stages of the two-dimensional Resnet-50 network to construct a dual-view spatio-temporal interaction network. The overall architecture of the network is as Figure 1 shown.

[0105] The behavior recognition framework based on the dual-view spatio-temporal interaction network (such as Figure 1 ) takes the forward and lateral depth dynamic image sequences as inputs, and realizes the interaction of multi-scale complementary feature information by using the dual-view spatio-temporal interaction unit in different convolutional layers. The two-way perspective uses spatio-temporal information interaction as a bridge and can be jointly trained. Finally, the convolutional feature maps after the forward and lateral interactions are respectively processed by the global average pooling layer, and the global average pooling layer is used to apply an average operation to the feature maps to generate the feature representations of each perspective. Subsequently, the two are concatenated to obtain the global feature for behavior recognition, and the global feature is input into the fully connected layer to realize an efficient end-to-end behavior recognition framework.

[0106] The dual-view spatio-temporal interaction unit (Dual-view Spatio-Temporal Interactive Module, DSTIM) is the core component of the dual-view spatio-temporal interaction network. Based on the spatio-temporal feature maps extracted by inputting the forward and lateral depth dynamic image sequences into the 2D convolutional neural network respectively, the dual-view spatio-temporal interaction unit first performs the feature perception and interaction between perspectives, and then models the motion information of video behaviors through the temporal feature displacement within the perspective, thereby realizing the deep interaction of dual-view spatio-temporal features.

[0107] Specifically, based on the forward and lateral convolutional feature maps, the present invention designs three spatio-temporal information interaction strategies, namely Channel Exchange (CE), Average Fusion-Channel Exchange (AF-CE), and Correlation Score View Interaction (CSVI).

[0108] The present invention focuses on the task of depth video action recognition, and proposes a dual-view spatio-temporal interaction network, which improves the accuracy of action recognition through feature interaction and fusion from different views. The overall architecture is as Figure 1 shown. Specifically, the present invention first performs forward and lateral projections on the depth video sequence, and uses uniform segmentation and sorting pooling to construct depth dynamic image sequences from the forward and lateral views, so as to fully capture the spatio-temporal motion information of human actions from different views.

[0109] The dual-view spatio-temporal interaction network uses a two-dimensional CNN as the backbone network, embeds a dual-view spatio-temporal interaction unit, and realizes the effective learning of spatio-temporal information of video actions through cross-view channel feature exchange and intra-view temporal feature modeling. The dual-view spatio-temporal interaction unit completes spatio-temporal information interaction through channel exchange and temporal displacement, without introducing other parameters additionally, and can be flexibly and efficiently embedded into the two-dimensional CNN model, endowing the network with powerful spatio-temporal feature modeling capabilities.

[0110] The present invention inserts it into the Resnet50 network to construct a dual-view spatio-temporal interaction network, and tests the effectiveness of the proposed method in the NTU RGB+D 60 and NTU RGB+D 120 datasets. The experimental results show that the method of the present invention achieves superior recognition performance only using depth data, and significantly improves the accuracy of action recognition.

[0111] The present invention evaluates the performance of the proposed action recognition framework based on the dual-view spatio-temporal interaction network to verify the effectiveness of the in-depth interaction and fusion of spatio-temporal feature information from different views. First, the video action dataset used for performance testing and the parameter settings during network training are introduced; secondly, the architecture design of the dual-view spatio-temporal interaction network is tested from different aspects; finally, the comparison results between the proposed method and other excellent algorithms are given.

[0112] NTU RGB+D 60 Dataset: This dataset contains 60 types of human daily behaviors performed by 40 individuals, with a total of approximately 56,880 video samples. The dataset uses three Microsoft Kinect v.2 cameras to record human behaviors simultaneously from different angles, providing data in four modalities: color, depth, skeleton, and infrared. For the behavior recognition task, there are two evaluation protocols: Cross-Subject (CS) and Cross-View (CV). The CS protocol uses video samples of 20 individuals for training and the remaining 20 for testing; the CV protocol uses video samples recorded by cameras 2 and 3 for training and those recorded by camera 1 for testing.

[0113] NTU RGB+D 120 Dataset: This dataset is an extended version of the NTU RGB+D 60 dataset, containing 120 types of human daily behaviors performed by 106 individuals, with a total of 114,480 video samples. Similarly, three cameras are used to record human behaviors simultaneously, and 32 parameter settings are designed by adjusting the height and distance of the cameras. The evaluation protocols provided by this dataset include Cross-Subject (CSub) and Cross-Setup (CSet). The CSub protocol uses video samples of 53 individuals for training and the remaining 53 for testing; the CSet protocol uses even IDs for training and odd IDs for testing.

[0114] The training and testing of the network in this invention are performed based on the Pytorch framework. The time lengths of the forward and lateral depth dynamic image sequences are both set to 16. The initial learning rate is 0.001, and the Stochastic Gradient Descent method with a momentum of 0.9 is used as the optimizer. This invention uses the Imagenet dataset and the pre-trained two-dimensional Resnet-50 as the backbone network. The data augmentation strategies used in the training phase include multi-scale cropping and random horizontal flipping. The model of this invention is trained on four 3090Ti GPUs with a batch size of 24 and a total of 80 iterations.

[0115] Comparison of Different Dual-View Spatiotemporal Information Interaction Strategies: This experiment compares three information interaction strategies designed in the dual-view spatiotemporal interaction unit: Channel Exchange (CE), Average Fusion - Channel Exchange (AF-CE), and Correlation Score View Interaction (CSVI). The experiment is based on the Cross-Subject (CS) and Cross-View (CV) evaluation protocols of the NTU RGB+D 60 dataset, using 16-frame forward and lateral depth dynamic image sequences as input, with the Resnet50 as the backbone network architecture to test the performance of different dual-view spatiotemporal information interaction strategies.

[0116] Table 1 Comparison of Three Dual-View Spatiotemporal Information Interaction Strategies

[0117]

[0118] The experimental results are shown in Table 1. It can be seen that the channel exchange strategy achieved the highest recognition accuracy under both the CS and CV evaluation protocols, which are 93.34% and 93.69% respectively. Under the CS evaluation protocol, the recognition rate of the channel exchange strategy is 1.03% higher than that of the average fusion-channel exchange and 2.18% higher than that of the correlation score perspective interaction, realizing a significant performance improvement. Under the CV evaluation protocol, the recognition rate of the channel exchange strategy is 0.54% higher than that of the average fusion-channel exchange and 1.65% higher than that of the correlation score perspective interaction. Therefore, based on the above experimental results, the present invention selects the channel exchange strategy as the information interaction method in the dual-perspective spatio-temporal interaction unit.

[0119] Comparison of different channel selection ratios in the channel exchange strategy: The channel exchange strategy performs spatio-temporal information exchange by selecting a certain proportion of feature channels from the forward convolutional feature map and the lateral convolutional feature map. In this experiment, different channel selection ratios were tested to evaluate the influence of the number of exchanged feature channels on the recognition performance. This experiment was based on the cross-person CS evaluation protocol of the NTU RGB+D 60 dataset, still using 16-frame forward and lateral depth dynamic image sequences as input, and the backbone network architecture was Resnet50.

[0120] The experimental results are shown in Table 2, where setting the channel ratio to 1 / 6 achieved the best recognition performance, significantly superior to the channel ratios of 1 / 2 and 1 / 3, with improvements of 2.14% and 1.01% respectively. Compared with the channel ratio of 1 / 4, the channel ratio of 1 / 6 only obtained a 0.11% performance improvement. In summary, the present invention selects the channel exchange ratio of 1 / 6 for dual-perspective spatio-temporal information interaction, and this parameter setting was used in all other experiments.

[0121] Table 2 Comparison of different channel selection ratios

[0122]

[0123] Performance evaluation of the dual-perspective spatio-temporal interaction unit: To fully evaluate the effectiveness of the proposed dual-perspective spatio-temporal interaction unit, this experiment compared it with different action recognition network architectures. The experiment was based on the CS evaluation protocol of the NTU RGB+D 60 dataset, and the experimental results are shown in Table 3.

[0124] Specifically, the "Resnet50" method refers to removing the dual-view spatio-temporal interaction unit, inputting 16-frame forward and lateral depth dynamic image sequences into the Resnet50 network respectively, and finally fusing the spatio-temporal feature information of the two views through feature concatenation to complete action recognition. "Resnet50 + Temporal Shift" means removing the channel exchange part in the dual-view spatio-temporal interaction unit and only performing temporal shift within the view. Finally, action recognition is still achieved by inputting the fused features into the fully connected layer. "Resnet50 + Dual-View Spatio-Temporal Interaction Unit" is the complete action recognition framework proposed in the present invention. In addition, in order to verify that the forward and lateral views can provide complementary and robust feature information, the present invention compares the performance of dual-view spatio-temporal interaction and single view respectively. The "single view" refers to the recognition results obtained by inputting the forward and lateral depth dynamic image sequences into the "Resnet50 + Temporal Shift" network architecture respectively.

[0125] As can be seen from Table 3, the proposed method achieves superior recognition performance. Compared with the single-view recognition method, the CS accuracy is 1.82% better than the forward view and 6.59% better than the lateral view. In addition, the proposed action recognition framework based on the dual-view spatio-temporal interaction unit is significantly better than the basic network architecture "Resnet50", and the performance improvement is about 2.39%. Moreover, only using the basic network architecture "Resnet50" also obtains an action recognition accuracy of 90.95%, which proves the effectiveness of the constructed forward and lateral depth dynamic image sequences and can provide spatio-temporal motion information of video actions with strong discriminability.

[0126] Compared with the "Resnet50 + Temporal Shift" method, it can be found that after adding the inter-view channel exchange, the performance is improved by 1.13%. Thus, it can be seen that the dual-view spatio-temporal interaction unit can achieve in-depth interaction and fusion of the spatio-temporal information of the forward and lateral views through inter-view channel information exchange and intra-view temporal feature shift, significantly improving the performance of action recognition.

[0127] Table 3 Performance Evaluation of Dual-View Spatio-Temporal Interaction Unit

[0128]

[0129] Comparison with Other Excellent Algorithms:

[0130] To verify the superiority of the proposed method, the present invention compares it with other excellent algorithms on the NTU RGB+D 60 and NTU RGB+D120 datasets. Table 4 shows the comparison results of the proposed method and other algorithms on the NTU RGB+D 60 dataset. The behavior recognition method based on the dual-view spatio-temporal interaction network proposed by the present invention has an identification accuracy of 93.34% under the cross-person CS evaluation protocol, which is significantly better than other excellent algorithms in the past, achieving the best recognition performance; the identification accuracy under the cross-view CV evaluation protocol is 93.69%. Compared with the highest recognition rate under this protocol, the performance has decreased slightly. Specifically, under the CS evaluation protocol, the proposed method has achieved a large performance improvement, about 0.64% higher than the current optimal behavior recognition method GeometryMotion-Net. However, for the CV evaluation protocol, the proposed method has comparable recognition performance with GlimpseClouds, which is lower than GeometryMotion-Net. The possible reason for the analysis is that the proposed method is sensitive to spatio-temporal feature information under different perspectives and does not focus on the significant motion regions of video behaviors, resulting in a decrease in CV recognition accuracy.

[0131] Table 5 presents the comparison results of the proposed method and other excellent algorithms on the NTU RGB+D 120 dataset. The identification accuracies of the proposed method under the cross-person CSub and cross-set CSet evaluation protocols are 89.48% and 91.76% respectively, both of which are better than the vast majority of algorithms on this dataset, achieving excellent recognition performance. Under the CSub evaluation protocol, the proposed method has achieved similar recognition performance to the current optimal behavior recognition method GeometryMotion-Net, with an identification accuracy only 0.62% lower, and is significantly better than other recognition algorithms. For the CSet evaluation protocol, the identification accuracy of the proposed method has decreased by 1.84% compared to GeometryMotion-Net and 1.74% compared to 3DV-PointNet++. However, the CSub accuracy of the method of the present invention exceeds that of 3DV-PointNet++ by about 7%.

[0132] In summary, the proposed dual-view spatio-temporal interaction network only uses depth video data and does not introduce additional model parameters, and can achieve advanced recognition performance, proving the effectiveness of the proposed method and the robustness of dual-view spatio-temporal information interaction.

[0133] Table 4 Comparison of Recognition Results on NTU RGB+D 60 Dataset

[0134]

[0135] Table 5 Comparison of Recognition Results on NTU RGB+D 120 Dataset

[0136]

[0137] The present invention proposes a dual - view spatio - temporal interaction unit, which realizes the in - depth interaction of spatio - temporal information under the forward and lateral views through the exchange of channel features between views and the displacement of temporal features within views. This module does not increase the additional computational burden, is plug - and - play, and can be flexibly embedded into the 2D deep network architecture to construct a dual - view spatio - temporal interaction network. Based on the output feature maps of different convolutional layers, the dual - view spatio - temporal interaction network builds a bridge between the forward data stream and the lateral data stream through spatio - temporal feature interaction, realizing the perception and fusion of multi - scale complementary feature information of the two views. In addition, in order to more effectively capture the spatio - temporal motion information of video behaviors, the present invention constructs the depth dynamic image sequences under the forward and lateral views as the input data of the dual - view spatio - temporal interaction network. The proposed method only uses depth data and outperforms the vast majority of existing methods on both the NTU RGB+D 60 and NTU RGB+D 120 datasets, achieving a significant performance improvement, which verifies the effectiveness and applicability of the proposed method.

[0138] Embodiment 2

[0139] This embodiment provides a video behavior recognition system based on a dual - view spatio - temporal interaction network;

[0140] A video behavior recognition system based on a dual - view spatio - temporal interaction network includes: [[ID=1⑤]]

[0141] A projection module, which is configured to project the depth video sequence of the video human behavior in the forward view and the lateral view to obtain the depth projection map sequences of the forward view and the lateral view;

[0142] A pre - processing module, which is configured to perform pre - processing on the depth projection map sequences of the forward view and the lateral view to obtain the depth dynamic image sequences of the forward view and the lateral view;

[0143] An identification module, which is configured to input the depth dynamic image sequences of the forward view and the lateral view into the dual - view spatio - temporal interaction network to obtain the video behavior recognition result;

[0144] Among them, the dual - view spatio - temporal interaction network replaces all the standard residual blocks in the second, third, fourth, and fifth stages of the two - dimensional Resnet - 50 network with dual - view spatio - temporal interaction blocks; the dual - view spatio - temporal interaction block is obtained by embedding a dual - view spatio - temporal interaction unit after the first two - dimensional convolutional layer and before the second two - dimensional convolutional layer of the standard residual block of the two - dimensional Resnet - 50 network;

[0145] The dual-view spatio-temporal interaction unit first performs cross-view channel feature exchange and then performs intra-view temporal feature displacement to achieve in-depth interaction of spatio-temporal information in the forward view and the lateral view. Among them, the interaction-convolved forward and lateral feature maps output by the last dual-view spatio-temporal interaction block are both subjected to average pooling operations along the spatial and temporal dimensions respectively to generate feature representations of the forward view and the lateral view. The feature representations of the forward view and the lateral view are concatenated to obtain a video global feature representation, and the video global feature representation is input into a fully-connected layer to obtain a behavior recognition result.

[0146] It should be noted here that the above projection module, preprocessing module, and recognition module correspond to steps S101 to S103 in Embodiment 1. The examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1 above. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0147] In the above embodiments, the descriptions of each embodiment have their own emphases. For parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0148] The proposed system can be implemented in other ways. For example, the above-described system embodiments are merely illustrative. For example, the above module division is only a logical function division. In actual implementation, there can be other division methods. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0149] Embodiment 3

[0150] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs. Among them, the processor is connected to the memory, and the above one or more computer programs are stored in the memory. When the electronic device runs, the processor executes the one or more computer programs stored in the memory so that the electronic device executes the method described in Embodiment 1 above.

[0151] It should be understood that in this embodiment, the processor can be a central processing unit CPU, and the processor can also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0152] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A part of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0153] In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor or the instructions in the form of software.

[0154] The method in the first embodiment can be directly embodied as being executed by the hardware processor, or completed by the combination of the hardware and software modules in the processor. The software module can be located in mature storage media in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. This storage media is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0155] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with this embodiment can be implemented by electronic hardware or the combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but this implementation should not be considered to exceed the scope of the present invention.

[0156] Embodiment 4

[0157] This embodiment also provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by the processor, the method described in the first embodiment is completed.

[0158] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A video behavior recognition method based on a dual-view spatio-temporal interaction network, characterized in that Including: Projecting the depth video sequence of video human behaviors in the front view and the side view to obtain a depth projection map sequence in the front view and the side view; Preprocessing the depth projection map sequence in the front view and the side view to obtain a depth dynamic image sequence in the front view and the side view; Inputting the depth dynamic image sequence in the front view and the side view into a dual-view spatio-temporal interaction network to obtain a video behavior recognition result; Among them, the dual-view spatio-temporal interaction network replaces all the standard residual blocks in the second, third, fourth, and fifth stages of the two-dimensional Resnet-50 network with dual-view spatio-temporal interaction blocks; the dual-view spatio-temporal interaction block is obtained by embedding a dual-view spatio-temporal interaction unit after the first two-dimensional convolutional layer and before the second two-dimensional convolutional layer of the standard residual block in the two-dimensional Resnet-50 network; The dual-view spatio-temporal interaction unit first performs inter-view channel feature exchange and then performs intra-view temporal feature displacement to achieve in-depth interaction of spatio-temporal information in the front view and the side view; among them, the interacted front and side convolutional feature maps output by the last dual-view spatio-temporal interaction block are respectively subjected to average pooling operations along the spatial and temporal dimensions to generate feature representations in the front view and the side view; the feature representations in the front view and the side view are concatenated to obtain a video global feature representation, and the video global feature representation is input into a fully connected layer to obtain a behavior recognition result.

2. The video behavior recognition method based on the dual-view spatio-temporal interaction network according to claim 1, characterized in that, Preprocessing the depth projection map sequence in the front view and the side view to obtain a depth dynamic image sequence in the front view and the side view, specifically including: uniformly segmenting the depth projection map sequence in the front view and the side view; performing sorting pooling on the uniform segmentation result to obtain a front and side depth dynamic image sequence.

3. The video behavior recognition method based on the dual-view spatio-temporal interaction network according to claim 1, characterized in that, Inputting the depth dynamic image sequence in the front view and the side view into a dual-view spatio-temporal interaction network to obtain a video behavior recognition result, where the training process of the dual-view spatio-temporal interaction network includes: Constructing a training set and a test set, both of which are depth videos with known video behavior recognition results; constructing the first, second, and third dual-view spatio-temporal interaction networks; the first, second, and third dual-view spatio-temporal interaction networks include the same parts and different parts; Among them, the same parts refer to: the first, second, and third dual-view spatio-temporal interaction networks all include: replacing all the standard residual blocks in the second, third, fourth, and fifth stages of the two-dimensional Resnet-50 network with dual-view spatio-temporal interaction blocks; the dual-view spatio-temporal interaction block is obtained by embedding a dual-view spatio-temporal interaction unit after the first two-dimensional convolutional layer and before the second two-dimensional convolutional layer of the standard residual block in the two-dimensional Resnet-50 network; Among them, the different parts refer to: the dual-view spatio-temporal interaction unit of the first dual-view spatio-temporal interaction network, including: a channel exchange subunit and a temporal displacement subunit connected in sequence; the dual-view spatio-temporal interaction unit of the second dual-view spatio-temporal interaction network, including: an average fusion-channel exchange subunit and a temporal displacement subunit connected in sequence; the dual-view spatio-temporal interaction unit of the third dual-view spatio-temporal interaction network, including: a correlation score view interaction subunit and a temporal displacement subunit connected in sequence. Input the training set into the first, second, and third dual-view spatio-temporal interaction networks respectively, and train the three networks separately. When the loss function value of each network no longer decreases, or the number of iterations reaches the set number, stop training to obtain the trained first, second, and third dual-view spatio-temporal interaction networks. Input the test set into the first, second, and third dual-view spatio-temporal interaction networks, and select the dual-view spatio-temporal interaction network with the highest test accuracy as the dual-view spatio-temporal interaction network.

4. The video behavior recognition method based on the dual-view spatio-temporal interaction network according to claim 3, characterized in that, The first, second, and third dual-view spatio-temporal interaction networks all include: a first-stage Conv1, a second-stage Conv2_x, a third-stage Conv3_x, a fourth-stage Conv4_x, a fifth-stage Conv5_x, a global average pooling layer, and a fully connected layer; the first-stage Conv1 is a two-dimensional convolutional layer; the second-stage Conv2_x includes: a max pooling layer and three dual-view spatio-temporal interaction blocks connected in sequence; the third-stage Conv3_x includes: four dual-view spatio-temporal interaction blocks connected in sequence; the fourth-stage Conv4_x includes: six dual-view spatio-temporal interaction blocks connected in sequence; the fifth-stage Conv5_x includes: three dual-view spatio-temporal interaction blocks connected in sequence.

5. The video behavior recognition method based on a dual-view spatio-temporal interaction network according to any one of claims 1, 3, or 4, characterized in that, The dual-view spatio-temporal interaction block includes: a 1×1 two-dimensional convolutional layer, a dual-view spatio-temporal interaction unit, a 3×3 two-dimensional convolutional layer, a 1×1 two-dimensional convolutional layer, and an adder connected in sequence; the input end of the first 1×1 two-dimensional convolutional layer is connected to the input end of the adder; the input end of the first 1×1 two-dimensional convolutional layer is the input end of the dual-view spatio-temporal interaction block; the output end of the adder is the output end of the dual-view spatio-temporal interaction block.

6. The video behavior recognition method based on the dual-view spatio-temporal interaction network according to claim 3, characterized in that, The channel exchange sub-unit includes: assuming that the forward and lateral convolutional feature maps are respectively represented as ConvF ∈ R T×C×H×W and ConvS ∈ R T×C×H×W , channel exchange is to swap the feature channels with a set ratio P along the channel dimension C of the feature maps ConvF and ConvS, and the remaining channel features remain unchanged; The average fusion-channel exchange sub-unit includes: first, the forward and lateral convolutional feature maps are fused by average operation; the generated fused feature map ConvAve ∈ R T×C×H×W respectively performs channel feature exchange with the forward feature map ConvF and the lateral feature map ConvS: ConvAve is evenly divided into two parts along the channel dimension, the first 1 / 2 channels are taken to replace the first 1 / 2 feature channels of ConvF, and the last 1 / 2 channels are taken to replace the last 1 / 2 feature channels of ConvS, and the remaining channels of ConvF and ConvS keep the original feature information unchanged.

7. The video behavior recognition method based on the dual-view spatio-temporal interaction network according to claim 3, characterized in that The correlation score perspective interaction subunit includes: realizing the interaction and fusion of the two-way perspective by calculating the correlation between the forward feature and the lateral feature at each moment: for each moment, first calculate the correlation score matrix S of the forward spatial feature and the lateral spatial feature of each channel along the channel dimension; wherein, and respectively represent the spatial feature vectors of the k-th channel at the t-th moment in the forward and lateral directions, t ∈ {1, 2... T}, k ∈ {1, 2... C}, the symbol ⊙ represents the vector dot product operation, S ∈ R T×C represents the correlation score matrix; the correlation score matrix S is multiplied by the lateral convolutional feature map ConvS along the spatial dimension, and then added to the forward convolutional feature map ConvF to obtain the feature information after the forward and lateral interaction; similarly, S is multiplied by the forward convolutional feature map ConvF along the spatial dimension, and then added to the lateral convolutional feature map ConvS to obtain the feature information after the lateral and forward interaction; The temporal displacement subunit includes: moving the interacted forward feature map forward and backward along the time dimension by a number of feature channels; moving the interacted lateral feature map forward and backward along the time dimension by a number of feature channels.

8. The video behavior recognition system based on the dual-view spatio-temporal interaction network is characterized in that Include: A projection module configured to project the depth video sequence of the video human behavior in the forward view and the lateral view to obtain a depth projection map sequence in the forward view and the lateral view. ​ ​ Among them, the dual-view spatio-temporal interaction network replaces all standard residual blocks in the second, third, fourth, and fifth stages of the two-dimensional Resnet-50 network with dual-view spatio-temporal interaction blocks; the dual-view spatio-temporal interaction block is obtained by embedding a dual-view spatio-temporal interaction unit after the first two-dimensional convolutional layer and before the second two-dimensional convolutional layer of the standard residual block in the two-dimensional Resnet-50 network; The dual-view spatio-temporal interaction unit first performs cross-view channel feature exchange and then performs intra-view temporal feature displacement to achieve in-depth interaction of spatio-temporal information in the forward view and the lateral view; among them, the interacted forward and lateral convolutional feature maps output by the last dual-view spatio-temporal interaction block are respectively subjected to average pooling operations along the spatial and temporal dimensions to generate feature representations of the forward view and the lateral view; the feature representations of the forward view and the lateral view are concatenated to obtain a video global feature representation, and the video global feature representation is input into a fully connected layer to obtain a behavior recognition result.

9. An electronic device, characterized in that it includes: A memory for non-temporarily storing computer-readable instructions; And A processor for running the computer-readable instructions, wherein when the computer-readable instructions are run by the processor, the method according to any one of claims 1-7 above is executed.

10. A storage medium, characterized in that, Non-temporarily storing computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the instructions for executing the method according to any one of claims 1-7 are executed.

Images