Human behavior recognition method based on two-stream three-dimensional convolutional neural network

Abstract

The present invention proposes a method for human behavior recognition based on a dual-stream three-dimensional convolutional neural network, which mainly solves the problems of limited feature representation caused by single-modal data and high computational complexity of cross-modal methods in the existing technology. It includes: 1) constructing a multimodal dataset; 2) constructing a separable three-dimensional residual module for decomposing the three-dimensional convolution into time and space dimensions; 3) constructing a dynamic similar frame aggregation module for learning the similarity of feature maps, grouping similar feature maps, and aggregating grouped feature maps; 4) using the modules in steps 2) and 3) to construct a skeleton stream network and an RGB stream network; 5) introducing a feature fusion module between the two stream networks, building a dual-stream three-dimensional convolutional neural network and training it; 6) inputting the behavior video into the trained network to obtain recognition results. The present invention can fully extract behavior features, reduce information redundancy and noise interference, and effectively improve the accuracy and performance of human behavior recognition.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology and further relates to human behavior recognition. Specifically, it is a human behavior recognition method based on a dual-stream three-dimensional convolutional neural network, which can be used in intelligent security, human-computer interaction, auxiliary medical care and other fields. Background Art

[0002] The primary goal of human action recognition is to extract valid action information from given video data and determine the action category. With the rise of deep learning technology, current action recognition methods based on deep learning have become a research hotspot. Due to the complexity of video data, various deep learning-based action recognition methods have varying performance.

[0003] According to different input data modalities, these methods are mainly divided into RGB video-based behavior recognition methods, human skeleton-based behavior recognition methods, and multimodal multi-stream-based behavior recognition methods.

[0004] RGB data is the most common data type used in human action recognition research. It contains rich appearance information that captures the scene context. However, action recognition from RGB data is often challenging due to variations in background, viewpoint, human scale, and lighting conditions. For example, in 2020, Jun Li et al. published an article titled "Spatio-temporal deformable 3D ConvNets with attention for action recognition" in Pattern Recognition. This paper describes a spatio-temporal deformable convolutional network method with an attention mechanism. This method considers intercorrelations in both the temporal and spatial domains, effectively capturing long-range and long-distance dependencies in video actions. This allows the attention-based deformable module to adaptively learn more accurate spatiotemporal offsets to model the irregularities of the action. However, this method uses only RGB data as input and is easily affected by factors such as lighting, occlusion, and motion speed, failing to capture sufficient detailed information, thus affecting the accuracy of action recognition.

[0005] Skeleton sequences encode the trajectories of human joints, representing information-rich human motion. Using skeleton data in action recognition has many advantages because it provides information about the structure and posture of the human body and is essentially a concise information representation. Skeleton data is scale-invariant and robust to clothing texture and background changes. For example, in 2022, Hyung-gun Chi et al. published an article titled "InfoGCN: Representation Learning for Human Skeleton-Based Action Recognition" in the "Conference on Computer Vision and Pattern Recognition". They disclosed an action recognition learning method that combines a new learning objective and encoding. The method includes an information bottleneck objective to learn action information representation as much as possible, and an attention-based graph convolutional network to infer context-related skeleton topology. However, this method does not take into account the lack of detailed features related to action recognition in skeleton sequences. Therefore, this method cannot effectively obtain semantic information related to the action category in the scene and scene change information outside the human body, resulting in low improvement in action recognition accuracy.

[0006] Different data modalities have their own advantages, and many studies have explored deep learning architectures that fuse RGB video and skeleton sequence data for action recognition. For example, Yanhao Jing et al. published an article titled "TP-VIT: A Two-Pathway Vision Transformer for Video Action Recognition" in the IEEE International Conference on Acoustics, Speech and Signal Processing in 2022, disclosing a dual-path visual Transformer method for fusing RGB and skeleton data, which includes a high-resolution path focusing on spatial information and a high-frame rate path emphasizing temporal information. However, this method consumes a lot of computing resources, and processing a single clip of RGB video requires more than 100GFLOPs. Therefore, cross-modal methods face a dilemma between accuracy and computational efficiency. Summary of the Invention

[0007] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a human behavior recognition method based on a dual-stream three-dimensional convolutional neural network to solve the problems in the prior art of limited feature representation due to single-modal data and high computational complexity of cross-modal methods. When constructing the human behavior recognition model structure, the present invention first constructs the skeleton flow network and RGB flow network model structures, and then uses multimodal data as the input of the model to enhance the complementary features between different modalities; a dynamic similar frame aggregation module is constructed in the dual-stream network to eliminate the redundancy of similar frames in the input features. This effectively improves the recognition accuracy.

[0008] The present invention achieves the above-mentioned purpose by the following specific steps:

[0009] (1) Constructing a multimodal dataset:

[0010] (1.1) Select V behavior videos;

[0011] (1.2) Preprocess the V action videos selected in step 1.1 using the posture extraction network HRNet to obtain a set of frame skeleton sequences; let the coordinates of the skeleton key points in the u-th frame skeleton sequence be (x k ,y k ,c k ), where (x k ,y k ) is the plane two-dimensional coordinate of the point, c k Indicates the score of the heat map at this point;

[0012] (1.3) Calculate the intensity value J of the kth key point at the pixel position (i, j) on the heat map according to the following formula: kij , get the heat map of the kth key point:

[0013]

[0014] Where σ represents the variance of the Gaussian function, e is a natural constant; k = 1, 2, ..., K, K is the total number of key points;

[0015] (1.4) Combine the heat maps of all key points to obtain the single-frame joint heat map J of the u-th frame skeleton sequence m , the single-frame joint heat map obtained from each sequence in the frame skeleton sequence set is stacked along the time dimension to obtain a 3D heat map stack with a size of K×T×H×W, where T is the total number of frames, H is the height of the heat map, and W is the width of the heat map; finally, the 3D heat map stacks obtained from processing V behavioral videos are combined into a complete skeleton dataset;

[0016] (1.5) Preprocess the V behavioral videos selected in step 1.1 to a uniform size, adjust the number of video channels to 8 and the pixel size to 224×224, and obtain an RGB video frame dataset;

[0017] (1.6) Combining the skeleton dataset and the RGB video frame dataset into a multimodal dataset, and obtaining a label set corresponding to the dataset; partitioning the multimodal dataset into a training sample set, a validation sample set, and a test sample set in a 6:2:2 ratio;

[0018] (2) Constructing a separable 3D residual module to decompose the 3D convolution in the 3D residual network into a 1D convolution in the time dimension and a 2D convolution in the spatial dimension; then using three, four, six and three separable 3D residual modules to form four residual units D1, D2, D3 and D4 respectively;

[0019] (3) Construct a dynamic similar frame aggregation module to learn the similarity of feature maps, group similar feature maps, and aggregate grouped feature maps;

[0020] (4) Constructing skeleton flow network and RGB flow network:

[0021] Construct a skeleton flow network with the following structure: 2D convolution block → D2 → D3 → dynamic similar frame aggregation module → D4 → pooling layer → fully connected layer; simultaneously construct an RGB flow network with the following structure: 2D convolution block → pooling layer → D1 → D2 → D3 → dynamic similar frame aggregation module → D4 → pooling layer → fully connected layer;

[0022] (5) Constructing feature fusion module:

[0023] Constructing two early feature fusion modules and one late feature fusion module, wherein the early feature fusion module includes a three-dimensional convolution, a normalization layer and a ReLU activation function; the late feature fusion module is used to fuse classification probabilities;

[0024] (6) Building a two-stream three-dimensional convolutional neural network:

[0025] With the skeleton flow network and RGB flow network as the main body, an early feature fusion module is connected after D2 of the skeleton flow network and the RGB flow network respectively, and another early feature fusion module is connected after the skeleton dynamic similarity frame aggregation module of the skeleton flow network and the RGB flow network respectively. Finally, the late feature fusion module is connected after the fully connected layer of the skeleton flow network and the fully connected layer of the RGB network to realize the construction of a two-stream 3D convolutional neural network;

[0026] (7) Input the training sample set data into the constructed two-stream three-dimensional convolutional neural network, and use the optimizer to iteratively update the weights of the network until the network loss function converges to obtain a trained neural network;

[0027] (8) Input the data in the test sample set into the trained neural network to extract and fuse behavioral features to obtain the final recognition result.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] First, because the present invention adopts a two-stream three-dimensional convolutional neural network to extract features from RGB video and skeleton data respectively, the network model can better obtain spatial structure information and appearance information, thereby improving the accuracy of human behavior recognition;

[0030] Second, the present invention constructs a dynamic similar frame aggregation module to reduce the redundancy and noise interference of input information. This module groups features in the time dimension and then aggregates redundant features, thereby improving the accuracy of human behavior recognition while reducing the computational complexity of the model.

[0031] Third, the present invention adopts a multimodal feature fusion strategy and constructs a feature fusion module. Through early bidirectional feature fusion and late decision-level feature fusion methods, it fuses information from different modalities, thereby realizing feature complementarity and achieving the effect of feature enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart for implementing the present invention.

[0033] Figure 2 This is a schematic diagram of the implementation process of dynamic similar frame aggregation and feature fusion using the skeleton flow network and RGB flow network in the present invention. DETAILED DESCRIPTION

[0034] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] Example 1: Reference Figure 1 The present invention proposes a method for human behavior recognition based on a two-stream three-dimensional convolutional neural network, which specifically includes the following steps:

[0036] Step 1. Build a multimodal dataset:

[0037] (1.1) Select V behavior videos;

[0038] (1.2) Preprocess the V action videos selected in step 1.1 using the posture extraction network HRNet to obtain a set of frame skeleton sequences; let the coordinates of the skeleton key points in the u-th frame skeleton sequence be (x k ,y k , c k )where (x k ,y k ) is the plane two-dimensional coordinate of the point, c kIndicates the score of the heat map at this point;

[0039] (1.3) Calculate the intensity value J of the kth key point at the pixel position (i, j) on the heat map according to the following formula: kij , get the heat map of the kth key point:

[0040]

[0041] Where σ represents the variance of the Gaussian function, e is a natural constant; k = 1, 2, ..., K, K is the total number of key points;

[0042] (1.4) Combine the heat maps of all key points to obtain the single-frame joint heat map J of the u-th frame skeleton sequence m , the single-frame joint heat map obtained from each sequence in the frame skeleton sequence set is stacked along the time dimension to obtain a 3D heat map stack with a size of K×T×H×W, where T is the total number of frames, H is the height of the heat map, and W is the width of the heat map; finally, the 3D heat map stacks obtained from processing V behavioral videos are combined into a complete skeleton dataset;

[0043] (1.5) Preprocess the V behavioral videos selected in step 1.1 to a uniform size, adjust the number of video channels to 8 and the pixel size to 224×224, and obtain an RGB video frame dataset;

[0044] (1.6) The skeleton dataset and the RGB video frame dataset are combined into a multimodal dataset, and a label set corresponding to the dataset is obtained; the multimodal dataset is divided into a training sample set, a validation sample set, and a test sample set in a ratio of 6:2:2. Specifically, in this embodiment, 60% of the multimodal dataset and the label set are randomly selected as the training sample set, 20% of the remaining samples are randomly selected as the validation sample set, and the remaining 20% ​​are used as the test sample set.

[0045] Step 2. Construct a separable 3D residual module to decompose the 3D convolution in the 3D residual network into a 1D convolution in the time dimension and a 2D convolution in the spatial dimension; then use three, four, six and three separable 3D residual modules to form four residual units D1, D2, D3 and D4 respectively.

[0046] The three-dimensional convolution refers to N i The dimension is N i-1 ×t×d×d three-dimensional convolution kernel; the separable three-dimensional residual module is specifically composed of M i Two-dimensional spatial convolution kernel and N i The one-dimensional time dimension convolution kernel is composed of two-dimensional spatial dimension convolution kernel, the dimension of which is N i-1 ×1×d×d,N iThe dimension of the one-dimensional time-dimensional convolution kernel is M i-1 ×t×1×1, where N i-1 is the number of convolution kernels in the previous layer, t is the time dimension of the input, and d is the width and height of the input feature map.

[0047] Step 3. Build a dynamic similar frame aggregation module to learn the similarity of feature maps, group similar feature maps, and aggregate grouped feature maps;

[0048] The similarity of the learning feature graph is specifically achieved by building a similarity network f that performs a mapping operation s , using V t =f s (X t ) Input feature X t Mapped into a vector V t ,in is the input feature map of the t-th frame, C is the number of channels, H t is the height, W t is the width; the similar network f s The structure is as follows: global pooling layer → first one-dimensional convolution layer → ReLU activation function → second one-dimensional convolution layer → normalization layer, where the convolution kernel size of the first one-dimensional convolution layer and the second one-dimensional convolution layer are both 1, the stride is 1, and the output channel sizes are C and L respectively.

[0049] The method of grouping similar feature maps is to group the input feature map X t According to the mapping vector V t The size of Δ t Find the nearest similarity interval center and complete the grouping; the implementation is as follows:

[0050] The size Δ of each feature map mapping vector is calculated according to the following formula t , and the half-width δ of each similarity interval:

[0051] Δ t =||V t ||,

[0052] Δ max =max(Δ1,...,Δ T ),

[0053]

[0054] Among them, Δ max is the maximum amplitude, F is the number of similarity intervals;

[0055] Calculate the estimated center α of each similarity interval f , group similar feature maps:

[0056]

[0057] The grouped feature maps are aggregated, specifically by sampling and adding spatially similar feature maps belonging to the same similar interval, to achieve joint grouping and aggregation of all feature maps; let represents the aggregated features of each interval b, using a general differentiable sampling kernel Ψ(·,α f ), the feature O after the similarity interval f is aggregated f The calculation formula is as follows:

[0058]

[0059] Construct a GELU function to sample the feature map within the interval. The calculation expression is as follows:

[0060]

[0061] Step 4. Build the skeleton flow network and RGB flow network:

[0062] Construct a skeleton flow network with the following structure: 2D convolution block → D2 → D3 → dynamic similar frame aggregation module → D4 → pooling layer → fully connected layer; at the same time, construct an RGB flow network with the following structure: 2D convolution block → pooling layer → D1 → D2 → D3 → dynamic similar frame aggregation module → D4 → pooling layer → fully connected layer.

[0063] In this embodiment, the two-dimensional convolution block in the above-mentioned skeleton flow network consists of 16 convolution kernels; D2, D3 and D4 are each composed of three convolution layers, and the first two convolution layers in D2 are each composed of 16 convolution kernels, and the third convolution layer is composed of 64 convolution kernels; the first two convolution layers in D3 are each composed of 32 convolution kernels, and the third convolution layer is composed of 128 convolution kernels; the first two convolution layers in D4 are each composed of 64 convolution kernels, and the third convolution layer is composed of 256 convolution kernels; the time step of all stages is 1, and the spatial step is 2×2.

[0064] The two-dimensional convolution block in the above RGB network consists of 32 convolution kernels; D1, D2, D3 and D4 are composed of three convolution layers, and the first two convolution layers in D1 are composed of 32 convolution kernels, and the third convolution layer is composed of 128 convolution kernels; the first two convolution layers in D2 are composed of 64 convolution kernels, and the third convolution layer is composed of 256 convolution kernels; the first two convolution layers in D3 are composed of 128 convolution kernels, and the third convolution layer is composed of 512 convolution kernels; the first two convolution layers in D4 are composed of 256 convolution kernels, and the third convolution layer is composed of 1024 convolution kernels; the time step of all stages is 1, and the spatial step is 2×2.

[0065] The above pooling layers all use maximum pooling, and the sliding window size is 1×3 2 , the time step is 1 and the space step is 2×2.

[0066] Step 5. Build feature fusion module:

[0067] Two early feature fusion modules and one late feature fusion module are constructed. The early feature fusion module includes a three-dimensional convolution, a normalization layer and a ReLU activation function; the late feature fusion module is used to fuse classification probabilities.

[0068] The above-mentioned early feature fusion module includes a three-dimensional convolution, a normalization layer and a ReLU activation function, wherein the convolution kernel size of the three-dimensional convolution is 3×3 2 , the time step is 1, and the space step is 2×2; the late feature fusion module is used to fuse the classification probability, which is achieved by calculating the mean classification probability of the skeleton flow network and the RGB flow network.

[0069] Step 6. Build a two-stream three-dimensional convolutional neural network:

[0070] With the skeleton flow network and RGB flow network as the main body, an early feature fusion module is connected after D2 of the skeleton flow network and the RGB flow network respectively, and another early feature fusion module is connected after the skeleton dynamic similarity frame aggregation module of the skeleton flow network and the RGB flow network respectively. Finally, the late feature fusion module is connected after the fully connected layer of the skeleton flow network and the fully connected layer of the RGB network to realize the construction of a two-stream 3D convolutional neural network;

[0071] Step 7. Input the training sample set data into the constructed two-stream three-dimensional convolutional neural network, and use the optimizer to iteratively update the weights of the network until the network loss function converges, thereby obtaining a trained neural network. In this embodiment, the optimizer uses the SGD stochastic gradient descent algorithm, and the network loss function is the cross entropy loss function.

[0072] Step 8. Input the data in the test sample set into the trained neural network to extract and fuse behavioral features to obtain the final recognition result.

[0073] Example 2: Reference Figure 1-2 The overall implementation steps of the human behavior recognition method proposed in this embodiment are the same as those in the first embodiment. Now, a specific example is given to further describe the implementation process of the present invention in detail:

[0074] Step 1. Build a multimodal dataset:

[0075] This example uses the NTU RGB+D 120 dataset to obtain V behavioral videos. Alternatively, you can create your own video dataset and select from it, or search for the required video data online. Two preprocessing methods are used on the selected V behavioral videos to obtain a skeleton dataset and an RGB video frame dataset, respectively, as follows:

[0076] 1) Preprocess the selected V action videos using the posture extraction network HRNet to obtain a frame skeleton sequence set; let the coordinates of the skeleton key points in the u-th frame skeleton sequence be (x k ,y k ,c k )where (x k ,y k ) is the plane two-dimensional coordinate of the point, c k Represents the score of the heat map at this point; the intensity value J of the k-th key point on the heat map at the pixel position (i, j) is calculated according to the following formula kij , get the heat map of the kth key point:

[0077]

[0078] Where σ represents the variance of the Gaussian function, e is a natural constant; k = 1, 2, ..., K, K is the total number of key points; the heat maps of all key points are combined to obtain the single-frame joint heat map J of the u-th frame skeleton sequence m , the single-frame joint heat map obtained from each sequence in the frame skeleton sequence set is stacked along the time dimension to obtain a 3D heat map stack with a size of K×T×H×W, where T is the total number of frames, H is the height of the heat map, and W is the width of the heat map; finally, the 3D heat map stacks obtained from processing V behavioral videos are combined into a complete skeleton dataset;

[0079] 2) Preprocess the selected V behavioral videos to a uniform size, adjust the number of video channels to 8 and the pixel size to 224×224, and obtain an RGB video frame dataset;

[0080] Finally, the skeleton dataset and the RGB video frame dataset are combined into a multimodal data set, and the label set corresponding to the set is obtained; 60% are randomly selected from the multimodal data set and the label set as the training sample set, and 20% are randomly selected from the remaining samples as the verification sample set, and finally the remaining 20% ​​are used as the test sample set.

[0081] Step 2. Build a separable 3D residual module:

[0082] Build a one-dimensional time partial convolution and a two-dimensional space partial convolution, and decompose the three-dimensional convolution in the three-dimensional residual network into a one-dimensional convolution in the time dimension and a two-dimensional convolution in the space dimension. In this embodiment, the separable convolution is achieved by decomposing the three-dimensional convolution into a one-dimensional time convolution and a two-dimensional space convolution. The three-dimensional convolution refers to N i The dimension is N i-1 ×t×d×d three-dimensional convolution kernel, the separable convolution block in the separable three-dimensional residual module refers to the one composed of M i Two-dimensional convolution kernels and N i The size of the two-dimensional convolution kernel is N. i-1 ×1×d×d,N i The dimension of the one-dimensional time-dimensional convolution kernel is M i-1 ×t×1×1, where N i-1 is the number of convolution kernels in the previous layer, M i is the number of two-dimensional convolution kernels, which determines the subspace dimension size of the spatiotemporal signal mapping, t is the time dimension of the input, and d is the width and height of the input feature map.

[0083] Step 3. Build a dynamic similar frame aggregation module:

[0084] Constructing a dynamic similar frame aggregation module includes learning the similarity of feature maps, grouping similar feature maps, and aggregating the grouped feature maps. In this embodiment, in order to eliminate the redundancy of similar frame features and dynamically extract key frames containing behavioral information based on different input videos, a dynamic similar frame aggregation module is constructed. This module is a differentiable module that samples spatially similar feature maps of the current input features in the time dimension and aggregates these spatially similar redundant feature maps into one feature map. Since the number of output feature maps is lower than the number of input feature maps, redundant information is removed. The number of feature maps output by the dynamic similar frame aggregation module is not fixed, but varies according to different input videos. If there is no redundancy in the input feature map, no information in the time dimension is deleted.

[0085] The similarity of learning feature maps in the dynamic similar frame aggregation module refers to building a similarity network f that performs mapping operations. s , a global pooling layer, a one-dimensional convolution layer, a ReLU activation function, a one-dimensional convolution layer and a normalization layer are sequentially connected. The two one-dimensional convolution layers have a convolution kernel size of 1, a stride of 1, and output channel sizes of C and L respectively. V t =f s (X t ) Input feature X t Mapped into a vector V t ,in is the input feature map of the t-th frame, C is the number of channels, Ht is the height, W t The similarity space in this structure is an L-dimensional vector space. The purpose of the pooling operation is to extract features from the spatial dimension of the feature map while maintaining the temporal dimension of the feature map. The convolution operation of the two one-dimensional convolutional layers further extracts features and reduces the feature map to L dimensions. The normalization layer unifies the data, reduces unnecessary differences, and improves the training efficiency of the neural network.

[0086] The grouping of similar feature maps in the dynamic similar frame aggregation module refers to the input feature map X t According to the mapping vector V t The size of Δ t Find the nearest similarity interval center and complete the grouping of all feature maps. The specific steps are: t Grouping, using Δ t =||V t ||Calculate the size of the vector Δ mapped by each feature map t , calculate the half-width δ of each similarity interval:

[0087] Δ max =max(Δ1,...,Δ T ),

[0088]

[0089] Among them, Δ max is the maximum amplitude, F is the number of similarity intervals, if Δ t and Δ t′ The value of is in a similar range, then the feature map X t and X t′ are considered similar, the number of similar intervals is set to the same as the size of the time dimension so that no information is lost if there is no redundancy between the feature maps of all current frames; the estimated center α of each interval is calculated f :

[0090]

[0091] Input feature map X t According to the size of the vector mapped by each feature map Δ t Find the nearest similar interval center to group all feature maps.

[0092] Aggregating the grouped feature maps in the dynamic similarity frame aggregation module refers to sampling and adding the spatially similar feature maps belonging to the same similar interval to achieve joint grouping and aggregation of all feature maps. represents the aggregated features of each interval b, using a general differentiable sampling kernel Ψ(·,α f ), the feature O after the similarity interval f is aggregated f The calculation formula is as follows:

[0093]

[0094] Construct a GELU function to sample the feature maps within the interval, improve nonlinear transformation, and enhance the overall performance of feature map aggregation. The calculation method is as follows:

[0095]

[0096] Step 4. Build the skeleton flow network and RGB flow network:

[0097] like Figure 2 As shown, the skeleton flow network includes a sequentially connected two-dimensional convolution block Conv1, four separable three-dimensional residual modules Res2, six separable three-dimensional residual modules Res3, a dynamic similar frame aggregation module, three separable three-dimensional residual modules Res4, a pooling layer and a fully connected layer; the RGB flow network includes a sequentially connected two-dimensional convolution block Conv1', three separable three-dimensional residual modules Res1', a pooling layer, four separable three-dimensional residual modules Res2', six separable three-dimensional residual modules Res3', a dynamic similar frame aggregation module, three separable three-dimensional residual modules Res4', a pooling layer and a fully connected layer;

[0098] In this embodiment, the skeleton flow feature extraction network extracts human motion information from multiple frames of skeleton heat maps. The Conv1 layer in the skeleton flow network consists of 16 convolution kernels with a size of 1×7. 2 The Res2 module consists of three convolutional layers. The first convolutional layer consists of 16 convolutional layers of size 1×1. 2 The second convolution layer consists of 16 convolution kernels of size 1×3 2 The convolution layer consists of 64 convolution kernels C2, and the third convolution layer consists of 64 convolution kernels C1; the Res3 module consists of three layers of convolution kernels, the first convolution layer consists of 32 convolution kernels of size 3×1 2The first convolution layer consists of 64 convolution kernels C3, the second convolution layer consists of 32 convolution kernels C2, and the third convolution layer consists of 128 convolution kernels C1; the Res4 module consists of three layers of convolution kernels, the first convolution layer consists of 64 convolution kernels C3, the second convolution layer consists of 64 convolution kernels C2, and the third convolution layer consists of 256 convolution kernels C1; the time step of all stages is 1 and the spatial step is 2×2. In order to maintain the number of frames of the input skeleton heat map volume in the temporal dimension, the skeleton flow feature extraction network does not perform downsampling operations in the temporal dimension. Except for the global average pooling operation before the final Softmax layer classification, the temporal resolution characteristics in the video are maintained as much as possible.

[0099] The RGB stream feature extraction network captures the spatial appearance information of the input RGB video frames and extracts high-level semantic information related to the behavior category. The RGB stream network inputs fewer RGB video frames, which contain the human appearance behavior information missing in the skeleton sequence and the complex scene information in the human movement environment. The RGB stream network has more feature channels than the skeleton stream network. The Conv1' layer consists of 32 convolution kernels C0; the Res1' module consists of three convolution layers, the first convolution layer consists of 32 convolution kernels C1, the second convolution layer consists of 32 convolution kernels C2, and the third convolution layer consists of 128 convolution kernels C1; the Res2' module consists of three layers of convolution kernels, the first convolution layer consists of 64 convolution kernels C1, the second convolution layer consists of 64 convolution kernels C2, and the third convolution layer consists of 128 convolution kernels C1. The second convolution layer consists of 64 convolution kernels C2, and the third convolution layer consists of 256 convolution kernels C1; the Res3' module consists of three layers of convolution kernels, the first convolution layer consists of 128 convolution kernels C3, the second convolution layer consists of 128 convolution kernels C2, and the third convolution layer consists of 512 convolution kernels C1; the Res4' module consists of three layers of convolution kernels, the first convolution layer consists of 256 convolution kernels C3, the second convolution layer consists of 256 convolution kernels C2, and the third convolution layer consists of 1024 convolution kernels C1; the time step of all stages is 1, and the spatial step is 2×2. The RGB stream network does not downsample the information in the time dimension, but uses the dynamic similar frame aggregation module to process the redundant information in the time dimension;

[0100] The pooling layers of the two sub-networks use maximum pooling, and the sliding window size is 1×3 2 , the time step is 1 and the space step is 2×2;

[0101] Step 5. Build feature fusion module:

[0102] like Figure 2As shown in the figure, two early feature fusion modules and one late feature fusion module are constructed between the skeleton stream network and the RGB stream network, which are sequentially placed after the separable 3D residual modules Res3 and Res3' of the two-stream network, after the dynamic similar frame aggregation module and after the fully connected layer. The early feature fusion module includes a 3D convolution, a normalization layer and a ReLU activation function.

[0103] In this embodiment, the early feature fusion method performs multiple bidirectional side connections on the features extracted by the residual blocks of the skeleton stream network and the RGB stream network, respectively, to fuse the features extracted by one network with the features extracted by the other network, and then feeds the fused features together with the information of the other data modality into the next convolution layer for further feature extraction. Finally, after extracting features through the dual-stream model, behavior classification is performed using the late decision-level feature fusion. The two early feature fusion modules of this feature fusion module have the same structure, and the size of the three-dimensional convolution kernel is 3×3 2 The time step is 1 and the spatial step is 2×2. The late fusion module obtains the feature fusion result by calculating the classification probability of the skeleton flow network and the RGB flow network. The early and late fusion operations enable the RGB flow network to integrate the effective human motion expression extracted from the skeleton data, reduce the noise interference caused by irrelevant information in the environment, and make full use of the advantages of the two modal data.

[0104] Step 6. Train the two-stream 3D convolutional neural network:

[0105] Input the training set in step 1 into the two-stream 3D convolutional neural network, and use the optimizer to iteratively update the network weights until the network loss function converges to obtain a trained network.

[0106] In this embodiment, the initialization iteration number is t, the maximum iteration number is T, in this example t = 1, T = 30, the optimizer uses the SGD stochastic gradient descent algorithm, the network loss function is the cross entropy loss function, the momentum coefficient is set to 0.9, and the weight decay value is set to 1.0×10 -4 , the initial learning rate is set to 1.875×10 -3 , judge whether t≥T is established. If so, obtain the trained hyperspectral image classification network model. Otherwise, set t=t+1 and continue iterative training;

[0107] Step 7. Input the behavior video into the network to obtain the recognition result:

[0108] The behavior video is preprocessed in step 1 to obtain a test set of multimodal data. This set is then input into a trained two-stream 3D convolutional neural network for forward propagation. Behavioral features are extracted from the input RGB video frames and skeleton heat map stacks, respectively. The behavior recognition results of all test sets are obtained by fusing early and late features.

[0109] When constructing the behavior recognition model structure, the present invention uses a dual-stream three-dimensional convolutional neural network to fully extract the RGB video frame and skeleton information features, reduces information redundancy and noise interference through a dynamic similar frame aggregation module, and uses a feature fusion module to fuse the features of the dual-stream network in stages, effectively improving the accuracy and performance of the human behavior recognition method.

[0110] The following simulation experiments are used to further illustrate the technical effects of the present invention:

[0111] 1. Simulation conditions and contents:

[0112] The simulation experiments were conducted on a server with an Intel Xeon E5-2678 CPU, a GeForce GTX 3090Ti GPU, and 24GB of video memory. The operating system was Ubuntu 18.04, the deep learning framework was PyTorch, and the programming language was Python 3.6.

[0113] The input dataset used in the simulation experiments of this paper is the NTU RGB+D 120 dataset. NTU RGB+D 120 is a large-scale dataset for human action recognition, containing 114,000 video samples and 120 different action categories. The dataset contains the following five types of actions:

[0114] 1) 82 daily actions: drinking water, eating, brushing teeth, combing hair, etc.;

[0115] 2) 12 health-related actions: sneezing / coughing, stumbling, falling, touching the head, etc.;

[0116] 3) 26 interactive actions: hitting / patting, kicking, pushing, shaking hands, etc.

[0117] 60% of the dataset was randomly selected as the training sample set, 20% of the remaining data was randomly selected as the validation sample set, and the remaining 20% ​​was used as the test sample set. If the behavior recognition results in the simulation experiment of the present invention are the same as the labels of the videos in the test set of the NTURGB+D120 dataset, the video behavior recognition results are considered correct; if the detection results in the simulation experiment of the present invention are different from the labels of the videos in the test set of the NTURGB+D120 dataset, the video behavior recognition results are considered incorrect.

[0118] A comparative simulation was conducted between the present invention and an existing human action recognition method based on a dual-stream spatiotemporal graph convolutional network. This method was proposed by Bruce XBYu et al. in a paper titled "MMNet: A Model-Based Multimodal Network for Human Action Recognition in RGB-D Videos" published in the 2022 IEEE Transactions on Pattern Analysis and Machine Intelligence. To quantitatively compare the change detection results, the experiment used two human action recognition evaluation metrics: Top-1 accuracy and giga-floating-point operations per second (GFLOPs). A higher Top-1 accuracy indicates better human action recognition results, while a lower GFLOPs indicates lower computational complexity and better performance. The simulation results are shown in Table 1.

[0119] Table 1

[0120] method Top-1 Accuracy GFLOPs Existing technology 96.4％ 57.9 The present invention 98.2％ 11.7

[0121] 2. Analysis of simulation results:

[0122] As shown in Table 1, compared to existing human action recognition methods based on dual-stream spatiotemporal graph convolutional networks, the proposed method significantly improves Top-1 accuracy and significantly reduces GFLOPs computational complexity. This demonstrates that the proposed method can capture rich feature representations, improve human action recognition accuracy, and reduce the computational complexity of the model.

[0123] The above simulation experiments show that when constructing the behavior recognition model structure, the present invention uses a dual-stream three-dimensional convolutional neural network to fully extract the RGB video frame and skeleton information features, constructs a dynamic similar frame aggregation module to reduce information redundancy and noise interference, and uses a feature fusion module to fuse the features of the dual-stream network in stages. This solves the problem of the existing technology that only uses single-modal information and has high computational complexity. It is a human behavior recognition method based on a dual-stream three-dimensional convolutional neural network with excellent performance, and therefore has important practical significance.

[0124] The above simulation analysis proves the correctness and effectiveness of the method proposed in the present invention.

[0125] Parts of the present invention that are not described in detail belong to common knowledge among those skilled in the art.

[0126] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Obviously, for professionals in this field, after understanding the content and principles of the present invention, they may make various modifications and changes in form and details without departing from the principles and structure of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A human behavior recognition method based on a two-stream three-dimensional convolutional neural network, characterized in that: The steps include: (1) Constructing a multimodal dataset: (1.1) Select V behavior videos; (1.2) Preprocess the V action videos selected in step (1.1) using the posture extraction network HRNet to obtain a set of frame skeleton sequences; let the coordinates of the skeleton key points in the u-th frame skeleton sequence be (x k ,y k , c k ), where (x k ,y k ) is the plane two-dimensional coordinate of the point, c k Indicates the score of the heat map at this point; (1.3) Calculate the intensity value J of the kth key point at the pixel position (i, j) on the heat map according to the following formula: kij , get the heat map of the kth key point: Where σ represents the variance of the Gaussian function, e is a natural constant; k = 1, 2, ..., K, K is the total number of key points; (1.4) Combine the heat maps of all key points to obtain the single-frame joint heat map J of the u-th frame skeleton sequence m , the single-frame joint heat map obtained from each sequence in the frame skeleton sequence set is stacked along the time dimension to obtain a 3D heat map stack with a size of K×T×H×W, where T is the total number of frames, H is the height of the heat map, and W is the width of the heat map; finally, the 3D heat map stacks obtained from processing V behavioral videos are combined into a complete skeleton dataset; (1.5) Preprocess the V behavioral videos selected in step (1.1) to a uniform size, adjust the number of video channels to 8 and the pixel size to 224×224, and obtain an RGB video frame dataset; (1.6) Combining the skeleton dataset and the RGB video frame dataset into a multimodal dataset, and obtaining a label set corresponding to the dataset; partitioning the multimodal dataset into a training sample set, a validation sample set, and a test sample set in a ratio of 6:2:2; (2) Constructing a separable 3D residual module to decompose the 3D convolution in the 3D residual network into a 1D convolution in the time dimension and a 2D convolution in the spatial dimension; then using three, four, six and three separable 3D residual modules to form four residual units D1, D2, D3 and D4 respectively; (3) Construct a dynamic similar frame aggregation module to learn the similarity of feature maps, group similar feature maps, and aggregate grouped feature maps; (4) Constructing skeleton flow network and RGB flow network: Construct a skeleton flow network with the following structure: 2D convolution block → D2 → D3 → dynamic similar frame aggregation module → D4 → pooling layer → fully connected layer; simultaneously construct an RGB flow network with the following structure: 2D convolution block → pooling layer → D1 → D2 → D3 → dynamic similar frame aggregation module → D4 → pooling layer → fully connected layer; (5) Construct feature fusion module: Constructing two early feature fusion modules and one late feature fusion module, wherein the early feature fusion module includes a three-dimensional convolution, a normalization layer and a ReLU activation function; the late feature fusion module is used to fuse classification probabilities; (6) Building a two-stream three-dimensional convolutional neural network: With the skeleton flow network and RGB flow network as the main body, an early feature fusion module is connected after D2 of the skeleton flow network and the RGB flow network respectively, and another early feature fusion module is connected after the skeleton dynamic similarity frame aggregation module of the skeleton flow network and the RGB flow network respectively. Finally, the late feature fusion module is connected after the fully connected layer of the skeleton flow network and the fully connected layer of the RGB network to realize the construction of a two-stream 3D convolutional neural network; (7) Input the training sample set data into the constructed two-stream three-dimensional convolutional neural network, and use the optimizer to iteratively update the weights of the network until the network loss function converges to obtain a trained neural network; (8) Input the data in the test sample set into the trained neural network to extract and fuse behavioral features to obtain the final recognition result.

2. The method according to claim 1, wherein: The three-dimensional convolution in step (2) refers to N i The dimension is N i-1 ×t×d×d three-dimensional convolution kernel; the separable three-dimensional residual module is specifically composed of M i Two-dimensional spatial convolution kernel and N i The one-dimensional time dimension convolution kernel is composed of two-dimensional spatial dimension convolution kernel, the dimension of which is N i-1 ×1×d×d,N i The dimension of the one-dimensional time-dimensional convolution kernel is M i-1 ×t×1×1, where N i-1 is the number of convolution kernels in the previous layer, t is the time dimension of the input, and d is the width and height of the input feature map.

3. The method according to claim 1, wherein: The similarity of the learning feature graphs in step (3) is specifically achieved by building a similarity network f that performs the mapping operation. s , using V t =f s (X t ) Input feature X t Mapped into a vector V t ,in is the input feature map of the t-th frame, C is the number of channels, H i is the height, W t is the width; the similar network f s The structure is as follows: global pooling layer → first one-dimensional convolution layer → ReLU activation function → second one-dimensional convolution layer → normalization layer, where the convolution kernel size of the first one-dimensional convolution layer and the second one-dimensional convolution layer are both 1, the stride is 1, and the output channel sizes are C and L respectively.

4. The method according to claim 3, wherein: The similar feature maps are grouped as described in step (3), specifically by dividing the input feature map X t According to the mapping vector V t The size of Δ t Find the nearest similarity interval center and complete the grouping; the implementation is as follows: The size Δ of each feature map mapping vector is calculated according to the following formula t , and the half-width δ of each similarity interval: D t =||V t ||, D max =max(Δ1,...,Δ T ), Among them, Δ max is the maximum amplitude, F is the number of similarity intervals; Calculate the estimated center α of each similarity interval f , group similar feature maps:

5. The method according to claim 4, characterized in that: The grouped feature maps described in step (3) are aggregated, specifically by sampling and adding the spatially similar feature maps belonging to the same similar interval to achieve joint grouping and aggregation of all feature maps; let represents the aggregated features of each interval b, using a general differentiable sampling kernel Ψ(·,α f ), the feature O after the similarity interval f is aggregated f The calculation formula is as follows: Construct a GELU function to sample the feature map within the interval. The calculation expression is as follows:

6. The method according to claim 1, wherein: The skeleton flow network described in step (4), wherein the two-dimensional convolution block consists of 16 convolution kernels; D2, D3 and D4 each consist of three convolution layers, and the first two convolution layers in D2 each consist of 16 convolution kernels, and the third convolution layer consists of 64 convolution kernels; the first two convolution layers in D3 each consist of 32 convolution kernels, and the third convolution layer consists of 128 convolution kernels; the first two convolution layers in D4 each consist of 64 convolution kernels, and the third convolution layer consists of 256 convolution kernels; the time step of all stages is 1, and the spatial step is 2×2.

7. The method according to claim 1, wherein: The RGB network described in step (4), wherein the two-dimensional convolution block consists of 32 convolution kernels; D1, D2, D3 and D4 each consist of three convolution layers, and the first two convolution layers in D1 each consist of 32 convolution kernels, and the third convolution layer consists of 128 convolution kernels; the first two convolution layers in D2 each consist of 64 convolution kernels, and the third convolution layer consists of 256 convolution kernels; the first two convolution layers in D3 each consist of 128 convolution kernels, and the third convolution layer consists of 512 convolution kernels; the first two convolution layers in D4 each consist of 256 convolution kernels, and the third convolution layer consists of 1024 convolution kernels; the time step of all stages is 1, and the spatial step is 2×2.

8. The method according to claim 1, wherein: The pooling layer described in step (4) uses maximum pooling, and the sliding window size is 1×3 2 , the time step is 1 and the space step is 2×2.

9. The method according to claim 1, wherein: The early feature fusion module in step (5) includes a three-dimensional convolution, a normalization layer and a ReLU activation function, where the convolution kernel size of the three-dimensional convolution is 3×3 2 , the time step is 1, and the space step is 2×2; the late feature fusion module is used to fuse the classification probability, which is achieved by calculating the mean classification probability of the skeleton flow network and the RGB flow network.

10. The method according to claim 1, wherein: The optimizer described in step (7) adopts the SGD stochastic gradient descent algorithm, and the network loss function is the cross entropy loss function.

Images