Human action recognition method and recognition system based on dual-stream information fusion
By using channel-level motion module and multiplication interaction module in ResNet-50's hybrid structure neural network to fuse time and spatial flow characteristics, the problem of slow recognition speed of traditional dual-stream networks is solved, and the accuracy and speed of motion recognition are improved. Reduced calculation costs.
Patent Information
- Application Number
- CN202411229887.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-09-03
AI Technical Summary
The traditional human body movement recognition method based on dual-stream networks is poor in recognition speed in complex backgrounds and requires additional computing and storage costs. The network parameters are large, which reduces the computing speed.
A hybrid structure neural network based on ResNet-50 is adopted to extract the motion characteristics of RGB video frames through the channel-level motion module, and the multiplication interaction module integrates the time and spatial flow characteristics to reduce the amount of network parameters and improve the recognition speed.
It improves the accuracy and computing speed of action recognition, reduces the demand for optical flow input, reduces the amount of calculation, and enhances network performance.
Smart Images

Figure CN118736682B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision, and in particular relates to a human action recognition method and a recognition system based on dual-stream information fusion. Background Art
[0002] As a hot research topic in the field of computer vision, video action recognition is widely used in video surveillance, intelligent medical care, human-computer interaction and other fields. However, there are often a lot of complex situations in videos, such as cluttered backgrounds, similar action objects and irregular camera movements, which may affect the feature extraction and recognition performance of video actions.
[0003] With the development of Convolutional Neural Network (CNN) technology in the field of recognition, CNN-based neural networks can achieve better recognition performance than traditional methods and are widely used in various practical scenarios. Among them, the CNN-based two-stream network is one of the most widely used and effective neural networks for human action recognition. However, the optical flow information input by the time stream channel of the traditional two-stream network requires additional computing and storage costs, the network parameters are large and the network operation speed is reduced.
[0004] Therefore, a new action recognition method is needed to eliminate the requirement for optical flow input and improve the recognition speed. Summary of the invention
[0005] In view of this, the present invention proposes a human action recognition method and a recognition system based on dual-stream information fusion, which are used to solve the problem of poor recognition speed of traditional action recognition based on dual-stream networks.
[0006] In a first aspect, the present invention discloses a method for human action recognition based on dual-stream information fusion, the method comprising:
[0007] Get the target video and extract the RGB video frame sequence by random sampling;
[0008] Using ResNet-50 as the backbone network, a hybrid neural network consisting of a temporal branch network and a spatial branch network is constructed;
[0009] In the hybrid structure neural network, the motion features of the RGB video frame are extracted through the channel-level motion module as the input of the time branch network; the time branch network and the space branch network perform spatiotemporal feature information fusion through multiplicative interaction;
[0010] The RGB video frame sequence is input into the pre-trained hybrid structure neural network for action recognition.
[0011] On the basis of the above technical solution, preferably, the hybrid structure neural network includes an input layer, a channel-level motion module, a time branch network, a space branch network, a multiplicative interaction module and an output layer;
[0012] The input layer, the first convolutional layer, the channel-level motion module, the time branch network and the output layer are connected in sequence; the space branch network is connected to the first convolutional layer and the output layer respectively; the multiplicative interaction module is connected to the time branch network and the space branch network respectively;
[0013] The input layer is used to input an RGB video frame sequence;
[0014] The first convolutional layer is used to perform a convolution operation on the input RGB video frame sequence to extract 3D spatiotemporal features;
[0015] The channel-level motion module is used to extract motion features of adjacent RGB video frames from the 3D spatiotemporal features output by the first convolutional layer;
[0016] The temporal branch network is used to extract temporal flow features from motion features of adjacent RGB video frames output by the channel-level motion module through a residual unit;
[0017] The spatial branch network is used to extract spatial flow features from the 3D spatiotemporal features output by the first convolutional layer through a residual unit;
[0018] The multiplicative interaction module is used to integrate the temporal flow features of the temporal branch network into the spatial flow features of the spatial branch network through a multiplicative connection operation;
[0019] The output layer is used to fuse the temporal stream features and the spatial stream features and output the action classification result.
[0020] On the basis of the above technical solution, preferably, the channel-level motion module includes a first 2D convolution unit, a second 2D convolution unit, a third 2D convolution unit, a fourth 2D convolution unit, a first frame difference operation unit, a second frame difference operation unit and a feature splicing unit;
[0021] The first 2D convolution unit, the second 2D convolution unit, the first frame difference operation unit, the feature splicing unit and the fourth 2D convolution unit are connected in sequence; the output end of the first 2D convolution unit is connected to the input end of the first frame difference operation unit and the second frame difference operation unit respectively; the first 2D convolution unit, the third 2D convolution unit, the second frame difference operation unit and the feature splicing unit are connected in sequence;
[0022] The first 2D convolution unit is used to perform channel dimension reduction on each input video frame to obtain a first feature map of each video frame;
[0023] The second 2D convolution unit is used to extract features from the first feature map of the current frame to obtain a second feature map of the current frame;
[0024] The third 2D convolution unit is used to extract features from the first feature map of the next frame to obtain a third feature map of the next frame;
[0025] The first frame difference operation unit is used to perform a frame difference operation on a first feature map of a previous frame and a second feature map of a current frame to obtain a first motion feature;
[0026] The second frame difference operation unit is used to perform a frame difference operation on the first feature map of the current frame and the third feature map of the next frame to obtain a second motion feature;
[0027] The feature splicing unit is used to splice the first motion feature and the second motion feature to obtain a third motion feature;
[0028] The fourth 2D convolution unit is used to perform dimension upgrading on the third motion feature output by the feature concatenation unit to obtain a time flow feature.
[0029] On the basis of the above technical solution, preferably, the time branch network and the space branch network both include a second convolution layer, a first splicing layer, a third convolution layer, a second splicing layer, a fourth convolution layer, a third splicing layer, a fifth convolution layer, a fourth splicing layer and a Softmax layer connected in sequence;
[0030] The second convolution layer, the third convolution layer, the fourth convolution layer and the fifth convolution layer each include a plurality of residual units, each of which includes a size of 1 1 1.1 3 3.1 1 1Three layers of convolution;
[0031] The first concatenation layer is used to concatenate the input data of the second convolution layer with the output data of the second convolution layer to obtain a first concatenation feature, and output the first concatenation feature to the third convolution layer;
[0032] The second concatenation layer is used to concatenate the first concatenation feature with the output data of the third convolution layer to obtain a second concatenation feature, and output the second concatenation feature to the fourth convolution layer;
[0033] The third concatenation layer is used to concatenate the second concatenation feature with the output data of the fourth convolution layer to obtain a third concatenation feature, and output the third concatenation feature to the fifth convolution layer;
[0034] The fourth concatenation layer is used to concatenate the third concatenation feature with the output data of the fifth convolution layer to obtain a fourth concatenation feature, and output the fourth concatenation feature to the Softmax layer;
[0035] The Softmax layer is used to perform a nonlinear transformation on the fourth concatenated feature and output a feature quantity whose feature element is between 0 and 1.
[0036] On the basis of the above technical solution, preferably, the multiplicative connection operation of the multiplicative interaction module specifically includes:
[0037] The temporal stream feature information output by the residual units of the third convolutional layer and the fifth convolutional layer of the temporal branch network and the spatial stream feature information input by the residual unit of the spatial branch network are respectively weighted multiplied pixel by pixel;
[0038] The forward propagation formula of the multiplicative interaction module is:
[0039]
[0040] in, They are the time branch network The input data and output data of the residual unit, a is the corresponding number of layers, F (∙) is the function used to learn the residual, represents the element-by-element multiplication between matrices, is the convolution kernel of the corresponding residual unit; represents the activation function, Represents the spatial branch network The input data of the residual unit is m is the corresponding number of layers.
[0041] On the basis of the above technical solution, preferably, the formula of the forward propagation process of the spatial branch network is:
[0042]
[0043] The gradient formula of the back propagation process of the spatial branch network is:
[0044]
[0045] Where L represents the loss function, S (∙) represents the Softmax function.
[0046] On the basis of the above technical solution, preferably, the gradient formula of the back propagation process of the time branch network is:
[0047]
[0048] in, They represent the input data and output data of the th residual unit of the spatial branch network respectively.
[0049] In a second aspect, the present invention discloses a human action recognition system based on dual-stream information fusion, the system comprising:
[0050] Data acquisition module: used to acquire the target video and extract the RGB video frame sequence through random sampling;
[0051] Network construction module: used to construct a hybrid structure neural network including a time branch network and a space branch network using ResNet-50 as the backbone network; in the hybrid structure neural network, the motion features of the RGB video frame are extracted through the channel-level motion module as the input of the time branch network; the time branch network and the space branch network perform spatiotemporal feature information fusion through multiplicative interaction;
[0052] Action recognition module: used to input RGB video frame sequences into the pre-trained hybrid structure neural network for action recognition.
[0053] In a third aspect of the present invention, an electronic device is disclosed, comprising: at least one processor, at least one memory, a communication interface and a bus;
[0054] Wherein, the processor, memory, and communication interface communicate with each other via the bus;
[0055] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the method according to the first aspect of the present invention.
[0056] According to a fourth aspect of the present invention, a computer-readable storage medium is disclosed, wherein the computer-readable storage medium stores computer instructions, and the computer instructions enable a computer to implement the method described in the first aspect of the present invention.
[0057] Compared with the prior art, the present invention has the following beneficial effects:
[0058] 1) The present invention uses ResNet-50 as the backbone network and constructs a hybrid structure neural network including a time branch network and a space branch network. In the hybrid structure neural network, a lightweight channel-level motion module is used to extract the motion features of the RGB video frame as the input of the time branch network. The lightweight channel-level motion module reduces the amount of calculation while capturing the motion features with obvious edges, making up for the fact that the existing action recognition method cannot eliminate the need for dual-stream time stream channel input optical flow based on spatiotemporal information, thereby reducing the amount of network parameters and improving the network operation speed and action recognition speed.
[0059] 2) The time branch network and the space branch network of the hybrid structure neural network of the present invention fuse the time and space feature information through multiplicative interaction, realizing the dual-stream information fusion of the time stream and the space stream, which can improve the performance of the network model and thus improve the accuracy of identifying related actions in the target video.
[0060] 3) The multiplicative interaction of the present invention is to add a multiplicative gate from the time stream to the space stream in the residual block of the residual network to achieve the purpose of spatiotemporal fusion. In order to obtain more favorable spatiotemporal features, the present invention adds a Sigmoid nonlinear function in the spatial molecular network before performing multiplicative operation fusion to convert the feature elements to 0~1, which can reduce the computational amount of the multiplicative fusion operation and further improve the computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0062] Figure 1 A flow chart of a method for human action recognition based on dual-stream information fusion according to the present invention;
[0063] Figure 2 It is a schematic diagram of the structure of the hybrid structure neural network of the present invention;
[0064] Figure 3 It is a structural schematic diagram of the channel-level motion module of the present invention;
[0065] Figure 4 A schematic diagram of the principle of multiplicative interaction between residual units of a temporal branch network and a spatial branch network of the present invention;
[0066] Figure 5 The present invention is a schematic diagram of the principle of adding the multiplicative interaction operation of the Sigmoid nonlinear function in the spatial branch network. DETAILED DESCRIPTION
[0067] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0068] See also Figure 1 The present invention discloses a method for human action recognition based on dual-stream information fusion, the method comprising:
[0069] S1. Obtain the target video and extract the RGB video frame sequence by random sampling.
[0070] The target video to be identified is obtained, and video frames of size 320×240 are extracted from the target video by random sampling. The video frames are preprocessed to obtain an RGB video frame sequence with a frame number of T=17.
[0071] S2. Using ResNet-50 as the backbone network, a hybrid structure neural network consisting of a time branch network and a spatial branch network is constructed.
[0072] like Figure 2 The hybrid structure neural network constructed by the present invention includes an input layer, a first convolutional layer, a channel-level motion module, a time branch network, a space branch network, a multiplicative interaction module and an output layer.
[0073] Among them, the input layer, the first convolutional layer, the channel-level motion module, the time branch network and the output layer are connected in sequence; the space branch network is connected to the first convolutional layer and the output layer respectively; the multiplicative interaction module is connected to the time branch network and the space branch network respectively.
[0074] The input layer is used to input an RGB video frame sequence.
[0075] The first convolution layer is used to perform a convolution operation on the input RGB video frame sequence to extract 3D spatiotemporal features;
[0076] In an embodiment of the present invention, the first convolutional layer is actually the first layer Conv1 of the ResNet-50 network, and 3D spatiotemporal features are extracted from the input RGB video frame sequence through the first convolutional layer.
[0077] The channel-level motion module is used to extract motion features of adjacent RGB video frames from the 3D spatiotemporal features output by the first convolutional layer.
[0078] like Figure 3 Shown is a schematic structural diagram of the channel-level motion module used in the present invention.
[0079] In an embodiment of the present invention, the channel-level motion module provides a feature expression method similar to optical flow as an additional feature input of the hybrid structure neural network.
[0080] like Figure 3As shown, the channel-level motion module includes a first 2D convolution unit, a second 2D convolution unit, a third 2D convolution unit, a fourth 2D convolution unit, a first frame difference operation unit, a second frame difference operation unit and a feature splicing unit.
[0081] Among them, the first 2D convolution unit, the second 2D convolution unit, the first frame difference operation unit, the feature splicing unit and the fourth 2D convolution unit are connected in sequence; the output end of the first 2D convolution unit is connected to the input ends of the first frame difference operation unit and the second frame difference operation unit respectively; the first 2D convolution unit, the third 2D convolution unit, the second frame difference operation unit and the feature splicing unit are connected in sequence.
[0082] The first 2D convolution unit has a convolution kernel size of 1×1 and is used to perform channel dimensionality reduction on each input video frame to obtain a first feature map of each video frame.
[0083] The second 2D convolution unit has a convolution kernel size of 3×3 and is used to extract features from the first feature map of the current frame to obtain the second feature map of the current frame.
[0084] The third 2D convolution unit has a convolution kernel size of 3×3 and is used to extract features from the first feature map of the next frame to obtain a third feature map of the next frame.
[0085] The first frame difference operation unit is used to perform a frame difference operation on the first feature map of the previous frame and the second feature map of the current frame to obtain a first motion feature.
[0086] The second frame difference operation unit is used to perform a frame difference operation on the first feature map of the current frame and the third feature map of the next frame to obtain a second motion feature.
[0087] The feature splicing unit is used to splice the first motion feature and the second motion feature to obtain a third motion feature.
[0088] The fourth 2D convolution unit has a convolution kernel size of 1×1 and is used to increase the dimension of the third motion feature output by the feature splicing unit to obtain a time flow feature.
[0089] The calculation principle of the channel-level motion module is as follows: assuming that the size of the feature map of a given input is G, first use a 1×1 first 2D convolution unit to reduce the dimension of the feature map on the channel. Here Set to 16, that is, compress the channel to 16 times the original input, reducing the amount of calculation. Use a 3×3 second 2D convolution unit and a third 2D convolution unit to convolve and calculate the action feature information of two adjacent continuous feature maps. and Represent the feature maps of the previous frame, current frame and next frame respectively.
[0090] Then the motion feature information H of two adjacent video frames is calculated by frame difference operation. t :
[0091]
[0092] Among them, c, t, h, w represent the spatial, temporal channels and two spatial dimensions of the feature. It is similar to the weighted coefficient. The subscripts i and j represent the spatial index of the convolution kernel. The convolution kernel size is set to 3×3, so Through the above processing, T-1 groups of action information similar to that captured by optical flow are generated between adjacent temporal features.
[0093] In order to ensure the dimension T, the last step feature is represented by 0, and then all the feature maps H obtained by the frame difference operation are t Connect them together and finally use 1×1 2D convolution to increase the dimension and restore the dimension to the number of channels of the original input to obtain time series information.
[0094] In the hybrid structure neural network of the present invention, a lightweight channel-level motion module is used to extract the motion features of the RGB video frame as the input of the time branch network. The lightweight channel-level motion module reduces the amount of calculation while capturing the motion features with obvious edges, making up for the fact that the existing action recognition method cannot eliminate the need for dual-stream time stream channel input optical flow based on spatiotemporal information, thereby reducing the amount of network parameters and improving the network operation speed.
[0095] The temporal branch network is used to extract temporal flow features from motion features of adjacent RGB video frames output by the channel-level motion module through a residual unit.
[0096] like Figure 2 As shown, the time branch network includes a second convolution layer Conv2_x, a first splicing layer, a third convolution layer Conv3_x, a second splicing layer, a fourth convolution layer Conv4_x, a third splicing layer, a fifth convolution layer Conv5_x, a fourth splicing layer and a Softmax layer connected in sequence;
[0097] The second convolution layer, the third convolution layer, the fourth convolution layer and the fifth convolution layer each include a plurality of residual units, each of which includes a size of 1 1 1.1 3 3.1 1 1 Three-layer convolution; the third convolution layer Conv3_x includes 4 residual units, and the fifth convolution layer Conv5_x includes 3 residual units.
[0098] The first concatenation layer is used to concatenate the input data of the second convolution layer with the output data of the second convolution layer to obtain a first concatenation feature, and output the first concatenation feature to the third convolution layer;
[0099] The second concatenation layer is used to concatenate the first concatenation feature with the output data of the third convolution layer to obtain a second concatenation feature, and output the second concatenation feature to the fourth convolution layer;
[0100] The third concatenation layer is used to concatenate the second concatenation feature with the output data of the fourth convolution layer to obtain a third concatenation feature, and output the third concatenation feature to the fifth convolution layer;
[0101] The fourth concatenation layer is used to concatenate the third concatenation feature with the output data of the fifth convolution layer to obtain a fourth concatenation feature, and output the fourth concatenation feature to the Softmax layer;
[0102] The Softmax layer is used to perform a nonlinear transformation on the fourth concatenated feature and output a feature quantity whose feature element is between 0 and 1.
[0103] The spatial branch network is used to extract spatial flow features from the 3D spatiotemporal features output by the first convolutional layer through a residual unit.
[0104] The structure of the spatial branch network is the same as that of the temporal branch network. Both use ResNet-50 as the backbone network. The parameters of each layer of the ResNet-50 backbone network are shown in Table 1.
[0105] Table 1 Parameters of each layer of the ResNet-50 backbone network
[0106]
[0107] For ResNet-50 in the embodiment of the present invention, N=2, M=4.
[0108] The multiplicative interaction module is used to integrate the temporal flow features of the temporal branch network into the spatial flow features of the spatial branch network through a multiplicative connection operation.
[0109] In ResNet-50, the residual network uses residual units to learn the residual representation of the input signal, and uses jump connections between residual units to solve the problem of gradient disappearance or explosion. Formula (2) represents the forward propagation formula for residual block learning in ResNet:
[0110]
[0111] The back propagation formula for residual block learning of the residual network is shown in (3):
[0112]
[0113] in, and It is The input and output data of the residual unit, F is the function used to learn the residual, is the corresponding convolution kernel, and Represents the activation function of a linear unit, such as the ReLU function.
[0114] Based on the above propagation formula, the present invention designs a multiplicative connection operation for the multiplicative interaction module, that is, the temporal stream feature information output by the residual unit of the temporal branch network and the spatial stream feature information input by the residual unit of the spatial branch network are weighted multiplied pixel by pixel to achieve the fusion of temporal and spatial feature information. Figure 2 In the hybrid structure neural network shown, the cross-stream connection from the time stream to the space stream is realized in two stages: the third convolutional layer Conv3_x of the time branch network and the fifth convolutional layer Conv5_x of the space branch network.
[0115] In the embodiment of the present invention, the forward propagation formula of the multiplicative interaction module is:
[0116]
[0117] in They are the time branch network The input data and output data of the residual unit, a is the corresponding number of layers, is the function used to learn the residual, represents the multiplication of corresponding elements, is the convolution kernel of the corresponding residual unit; represents the activation function, Represents the spatial branch network The input data of the residual unit is m is the corresponding number of layers.
[0118] Figure 4 Schematic diagram showing the principle of the multiplicative interaction operation between the residual units of the time branch network and the spatial branch network, that is, injecting the time stream into the spatial stream across layers for a multiplication gate. Specifically, the output of a residual unit in the time branch network is weighted multiplied with the input of the residual unit in the next spatial branch network at the pixel level, where is the corresponding convolution kernel. Figure 4The residual units of the temporal branch network and the spatial branch network are the residual units of the second convolutional layer Conv2_x, the third convolutional layer Conv3_x, the fourth convolutional layer Conv4_x, the fifth convolutional layer Conv5_x, etc., where each convolution layer contains several residual units, each of which includes a size of 1 1 1.1 3 3.1 1 1Three layers of convolution.
[0119] In this case, the gradient of the loss function of the spatial branch network during the backward pass can be expressed as formula (5):
[0120]
[0121] From formula (5), we can see that the gradient of the residual unit flowing through the spatial branch network is affected by the temporal signal impact.
[0122] Similarly, the gradient of the loss function of the time branch network during the reverse transfer process can be expressed by formula (6):
[0123]
[0124] In back propagation, the input of the spatial branch network is not only , but rather the motion signal of the temporal branch network As a gating module for spatial branching networks.
[0125] The output layer is used to fuse the temporal stream features of the temporal branch network and the spatial stream features of the spatial branch network and output the action classification result.
[0126] The above hybrid structure neural network uses ResNet-50 as the backbone network. By fusing the time branch network and the space branch network, it can obtain the spatiotemporal features that are beneficial to the network. The multiplicative interaction is to add a multiplicative gate from the time flow to the space flow in the residual block of the residual network to achieve the purpose of spatiotemporal fusion.
[0127] As another embodiment of the present invention, in order to obtain more favorable spatiotemporal features, the multiplication interaction module of the present invention fine-tunes the feature elements of the spatial branch network before fusion, that is, before performing the multiplication operation, the Sigmoid nonlinear function is added to the feature elements of the spatial branch network. On the one hand, the closer the feature element is to 0, the closer the result of fusion by element-by-element multiplication is to 0. Therefore, the use of the Sigmoid nonlinear function limits the output range of the element. On the other hand, performing the multiplication fusion operation after the feature elements are all located at (0,1) can reduce the amount of calculation. Figure 5 The figure shows the principle of adding the multiplicative interaction operation of the Sigmoid nonlinear function to the spatial branch network.
[0128] At this time, the formula for the forward propagation process of the spatial branch network is:
[0129]
[0130] The gradient formula of the back propagation process of the spatial branch network is:
[0131]
[0132] Where L represents the loss function, S (∙) represents the Softmax function.
[0133] Correspondingly, the gradient formula of the back propagation process of the time branch network is:
[0134]
[0135] in, Respectively represent the input data and output data of the residual unit of the spatial branch network, a , m is the corresponding number of layers.
[0136] The present invention proposes a hybrid structure neural network based on dual-stream information fusion for video action recognition. The hybrid structure neural network includes two branches, a time branch network and a space branch network, with ResNet-50 as the backbone network of the two branches, respectively performing time stream feature extraction and space stream feature extraction.
[0137] In the hybrid structure neural network of the present invention, both the space branch network and the time branch network take RGB video frames as input. A lightweight channel-level motion module is added before the time branch network. The motion features of the RGB video frames are extracted by the channel-level motion module as the input of the time branch network, replacing the optical flow input, which can reduce the number of network parameters and improve computational efficiency.
[0138] In the hybrid structure neural network of the present invention, the time branch network and the space branch network perform spatiotemporal feature information fusion through multiplicative interaction, specifically, the time stream is unidirectionally cross-flowed to the space stream for multiplicative connection to achieve spatiotemporal feature information fusion, and the spatiotemporal relationship of the interactive fusion of spatiotemporal features is learned, and finally the classification scores of the two streams are obtained for fusion to obtain the classification result.
[0139] S3. Input the RGB video frame sequence into the pre-trained hybrid structure neural network for action recognition.
[0140] Before performing action recognition through a hybrid structure neural network, it is necessary to establish a data set to train the hybrid structure neural network established in step S3. Specifically, the pre-training process mainly includes the following steps:
[0141] Data collection: Video data can be obtained by obtaining existing video datasets, downloading videos from public datasets, or shooting videos yourself.
[0142] Data cleaning and labeling: The collected videos are cleaned to remove noise and irrelevant data. Then the actions in the videos are labeled to mark the action categories in each time period.
[0143] Data preprocessing: Preprocess the video, including video segmentation, RGB video frame extraction, downsampling, etc., and establish a data set with RGB video frame sequence as sample attribute and action category as label to meet the needs of hybrid structure neural network.
[0144] Model training: Use the established dataset to train the hybrid structure neural network.
[0145] Model evaluation: Use the test set to evaluate the trained hybrid structure neural network. The evaluation indicators may include accuracy, recall rate, F1 value, etc.
[0146] Through the above pre-trained hybrid structure neural network, action recognition based on RGB video frame sequences can be performed.
[0147] The present invention proposes a hybrid structure neural network based on dual-stream information fusion, and performs action recognition on the target video based on the dual-stream information fusion, thereby making up for the fact that the existing action recognition method cannot eliminate the need for dual-stream time stream channel input optical flow based on spatiotemporal information, and can improve the accuracy of identifying related actions in the target video.
[0148] Corresponding to the above method embodiment, the present invention discloses a human action recognition system based on dual-stream information fusion, the system comprising:
[0149] Data acquisition module: used to acquire the target video and extract the RGB video frame sequence through random sampling;
[0150] Network construction module: used to construct a hybrid structure neural network including a time branch network and a space branch network using ResNet-50 as the backbone network; in the hybrid structure neural network, the motion features of the RGB video frame are extracted through the channel-level motion module as the input of the time branch network; the time branch network and the space branch network perform spatiotemporal feature information fusion through multiplicative interaction;
[0151] Action recognition module: used to input RGB video frame sequences into the pre-trained hybrid structure neural network for action recognition.
[0152] The above system embodiments and method embodiments correspond one to one, and for a brief description of the system embodiments, please refer to the method embodiments.
[0153] The present invention also discloses an electronic device, comprising: at least one processor, at least one memory, a communication interface and a bus; wherein the processor, memory, and communication interface communicate with each other through the bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the aforementioned method of the present invention.
[0154] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, wherein the computer instructions enable the computer to implement all or part of the steps of the method described in the embodiment of the present invention. The storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk, and other media that can store program codes.
[0155] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be distributed to multiple network units. A person skilled in the art may select some or all of the modules according to actual needs to achieve the purpose of the solution of this embodiment without creative effort.
[0156] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A human action recognition method based on dual-stream information fusion, characterized in that: The method comprises: Get the target video and extract the RGB video frame sequence by random sampling; Using ResNet-50 as the backbone network, a hybrid neural network consisting of a temporal branch network and a spatial branch network is constructed; In the hybrid structure neural network, the motion features of the RGB video frame are extracted through the channel-level motion module as the input of the time branch network; the time branch network and the space branch network perform spatiotemporal feature information fusion through multiplicative interaction; Input the RGB video frame sequence into the pre-trained hybrid structure neural network for action recognition; The hybrid structure neural network includes an input layer, a first convolutional layer, a channel-level motion module, a time branch network, a space branch network, a multiplicative interaction module and an output layer; The input layer, the first convolutional layer, the channel-level motion module, the time branch network and the output layer are connected in sequence; the space branch network is connected to the first convolutional layer and the output layer respectively; the multiplicative interaction module is connected to the time branch network and the space branch network respectively; The input layer is used to input an RGB video frame sequence; The first convolutional layer is used to perform a convolution operation on the input RGB video frame sequence to extract 3D spatiotemporal features; The channel-level motion module is used to extract motion features of adjacent RGB video frames from the 3D spatiotemporal features output by the first convolutional layer; The temporal branch network is used to extract temporal flow features from motion features of adjacent RGB video frames output by the channel-level motion module through a residual unit; The spatial branch network is used to extract spatial flow features from the 3D spatiotemporal features output by the first convolutional layer through a residual unit; The multiplicative interaction module is used to integrate the temporal flow features of the temporal branch network into the spatial flow features of the spatial branch network through a multiplicative connection operation; The output layer is used to fuse the temporal stream features and the spatial stream features and output the action classification result; The channel-level motion module includes a first 2D convolution unit, a second 2D convolution unit, a third 2D convolution unit, a fourth 2D convolution unit, a first frame difference operation unit, a second frame difference operation unit and a feature splicing unit; The first 2D convolution unit, the second 2D convolution unit, the first frame difference operation unit, the feature splicing unit and the fourth 2D convolution unit are connected in sequence; the output end of the first 2D convolution unit is connected to the input end of the first frame difference operation unit and the second frame difference operation unit respectively; the first 2D convolution unit, the third 2D convolution unit, the second frame difference operation unit and the feature splicing unit are connected in sequence; The first 2D convolution unit is used to perform channel dimension reduction on each input video frame to obtain a first feature map of each video frame; The second 2D convolution unit is used to extract features from the first feature map of the current frame to obtain a second feature map of the current frame; The third 2D convolution unit is used to extract features from the first feature map of the next frame to obtain a third feature map of the next frame; The first frame difference operation unit is used to perform a frame difference operation on a first feature map of a previous frame and a second feature map of a current frame to obtain a first motion feature; The second frame difference operation unit is used to perform a frame difference operation on the first feature map of the current frame and the third feature map of the next frame to obtain a second motion feature; The feature splicing unit is used to splice the first motion feature and the second motion feature to obtain a third motion feature; The fourth 2D convolution unit is used to perform dimension upgrading on the third motion feature output by the feature concatenation unit to obtain a time flow feature.
2. The method for human action recognition based on dual-stream information fusion according to claim 1, characterized in that: The time branch network and the space branch network both include a second convolution layer, a first splicing layer, a third convolution layer, a second splicing layer, a fourth convolution layer, a third splicing layer, a fifth convolution layer, a fourth splicing layer and a Softmax layer connected in sequence; The second convolution layer, the third convolution layer, the fourth convolution layer and the fifth convolution layer each include a plurality of residual units, each of which includes a size of 1 1 1.1 3 3.1 1 1Three layers of convolution; The first concatenation layer is used to concatenate the input data of the second convolution layer with the output data of the second convolution layer to obtain a first concatenation feature, and output the first concatenation feature to the third convolution layer; The second concatenation layer is used to concatenate the first concatenation feature with the output data of the third convolution layer to obtain a second concatenation feature, and output the second concatenation feature to the fourth convolution layer; The third concatenation layer is used to concatenate the second concatenation feature with the output data of the fourth convolution layer to obtain a third concatenation feature, and output the third concatenation feature to the fifth convolution layer; The fourth concatenation layer is used to concatenate the third concatenation feature with the output data of the fifth convolution layer to obtain a fourth concatenation feature, and output the fourth concatenation feature to the Softmax layer; The Softmax layer is used to perform a nonlinear transformation on the fourth concatenated feature and output a feature quantity whose feature element is between 0 and 1.
3. The method for human action recognition based on dual-stream information fusion according to claim 2 is characterized in that: The multiplicative connection operation of the multiplicative interaction module specifically includes: The temporal stream feature information output by the residual units of the third convolutional layer and the fifth convolutional layer of the temporal branch network and the spatial stream feature information input by the residual unit of the spatial branch network are respectively weighted multiplied pixel by pixel; The forward propagation formula of the multiplicative interaction module is: ; in, They are the time branch network The input data and output data of the residual unit, a is the corresponding number of layers, is the function used to learn the residual, represents the element-by-element multiplication between matrices, is the convolution kernel of the corresponding residual unit; represents the activation function, Represents the spatial branch network The input data of the residual unit is m is the corresponding number of layers.
4. The method for human action recognition based on dual-stream information fusion according to claim 3 is characterized in that: The formula for the forward propagation process of the spatial branch network is: ; The gradient formula of the back propagation process of the spatial branch network is: ; in, L represents the loss function, Represents the Softmax function.
5. The method for human action recognition based on dual-stream information fusion according to claim 4 is characterized in that: The gradient formula of the back propagation process of the time branch network is: ; in, Represents the spatial branch network l The input data and output data of the residual unit, m is the corresponding number of layers.
6. A human action recognition system based on dual-stream information fusion, characterized in that: The system comprises: Data acquisition module: used to acquire the target video and extract the RGB video frame sequence through random sampling; Network construction module: used to construct a hybrid structure neural network including a time branch network and a space branch network using ResNet-50 as the backbone network; in the hybrid structure neural network, the motion features of the RGB video frame are extracted through the channel-level motion module as the input of the time branch network; the time branch network and the space branch network perform spatiotemporal feature information fusion through multiplicative interaction; The hybrid structure neural network includes an input layer, a first convolutional layer, a channel-level motion module, a time branch network, a space branch network, a multiplicative interaction module and an output layer; The input layer, the first convolutional layer, the channel-level motion module, the time branch network and the output layer are connected in sequence; the space branch network is connected to the first convolutional layer and the output layer respectively; the multiplicative interaction module is connected to the time branch network and the space branch network respectively; The input layer is used to input an RGB video frame sequence; The first convolutional layer is used to perform a convolution operation on the input RGB video frame sequence to extract 3D spatiotemporal features; The channel-level motion module is used to extract motion features of adjacent RGB video frames from the 3D spatiotemporal features output by the first convolutional layer; The temporal branch network is used to extract temporal flow features from motion features of adjacent RGB video frames output by the channel-level motion module through a residual unit; The spatial branch network is used to extract spatial flow features from the 3D spatiotemporal features output by the first convolutional layer through a residual unit; The multiplicative interaction module is used to integrate the temporal flow features of the temporal branch network into the spatial flow features of the spatial branch network through a multiplicative connection operation; The output layer is used to fuse the temporal stream features and the spatial stream features and output the action classification result; The channel-level motion module includes a first 2D convolution unit, a second 2D convolution unit, a third 2D convolution unit, a fourth 2D convolution unit, a first frame difference operation unit, a second frame difference operation unit and a feature splicing unit; The first 2D convolution unit, the second 2D convolution unit, the first frame difference operation unit, the feature splicing unit and the fourth 2D convolution unit are connected in sequence; the output end of the first 2D convolution unit is connected to the input end of the first frame difference operation unit and the second frame difference operation unit respectively; the first 2D convolution unit, the third 2D convolution unit, the second frame difference operation unit and the feature splicing unit are connected in sequence; The first 2D convolution unit is used to perform channel dimension reduction on each input video frame to obtain a first feature map of each video frame; The second 2D convolution unit is used to extract features from the first feature map of the current frame to obtain a second feature map of the current frame; The third 2D convolution unit is used to extract features from the first feature map of the next frame to obtain a third feature map of the next frame; The first frame difference operation unit is used to perform a frame difference operation on a first feature map of a previous frame and a second feature map of a current frame to obtain a first motion feature; The second frame difference operation unit is used to perform a frame difference operation on the first feature map of the current frame and the third feature map of the next frame to obtain a second motion feature; The feature splicing unit is used to splice the first motion feature and the second motion feature to obtain a third motion feature; The fourth 2D convolution unit is used to perform dimension upscaling on the third motion feature output by the feature concatenation unit to obtain a time flow feature; Action recognition module: used to input RGB video frame sequences into the pre-trained hybrid structure neural network for action recognition.
7. An electronic device, characterized in that: include: at least one processor, at least one memory, a communication interface, and a bus; Wherein, the processor, memory, and communication interface communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the method according to any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions enable a computer to implement the method according to any one of claims 1 to 5.
