Human behavior recognition system and method based on feature fusion of segmented attention

Abstract

The present invention discloses a human behavior recognition system and method based on feature fusion of segmented attention in the field of computer vision action recognition technology, including: obtaining video samples; performing frame-by-frame decomposition sampling based on the video samples; inputting the sampled video frames into the segmented attention network for deep feature extraction, and outputting feature vectors; inputting the extracted feature vectors into the bidirectional long short-term memory network to model time information and extract time features; based on the extracted time features, using a softmax classifier to classify the results into action categories, wherein the action with the highest score is the recognition result, and the action score is the recognition accuracy of the action. The present invention can solve the problem that the accuracy of the features extracted by the traditional convolutional neural network is insufficient, and the problem that the unidirectional long short-term memory network cannot fully utilize the time information, and effectively improves the recognition performance.

Description

Technical Field

[0001] The present invention relates to a feature fusion human behavior recognition system and method based on segmented attention, belonging to the technical field of action recognition of computer vision. Background Art

[0002] Human behavior recognition refers to the use of pattern recognition, machine learning and other methods to automatically analyze and identify human behavior in videos. It is widely used in video retrieval, security monitoring, smart home and other fields. Compared with single image recognition, action recognition not only needs to extract the spatial features of the action, but also needs to model the time-related information between video frames. In addition, changes in the background, lighting, and viewing angle in the video will also affect the recognition effect. Therefore, it is very challenging to design an efficient and accurate behavior recognition algorithm.

[0003] Traditional human action recognition algorithms achieve the purpose of recognition by extracting manual features such as feature descriptors of moving targets to characterize human actions. In recent years, deep learning has developed rapidly, and convolutional neural networks have achieved great success in the field of images. Using convolutional neural networks in deep learning to extract action features through self-learning has become the mainstream method in the field of human action recognition.

[0004] At present, human action recognition algorithms based on deep learning can be roughly divided into three categories. The first category is to use a two-stream neural network to model the spatial and temporal information of video frames respectively. The two-stream network greatly improves the recognition accuracy, but the two-stream network only operates one frame in the spatial stream and only operates a single stack of frames in a short clip in the temporal stream, and the use of temporal information in the video is limited. The second category is to use a 3D convolutional neural network to directly model the spatiotemporal information of the video. The 3D convolutional neural network uses a convolution kernel of size 3×3×3 to directly operate 16 frames of video input in the temporal and spatial dimensions, and finally uses a Softmax classifier to obtain the video classification results. The 3D convolutional neural network only uses RGB video frames as input, and does not need to calculate additional optical flow, which improves the data preprocessing speed, but due to its three-dimensional structure, the network parameters also increase exponentially. The third category is an algorithm framework based on CNN-RNN. This framework uses a convolutional neural network to extract the spatial information of the video frame, and then uses a recurrent neural network to extract the temporal information of the action, avoiding the complex optical flow map extraction in the two-stream neural network, and effectively reducing the amount of calculation compared to the 3D convolutional neural network.

[0005] However, there are two major deficiencies in the CNN-RNN based algorithm:

[0006] (1) The convolutional neural network used in the traditional CNN-RNN algorithm is affected by redundant information such as background, and the extracted features are not accurate enough, resulting in less than ideal final recognition accuracy.

[0007] (2) The unidirectional LSTM network only utilizes information from the past and cannot fully model temporal information. Summary of the invention

[0008] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide a human behavior recognition system and method based on feature fusion of segmented attention, which can solve the problem that the accuracy of features extracted by traditional convolutional neural networks is insufficient, and the problem that unidirectional long short-term memory networks cannot fully utilize time information, thereby effectively improving the recognition performance.

[0009] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0010] In a first aspect, the present invention provides a method for human behavior recognition based on feature fusion of segmented attention, comprising:

[0011] Get video samples;

[0012] Perform frame-by-frame decomposition sampling based on video samples;

[0013] The sampled video frames are input into the segmentation attention network for deep feature extraction and the feature vector is output;

[0014] The extracted feature vector is input into the bidirectional long short-term memory network to model the time information and extract the time feature;

[0015] Based on the extracted time features, a softmax classifier is used to classify the results into action categories, where the action with the highest score is the recognition result, and the action score is the recognition accuracy of this type of action.

[0016] Furthermore, frame-by-frame sampling is performed based on the video samples, including:

[0017] For video samples with less than 30 frames, no sampling is done;

[0018] For video samples with a video frame number between 30 and 59 frames, sampling is performed frame by frame;

[0019] For video samples with a video frame number between 60 and 90 frames, one frame is sampled every two frames;

[0020] For video samples with more than 90 frames, one frame is sampled every three frames.

[0021] Furthermore, the sampled video frames are input into the segmentation attention network for deep feature extraction, including:

[0022] Divide the sampled video frame into multiple basis arrays, and then divide each basis array into multiple partitions, perform convolution operation on the input video frame in each partition to obtain multiple feature map groups;

[0023] The weights of different feature map groups in the channel dimension are calculated by using global context information;

[0024] Calculate the channel feature vector in each basis array based on the weight of the channel feature vector;

[0025] Each feature vector is concatenated along the channel dimension to obtain the output feature vector of the segmentation attention module.

[0026] Furthermore, the feature map group is:

[0027] U i =F i (X), i∈{1,2,...,G}

[0028] In the formula, F i is the 1×1 convolution and 3×3 convolution used for each independent feature map group, i represents different feature map groups, X represents the sampled video frame, and U i It is the feature map group output after convolution.

[0029] Furthermore, the global context information is:

[0030]

[0031] In the formula, is the global context information of the c-th channel, k represents the base array, H, W are the output sizes of the c-th channel feature vector, Represents the result of convolution of the feature vector of the cth channel, where i and j represent the width and height coordinates in the feature vector respectively.

[0032] Furthermore, the weight of the channel dimension is:

[0033]

[0034] In the formula, Represents the weight of the c-th channel feature vector, weight function Based on the global context information The weight of the cth channel in each feature map group is determined, and m and n represent the current feature map group.

[0035] Furthermore, the channel feature vector is:

[0036]

[0037] In the formula, is the c-th channel feature vector in the k-th basis array, represents the weight of the c-th channel feature vector, U r(k-1)+iRepresents the i-th original eigenvector in the k-th basis array. The eigenvector representation of the c-th channel can be obtained by weighted accumulation of the original eigenvectors.

[0038] Furthermore, the output feature vector of the segmentation attention module is:

[0039] V=Concat{V 1 ,V 2 ,...,V k}

[0040] Where Concat represents the concatenation operation along the channel dimension, V k Representing the feature vector of the kth basis array, the original input video frame x is short-circuited with the output V of the segmentation attention module to generate the final output of the segmentation attention module:

[0041] Y=V+T(x)

[0042] Where T represents the convolution operation with stride, which is used to align the output shape of the segmentation attention block, and x represents the original input.

[0043] Furthermore, the bidirectional long short-term memory network models the time information as:

[0044] h t =σ(w1x t +w2h t-1 +b t )

[0045] h' t =σ(w3x t +w5h' t+1 +b t )

[0046] O' t =tanh(w4h t +b t )

[0047] O” t =tanh(w6h' t +b t )

[0048]

[0049] where h t-1 ,h t They represent the output of the previous moment and the current moment of the forward reasoning process, respectively, h' t+1 , h' t They represent the output of the previous moment and the current moment of the reverse reasoning process, respectively, and w i represents weight, i=1, 2…, 6, b tis the bias term, x t Features extracted for segmentation attention network, o t is the time series output vector, o' t and o" t Respectively represent the output of forward propagation and back propagation, o t Take the average of the two, σ and tanh represent the sigmoid activation function and tanh activation function respectively.

[0050] In a second aspect, the present invention provides a human behavior recognition system based on feature fusion of segmented attention, comprising:

[0051] Input module: used to obtain video samples;

[0052] Sampling module: used to perform frame-by-frame decomposition sampling based on video samples;

[0053] Feature vector output module: used to input the sampled video frame into the segmentation attention network for deep feature extraction and output feature vector;

[0054] Time feature extraction module: used to input the extracted feature vector into the bidirectional long short-term memory network to model time information and extract time features;

[0055] Action category classification module: used to classify the results of the extracted time features using a softmax classifier into action categories, where the action with the highest score is the recognition result, and the action score is the recognition accuracy of the action.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] The present invention uses a segmented attention network to extract the apparent information of the action in the video, effectively suppressing the redundant background in the video frame, focusing on the action itself, and improving the feature accuracy. Secondly, a feature fusion mechanism is introduced in the segmented attention network to enhance the interaction between high-level and low-level features. Finally, a bidirectional long short-term memory network is used to replace the unidirectional long short-term memory network, and a reverse transmission structure is added while forward transmission, making full use of the front-to-back correlation characteristics of human body movements. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a flow chart of a method for human behavior recognition based on feature fusion of segmented attention provided in Embodiment 1 of the present invention;

[0059] Figure 2 It is a structural diagram of a segmented attention block of a feature fusion human behavior recognition method based on segmented attention provided in the first embodiment of the present invention;

[0060] Figure 3This is a bidirectional long short-term memory network structure diagram of the feature fusion human behavior recognition method based on segmented attention provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0061] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0062] Embodiment 1:

[0063] like Figure 1 As shown, a method for human behavior recognition based on feature fusion of segmented attention includes the following steps:

[0064] Step S1: Decompose each video frame by frame to obtain a video frame. The original size of the video frame is 320×240, and it is cropped and adjusted to 224×224. Its function is to standardize the input information. Secondly, use an adaptive sampling strategy to sample the cropped video frame to maximize the extraction of action information in the video.

[0065] Since the time information in the video is distributed throughout the video frame, the time information distribution is different for videos of different lengths. In order to maximize the collection of time information in the video, it is necessary to use different sampling rates for videos of different lengths. The present invention proposes an adaptive sampling strategy to sample video frames, which is as follows:

[0066] Through the analysis of the data set, it is found that the video samples with less than 30 frames are less than one thousandth of the total samples, so the video samples with less than 30 frames are eliminated. For video frames between 30 and 59 frames, the sampling rate is 1, that is, the first 30 frames are directly sampled without discarding the intermediate frames. For video frames between 60 and 90 frames, the sampling rate is 2, that is, one frame is sampled every two frames for 30 frames; for video frames greater than 90 frames, the sampling rate is 3, that is, one frame is sampled every three frames for 30 frames. Considering the correlation between the previous and next frames in the video, the sampling rate in the adaptive sampling strategy should not be set too large, which will lead to the loss of motion information and reduce the recognition accuracy.

[0067] The data set used in the present invention provides three different training and testing sample divisions. Experiments are conducted on these three training and testing samples respectively, and the final result takes the average value of the three different sample divisions as the final result.

[0068] Step S2: After sampling in step S1, the sampled video frame is input into the segmentation attention network. After convolution, weighted summation and pooling operations of the segmentation attention network, a one-dimensional feature vector is finally output and input into the BI-LSTM network to extract the deep features in the video frame. A feature fusion mechanism is introduced into the segmentation attention network to increase the information interaction between different convolutional layers, so as to more accurately extract the deep features of the video frame.

[0069] The proposed method first uses the segmentation attention network to extract the action information of the video. The segmentation attention network consists of a segmentation attention module, and its structure is as follows: Figure 2 As shown in the figure. In the segmentation attention module, the input video frame is first divided into k basis arrays, and then each basis array is divided into r segmentations. In each segmentation, 1×1 convolution and 3×3 convolution are used to perform convolution operations on the input video frame to reduce the spatial size of the input video frame, and finally G=kr feature map groups are obtained. Each feature map group can be expressed as:

[0070] U i =F i (X), i∈{1,2,...,G}

[0071] In the formula, F i is the 1×1 convolution and 3×3 convolution used for each independent feature map group, i represents different feature map groups, X represents input, and U i It is the feature map group output after convolution.

[0072] The global context information describes the implicit relationship between different pixels in the video frame. The global context information is used in the segmentation attention module. Calculate the weights a of different feature map groups in the channel dimension n , to highlight the important information in the video frame while suppressing the redundant background information, where k represents the base array and n represents the nth segmentation in the base array. Global context information of the cth channel The calculation process is as follows:

[0073]

[0074] Where H, W are the output sizes of the c-th channel feature vector; Represents the result of convolution of the feature vector of the cth channel; i and j represent the width and height coordinates of the feature vector respectively. Using global context information Calculated feature map group weights It can be expressed as:

[0075]

[0076] In the formula, the weight function Based on the global context information The weight of the cth channel in each feature map group is determined, and m and n represent the current feature map group.

[0077] The feature vector of each channel is generated by weighted combination of different feature map groups, that is, soft attention in the channel dimension. Then the feature vector of the cth channel in the kth basis array can be expressed as

[0078]

[0079] In the formula, represents the weight of the c-th channel feature vector, U r(k-1)+i Represents the i-th original eigenvector in the k-th basis array. The eigenvector representation of the c-th channel can be obtained by weighted accumulation of the original eigenvectors.

[0080] Finally, each feature vector is concatenated along the channel dimension to obtain the output of the segmentation attention module:

[0081] V=Concat{V 1 ,V 2 ,...,V k}

[0082] Where Concat represents the concatenation operation along the channel dimension, V 1 represents the eigenvector of the first basis array. Since there are k basis arrays in total, V k Represents the feature vector of the kth basis array. The original input video frame x is short-circuited with the output V of the segmentation attention module to generate the final output of the segmentation attention module:

[0083] Y=V+T(x)

[0084] Where T represents the convolution operation with stride, which is used to align the output shape of the segmentation attention block, and x represents the original input.

[0085] In order to improve the information interaction between different convolutional layers, a feature fusion mechanism is introduced in the segmentation attention network to improve the representation ability of CNN features. The structure of the segmentation attention network is as follows: Figure 1As shown. The output size of Layer4 is 7×7. In order to fuse Layer1 and Layer4, while the forward convolution of the split attention network is being performed, the output of Layer1 is reduced to 7×7 through global average pooling, and then fused with Layer4 as shallow features. The shallow features after pooling are concatenated with the deep features of Layer4 in the channel dimension to achieve the purpose of feature fusion. Then, global average pooling is used again to expand the fused features to 1×1×2048, and the one-dimensional feature vector is transmitted to BI-LSTM (bidirectional long short-term memory network).

[0086] In step S3, the feature vector extracted by the segmentation attention network is input into the BI-LSTM (bidirectional long short-term memory network), which is used to model the temporal information of the action.

[0087] The bidirectional long short-term memory network is a variant of the recurrent neural network. The network introduces gating units to update the time information passed into the network and is widely used in natural language processing and behavior recognition. The process of modeling time information in the bidirectional long short-term memory network is shown in the following formula:

[0088] h t =σ(w1x t +w2h t-1 +b t )

[0089] h' t =σ(w3x t +w5h' t+1 +b t )

[0090] O' t =tanh(w4h t +b t )

[0091] O” t =tanh(w6h' t +b t )

[0092]

[0093] where h t-1 ,h t Respectively represent the output of the previous moment and the current moment of the forward reasoning process; h' t+1 , h' t Respectively represent the output of the previous moment and the current moment of the reverse reasoning process; w i represents weight, i=1,…,6; b t is the bias term; xt The features extracted by the segmentation attention network are input into the BI-LSTM (bidirectional long short-term memory network) in chronological order; t is the time series output vector, o' t and o" t Respectively represent the output of forward propagation and back propagation, o t Take the average of the two; σ and tanh represent the sigmoid activation function and tanh activation function respectively.

[0094] Step 4: Based on the temporal features extracted by the bidirectional LSTM network, a softmax classifier is used to classify the results. The two datasets used have a total of 152 action categories.

[0095] The UCF101 dataset used in the present invention has 101 action categories, and the HMDB51 dataset has 51 action categories. The action with the highest score is taken as the final recognition result, and its score is taken as the final recognition accuracy of the action.

[0096] The present invention uses a segmented attention network to extract the apparent information of the action in the video, effectively suppressing the redundant background in the video frame, focusing on the action itself, and improving the feature accuracy. Secondly, a feature fusion mechanism is introduced in the segmented attention network to enhance the interaction between high-level and low-level features. Finally, a bidirectional long short-term memory network is used to replace the unidirectional long short-term memory network, and a reverse transmission structure is added while forward transmission, making full use of the front-to-back correlation characteristics of human body movements.

[0097] Embodiment 2:

[0098] A human behavior recognition system based on feature fusion of segmented attention, comprising:

[0099] Input module: used to obtain video samples;

[0100] Sampling module: used to perform frame-by-frame decomposition sampling based on video samples;

[0101] Feature vector output module: used to input the sampled video frame into the segmentation attention network for deep feature extraction and output feature vector;

[0102] Time feature extraction module: used to input the extracted feature vector into the bidirectional long short-term memory network to model time information and extract time features;

[0103] Action category classification module: used to classify the results of the extracted time features using a softmax classifier into action categories, where the action with the highest score is the recognition result, and the action score is the recognition accuracy of the action.

[0104] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0105] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0106] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0107] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0108] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A human behavior recognition method based on feature fusion of segmented attention, characterized by: include: Get video samples; Perform frame-by-frame decomposition sampling based on video samples; The sampled video frames are input into the segmentation attention network for deep feature extraction and the feature vector is output; The extracted feature vector is input into the bidirectional long short-term memory network to model the time information and extract the time feature; Based on the extracted time features, the softmax classifier is used to classify the results into action categories, where the action with the highest score is the recognition result, and the action score is the recognition accuracy of this type of action; Among them, the sampled video frames are input into the segmentation attention network for deep feature extraction, including: Divide the sampled video frame into multiple basis arrays, and then divide each basis array into multiple partitions, perform convolution operation on the input video frame in each partition to obtain multiple feature map groups; The weights of different feature map groups in the channel dimension are calculated by using global context information; Calculate the channel feature vector in each basis array based on the weight of the channel feature vector; Each feature vector is concatenated along the channel dimension to obtain the output feature vector of the segmentation attention module.

2. The method for human behavior recognition based on feature fusion of segmented attention according to claim 1, characterized in that: Frame-by-frame sampling based on video samples, including: For video samples with less than 30 frames, no sampling is done; For video samples with a video frame number between 30 and 59 frames, sampling is performed frame by frame; For video samples with a video frame number between 60 and 90 frames, one frame is sampled every two frames; For video samples with more than 90 frames, one frame is sampled every three frames.

3. The method for human behavior recognition based on feature fusion of segmented attention according to claim 1, characterized in that: The feature map groups are: U i =F i (X),i∈{1,2,...,G} In the formula, F i is the 1×1 convolution and 3×3 convolution used for each independent feature map group, i represents different feature map groups, X represents the sampled video frame, and U i It is the feature map group output after convolution.

4. The method for human behavior recognition based on feature fusion of segmented attention according to claim 3 is characterized in that: The global context information is: In the formula, is the global context information of the c-th channel, k represents the base array, H, W are the output sizes of the c-th channel feature vector, Represents the result of convolution of the feature vector of the cth channel, where i and j represent the width and height coordinates in the feature vector respectively.

5. The method for human behavior recognition based on feature fusion of segmented attention according to claim 4 is characterized in that: The weight of the channel dimension is: In the formula, Represents the weight of the c-th channel feature vector, weight function Based on the global context information The weight of the cth channel in each feature map group is determined, and m and n represent the current feature map group.

6. The method for human behavior recognition based on feature fusion of segmented attention according to claim 5, characterized in that: The eigenvectors are: Where V c k is the c-th channel feature vector in the k-th basis array, represents the weight of the c-th channel feature vector, U r(k-1)+i Represents the i-th original eigenvector in the k-th basis array. The eigenvector representation of the c-th channel can be obtained by weighted accumulation of the original eigenvectors.

7. The method for human behavior recognition based on feature fusion of segmented attention according to claim 6, characterized in that: The output feature vector of the segmentation attention module is: V=Concat{V 1 ,V 2 ,...,In k } Where Concat represents the concatenation operation along the channel dimension, V k Representing the feature vector of the kth basis array, the original input video frame x is short-circuited with the output V of the segmentation attention module to generate the final output of the segmentation attention module: Y=V+T(x) Where T represents the convolution operation with stride, which is used to align the output shape of the segmentation attention block, and x represents the original input.

8. The method for human behavior recognition based on feature fusion of segmented attention according to claim 1, characterized in that: The bidirectional long short-term memory network models time information as: h t =σ(w1x t +w2h t-1 +b t ) h’ t =σ(w3x t +w5h’ t+1 +b t ) O' t =tanh(w4h t +b t ) O” t =tanh(w6h' t +b t ) where h t-1 ,h t They represent the output of the previous moment and the current moment of the forward reasoning process, respectively, h' t+1 , h' t They represent the output of the previous moment and the current moment of the reverse reasoning process, respectively, and w i represents weight, i=1, 2…, 6, b t is the bias term, x t Features extracted for segmentation attention network, o t is the time series output vector, o' t and o" t Respectively represent the output of forward propagation and back propagation, o t Take the average of the two, σ and tanh represent the sigmoid activation function and tanh activation function respectively.

9. A human behavior recognition system based on feature fusion of segmented attention, characterized in that: include: Input module: used to obtain video samples; Sampling module: used to perform frame-by-frame decomposition sampling based on video samples; Feature vector output module: used to input the sampled video frame into the segmentation attention network for deep feature extraction and output feature vector; Time feature extraction module: used to input the extracted feature vector into the bidirectional long short-term memory network to model time information and extract time features; Action category classification module: used to classify the results of the extracted time features using a softmax classifier into action categories, where the action with the highest score is the recognition result, and the action score is the recognition accuracy of the action; Among them, the sampled video frames are input into the segmentation attention network for deep feature extraction, including: Divide the sampled video frame into multiple basis arrays, and then divide each basis array into multiple partitions, perform convolution operation on the input video frame in each partition to obtain multiple feature map groups; The weights of different feature map groups in the channel dimension are calculated by using global context information; Calculate the channel feature vector in each basis array based on the weight of the channel feature vector; Each feature vector is concatenated along the channel dimension to obtain the output feature vector of the segmentation attention module.

Images