Lightweight behavior recognition method
Through the MobileNetV3 network with TSN sparse sampling, SE attention mechanism and channel pruning optimization, the problem of high computational cost of existing behavior recognition methods is solved, and efficient and accurate behavior recognition is achieved, suitable for real-time application on resource-constrained devices. .
Patent Information
- Application Number
- CN202411644113.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-18
- Publication Date
- 2025-01-10
AI Technical Summary
Existing behavior recognition methods rely on complex deep learning networks, resulting in high computing costs and difficulty in operating efficiently on devices with limited computing power, and cannot meet the real-time monitoring needs of low power consumption and low latency.
The video data is time-domain segmented by TSN sparse sampling strategy, extracted optical flow image features and RGB image sequences, and feature extraction is performed using MobileNetV3 network, and lightweight processing is performed through SE attention mechanism and channel pruning technology, combined with segmented consensus The strategy and the maximum pooling method perform feature fusion.
It realizes a lightweight model, reduces computing overhead and inference time, improves the accuracy and real-time nature of behavior recognition, and is suitable for resource-constrained devices and scenarios.
Smart Images
Figure CN119296179A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of behavior recognition and relates to a lightweight behavior recognition method. Background Art
[0002] With the rapid development of domestic economy and technology, smart phones, high-definition cameras and other devices with shooting functions can be seen everywhere in daily life. People are happy to use these devices to shoot videos and record every bit of life. In the field of security, in order to protect people's lives and property and prevent the occurrence of criminal incidents, a large number of high-definition monitoring equipment are installed in public places such as schools, shopping malls, and streets. These devices are generating massive amounts of video data every moment.
[0003] Behavior recognition technology is an important application of computer vision. It can automatically identify and analyze human behavior in videos, thereby helping to quickly and efficiently screen out abnormal situations, provide real-time warnings, reduce missed reports, and maintain public safety at a lower cost. However, most current behavior recognition methods rely on complex deep learning networks, and the models are relatively large and not lightweight enough, resulting in high computing costs. In particular, they cannot run efficiently on devices with limited computing power, and it is difficult to meet the needs of low-power, low-latency real-time monitoring. Therefore, improving the lightweight level of behavior recognition methods, both to ensure recognition accuracy and to reduce computing overhead, has become the key to improving their application effect in monitoring scenarios. Summary of the invention
[0004] In view of this, the purpose of the present invention is to provide a lightweight behavior recognition method to effectively identify the behavior of people in the video, replace manual patrols, reduce labor costs, and make full use of video information to identify dangerous behaviors.
[0005] In order to achieve the above object, the present invention provides the following technical solutions:
[0006] A lightweight behavior recognition method, the method comprising:
[0007] S1, input the collected video data and perform preprocessing;
[0008] S2. Perform time domain segmentation on the preprocessed video data through the TSN sparse sampling strategy, and then extract the optical flow image features and the RGB image sequence; wherein the optical flow image features can be extracted by a dense optical flow algorithm, and the RGB image sequence can be extracted by OpenCV;
[0009] S3, input the obtained optical flow image features and RGB image sequence into the MobileNetV3 network for feature extraction, use the SE attention mechanism to weight the features, and use channel pruning to lightweight the entire network;
[0010] S4. A segmented consensus strategy is used to fuse the features extracted from multiple time periods, and the maximum pooling method is used to combine the classification results of temporal features and spatial features to complete behavior recognition.
[0011] Further, step S1, preprocessing the input video data includes: performing convolution noise reduction in the spatial domain on the current frame and the adjacent frames; aligning the current frame with the adjacent frames through the optical flow network; performing temporal noise reduction on the aligned image;
[0012] In order to reduce the changes between adjacent frames and ensure the smooth transition of the denoised video, the loss function shown in the following formula is used:
[0013]
[0014] Where, L t Represents the final noise loss, which is used to measure the denoising effect of each frame; λ t represents the spatiotemporal regularization loss weight
[0015] Coefficient, used to ensure consistency between adjacent frames; N represents the total number of video frames; λ s represents the spatial loss weight coefficient; represents the t-th frame image optimized during the iteration, x t represents the input original denoised image.
[0016] Further, in step S2, the pre-processed video data is segmented in the time domain by using the TSN sparse sampling strategy, including dividing the pre-processed video data into K segments {S 1 ,S 2 ,S 3 ,…,S K}; then model a series of fragments as follows:
[0017] TSN 1 ,T 2 ,...,T K )=H(G(F(T 1 ; W), F(T 2 ;W),...,F(T K ;W))·W t )
[0018] In the formula, (T 1 ,T 2 ...T K ) represents a fragment sequence, each fragment T K From the corresponding segment S K F(T K ; W) function represents the convolutional network with W as parameter acting on the short segment T K, return T K Relative to the score of all categories; W t represents the time weight parameter; the segment consensus function G combines the category score outputs of multiple short clips to obtain the consensus on the category hypothesis between the short clips. Based on this consensus, the prediction function H predicts the probability that the entire video belongs to each behavior category;
[0019] Combined with the standard classification cross entropy loss, the loss function for partial consensus is in the form of:
[0020]
[0021] In the formula, C represents the total number of behavior categories, y i represents the groundtruth of category i; λ represents a hyperparameter used to control the weight of the regularization term in the loss; R(G) represents a regularization term used to prevent overfitting.
[0022] Furthermore, in step S3, the SE attention mechanism is used to perform weighted processing on the features, and global average pooling is used to globally aggregate the features of each channel and dynamically adjust the shape of the activation function during the training process to optimize the amount of calculation and model performance; the activation function is expressed as:
[0023]
[0024] Where α is the expansion or compression factor used to adjust the input x scaling; β is the translation factor used to control the offset of the function; and γ is the scaling factor used to adjust the output amplitude.
[0025] Furthermore, in step S3, channel pruning is used to lighten the entire network. Specifically, the activation value generated by the channel during the forward propagation of the network is analyzed by pruning the activation value, the importance of the channel is judged, and the channel with low activation value is pruned; wherein the pruning function is shown in the following formula:
[0026]
[0027] In the formula, A c represents the average activation value of channel c, Y i,j,c represents the activation value of channel c at position (i, j), H and W represent the height and width of the feature map respectively.
[0028] Furthermore, step S4 is specifically as follows: after the segmented sampled optical flow image features and the RGB image sequence are input into the MobileNetV3 network for feature extraction, the temporal flow features and the spatial flow features are obtained, and the features for multiple time periods obtained by different networks are fused using a segmented consensus strategy to obtain a stable behavior recognition result, wherein the segmented consensus strategy is shown in the following formula:
[0029]
[0030] In the formula, G′ represents the final feature fusion result, N represents the number of time periods into which the video is segmented; f t represents the feature vector extracted in the tth time period; e represents the query vector, which comes from the global information or the output of the previous layer; k t represents the key vector of the tth time period, indicating the feature importance of this time period; a and b represent learnable parameters;
[0031] The fusion features obtained by the segmented consensus strategy are combined with the classification results of the temporal features and spatial features using the maximum pooling to finally obtain the final behavior recognition result; the maximum pooling function is shown in the following formula:
[0032]
[0033] Where W i,j represents a rectangular area, p and q represent positions, and α p,q Represents the weight coefficient of each position in the pooling window. The classification results of the above time domain features and spatial features are combined using the maximum pooling method to obtain the final behavior recognition features.
[0034] The beneficial effects of the present invention are:
[0035] (1) Reduce computational overhead and achieve lightweight models: The TSN sparse sampling strategy is used to segment video data in the time domain, extract features from multiple time periods of the video, and reduce the computational complexity of processing each video frame. At the same time, the channel pruning technology is used in MobileNetV3 to remove redundant channels, making the network more lightweight, reducing the amount of computation and storage requirements, and facilitating deployment on resource-constrained devices.
[0036] (2) Enhance the model’s behavior recognition capability: By adopting the SE attention mechanism, the model can adaptively focus on important features and weaken unimportant features, thereby increasing the sensitivity to key behavior features and improving the model’s classification accuracy in video behavior recognition tasks.
[0037] (3) Efficient feature fusion to improve recognition accuracy: The segmented consensus strategy is used to fuse the features extracted from multiple time periods, effectively capturing the temporal information in the video and the feature changes of different frames, and using maximum pooling to integrate the temporal and spatial domain features. The final classification result combines multi-dimensional information, improves the model's ability to recognize complex behaviors, and ensures high accuracy.
[0038] (4) Model inference speed is accelerated and applicability is wider: Through optimization measures such as TSN sparse sampling, lightweight MobileNetV3 and channel pruning, the model inference time is significantly reduced, allowing the model to be efficiently applied in real-time or near real-time scenarios. It is suitable for scenarios with high speed requirements such as video surveillance, smart devices and embedded devices.
[0039] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:
[0041] Figure 1 A schematic diagram of the process of a lightweight behavior recognition method provided by an embodiment of the present invention;
[0042] Figure 2 The structure of a lightweight behavior recognition deep network provided by an embodiment of the present invention;
[0043] Figure 3 This is the structure of the SE-block attention mechanism module in the MobileNetV3 network. DETAILED DESCRIPTION
[0044] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.
[0045] like Figure 1 As shown, a lightweight behavior recognition method is provided in one embodiment of the present invention, and the method is specifically described as follows:
[0046] S1: Input the commonly used behavior classification UCF101 public dataset, and perform preprocessing operations on the public dataset, such as video denoising, compression, etc. For example, in this embodiment, the video denoising algorithm FastDVDNet can be used for preprocessing, and the processing process includes: the first step is to perform convolution denoising on the current frame and the adjacent frames in the spatial domain; the second step is to align the current frame with the adjacent frames through the optical flow network; the third step is to perform temporal denoising on the aligned image.
[0047] Considering reducing the changes between adjacent frames and ensuring the smooth transition of the denoised video, a video denoising algorithm that reduces the temporal and spatial regularization loss and spatial loss is proposed, where the loss function is expressed as:
[0048]
[0049] Where, L t is the final noise loss, used to measure the denoising effect of each frame, λ t is the spatiotemporal regularization loss weight coefficient, which is used to ensure the consistency between adjacent frames, N is the total number of video frames, and λ s is the spatial loss weight coefficient, is the t-th frame image optimized during the algorithm iteration, x t is the original denoised image of the input frame t.
[0050] S2: Use the TSN sparse sampling strategy to perform time domain segmentation on the denoised video data, and then sparsely sample a series of short clips from the entire video to obtain optical flow image features and RGB image sequences.
[0051] After obtaining the denoised video data, the TSN sparse sampling strategy is used to perform time domain segmentation on the video data. It sparsely samples a series of short clips from the entire video, and then begins to extract the optical flow and RGB image frames in the video data. The dense optical flow (Horn-Schunck) algorithm is used. Based on the fact that the optical flow in the image is smooth (that is, the optical flow of adjacent pixels changes little), the motion of each pixel is calculated through global optimization, and the optical flow in the image sequence is estimated by minimizing the overall energy function of the optical flow constraint equation. The energy function is shown as follows:
[0052]
[0053] In the formula, ε is the weight of the control smoothing term, I x ,I y is the gradient of the image, which represents the local change of the image in the x and y directions. t is the time gradient of the image, which indicates the change of the image on the time axis; It is a smoothing term used to constrain the smoothness of the optical flow. Specifically, it represents the sum of the squared gradients of the optical flow in space, where u and v are the optical flow components of each pixel.
[0054] The TSN (Temporal Segment Networks) network is an end-to-end two-stream network. It is an action recognition network that improves the sampling method of the input video sequence of the original two-stream network. The essence of TSN is to divide the input video into K segments, and then randomly sample a short segment from each segment. The category scores of different segments are fused using the segment consensus function to generate a segment consensus, which is a video-level prediction. Then the predictions of all modes are fused to produce the final prediction result.
[0055] Specifically, given a public dataset video V, divide it into K segments {S 1 ,S 2 ,S 3 ,…,S K TSN then models the series of fragments as follows:
[0056] TSN 1 ,T 2 ,...,T K )=H(G(F(T 1 ; W), F(T 2 ;W),...,F(T K ;W))·W t )
[0057] Among them, (T 1 ,T 2 ...T K ) represents a fragment sequence, each fragment T K From its corresponding segment S K F(T K ; W) function represents the convolutional network with W as parameter acting on the short segment T K , the function returns T K Relative to the scores of all categories, W t is the time weight parameter. The segment consensus function G combines the category score outputs of multiple short clips to obtain a consensus between them on the category hypothesis. Based on this consensus, the prediction function H predicts the probability that the entire video belongs to each behavior category (in this embodiment, H selects the Softmax function). Combined with the standard classification cross entropy loss, the final loss function for partial consensus is in the form of:
[0058]
[0059] Among them, C is the total number of behavior categories, yi is the groundtruth of category i, λ is a hyperparameter used to control the weight of the regularization term in the loss, and R(G) is a regularization term used to prevent overfitting.
[0060] S3: The optical flow image features and RGB image sequences after segmentation sampling are input into the MobileNetV3 network for feature extraction, the features are weighted using the SE attention mechanism, and channel pruning is used to lightweight the entire network.
[0061] The MobileNetV3 network is a lightweight behavior recognition deep network. Thanks to the sub-networks in the network, the network can effectively recognize behaviors with long-term structures and learn the correlation between behavior sub-features, such as Figure 2 Identify the structure of the deep network for this lightweight behavior.
[0062] Among them, the SE-block attention mechanism module structure in the MobileNetV3 network is as follows Figure 3 As shown in the figure, the features are weighted by the SE-block attention mechanism, as follows:
[0063] During the training process, global average pooling is used to globally aggregate the features of each channel and dynamically adjust the shape of the activation function to optimize the amount of calculation and model performance. The A-switch activation function is used as shown in the following formula:
[0064]
[0065] Among them, α represents the expansion or compression factor, which is used to adjust the input x scaling; β is the translation factor, which is used to control the offset of the function; γ is the scaling factor, which is used to adjust the amplitude of the output.
[0066] Channel pruning is used to lighten the entire network structure. The activation values generated by the channels during the network forward propagation are analyzed by pruning the activation values to determine the importance of the channels. Channels with low activation values are usually considered to contribute less to the final behavior recognition results and can be pruned. The pruning function is as follows:
[0067]
[0068] Among them, A c represents the average activation value of channel c, Y i,j,c represents the activation value of channel c at position (i, j), H and W are the height and width of the feature map respectively.
[0069] S4: A segmented consensus strategy is used to fuse the features extracted from multiple time periods, and the maximum pooling method is used to combine the classification results of temporal and spatial features.
[0070] After the segmented and sampled optical flow image features and RGB image sequences are input into the MobileNetV3 network for feature extraction, the temporal flow features and spatial flow features are obtained. The segmented consensus is used to fuse the features obtained by different networks for multiple time periods to obtain a stable behavior recognition result. The segmented consensus strategy is as follows:
[0071]
[0072] Among them, G is the final feature fusion result, and N is the number of time periods into which the video is divided. t is the feature vector extracted in the tth time period; e is the query vector, which comes from the global information or the output of the previous layer; k t is the key vector of the tth time period, indicating the importance of the features in this time period. a and b are the learnable parameters obtained through training.
[0073] The fusion features obtained by the segmented consensus strategy are combined with the classification results of the temporal features and spatial features using the maximum pooling to finally obtain the final behavior recognition result. The maximum pooling function is as follows:
[0074]
[0075] Among them, W i,j represents a rectangular area, p and q represent positions, and α p,q is the weighting coefficient for each position in the pooling window, which is usually initialized to 1, but can be adjusted by backpropagation during training.
[0076] In summary, the present invention achieves efficient and high-precision behavior recognition by combining the TSN sparse sampling strategy, SE attention mechanism and channel pruning optimization lightweight MobileNetV3 network. Through efficient data processing and feature fusion means, the problems of low recognition accuracy and poor real-time performance in traditional methods are solved, so that the present invention has good real-time performance while maintaining accuracy, and is suitable for deployment and application in resource-constrained environments.
[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.
Claims
1. A lightweight behavior recognition method, characterized in that: The method comprises: inputting collected video data and performing preprocessing; The pre-processed video data is segmented in the time domain through the TSN sparse sampling strategy, and then the optical flow image features and RGB image sequence are extracted; The obtained optical flow image features and RGB image sequences are input into the MobileNetV3 network for feature extraction, the features are weighted using the SE attention mechanism, and the entire network is lightweight using channel pruning; A segmented consensus strategy is used to fuse the features extracted from multiple time periods, and the maximum pooling method is used to combine the classification results of temporal features and spatial features to complete behavior recognition.
2. The lightweight behavior recognition method according to claim 1, characterized in that: Preprocessing the input video data includes: performing spatial convolution noise reduction on the current frame and adjacent frames; aligning the current frame with adjacent frames through an optical flow network; and performing temporal noise reduction on the aligned images. In order to reduce the changes between adjacent frames and ensure the smooth transition of the denoised video, the loss function shown in the following formula is used: Where, L t Represents the final noise loss, which is used to measure the denoising effect of each frame; λ t represents the spatiotemporal regularization loss weight coefficient, which is used to ensure the consistency between adjacent frames; N represents the total number of video frames; λ s represents the spatial loss weight coefficient; represents the t-th frame image optimized during the iteration, x t represents the input original denoised image.
3. The lightweight behavior recognition method according to claim 1, characterized in that: The time domain segmentation of the pre-processed video data by using the TSN sparse sampling strategy includes dividing the pre-processed video data into K segments {S1, S2, S3, ..., S K }; then model a series of fragments as follows: TSN(T1,T2,...,T K )=H(G(F(T1;W),F(T2;W),...,F(T K ;W))·W t ) In the formula, (T1, T2...T K ) represents a fragment sequence, each fragment T K From the corresponding segment S K F(T K ; W) function represents the convolutional network with W as parameter acting on the short segment T K , return T K Relative to the score of all categories; W t represents the time weight parameter; the segment consensus function G combines the category score outputs of multiple short clips to obtain the consensus on the category hypothesis between the short clips. Based on this consensus, the prediction function H predicts the probability that the entire video belongs to each behavior category; Combined with the standard classification cross entropy loss, the loss function for partial consensus is in the form of: In the formula, C represents the total number of behavior categories, y i represents the groundtruth of category i; λ represents a hyperparameter used to control the weight of the regularization term in the loss; R(G) represents a regularization term used to prevent overfitting.
4. The lightweight behavior recognition method according to claim 1, characterized in that: The optical flow image features are extracted by a dense optical flow algorithm, and the RGB image sequence is extracted by OpenCV.
5. The lightweight behavior recognition method according to claim 1, characterized in that: The method of using the SE attention mechanism to perform weighted processing on the features includes: using the SE attention mechanism to perform weighted processing on the features, using global average pooling to globally aggregate the features of each channel during the training process and dynamically adjusting the shape of the activation function to optimize the amount of calculation and model performance; the activation function is expressed as: Where α is the expansion or compression factor used to adjust the input x scaling; β is the translation factor used to control the offset of the function; and γ is the scaling factor used to adjust the output amplitude.
6. The lightweight behavior recognition method according to claim 1, characterized in that: The method of using channel pruning to lighten the entire network includes: analyzing the activation values generated by the channels during the forward propagation of the network by pruning the activation values, judging the importance of the channels, and pruning the channels with low activation values; wherein the pruning function is shown in the following formula: In the formula, A c represents the average activation value of channel c, Y i,j,c represents the activation value of channel c at position (i, j), H and W represent the height and width of the feature map respectively.
7. The lightweight behavior recognition method according to claim 1, characterized in that: The segmented consensus strategy is used to fuse the features extracted from multiple time periods, and the classification results of the maximum pooling method combined with the temporal features and the spatial features include: After the segmented sampled optical flow image features and RGB image sequence are input into the MobileNetV3 network for feature extraction, the temporal flow features and spatial flow features are obtained. The segmented consensus strategy is used to fuse the features obtained by different networks for multiple time periods to obtain stable behavior recognition results. The segmented consensus strategy is shown in the following formula: In the formula, G′ represents the final feature fusion result, N represents the number of time periods into which the video is segmented; f t represents the feature vector extracted in the tth time period; e represents the query vector, which comes from the global information or the output of the previous layer; k t represents the key vector of the tth time period, indicating the feature importance of this time period; a and b represent learnable parameters; The fusion features obtained by the segmented consensus strategy are combined with the classification results of the temporal features and spatial features using the maximum pooling to finally obtain the final behavior recognition result; the maximum pooling function is shown in the following formula: Where W i,j represents a rectangular area, p and q represent positions, and α p,q Represents the weighting coefficient for each position within the pooling window.