Indoor activity action detection algorithm based on adaptive fusion CBAM attention mechanism

Abstract

The invention belongs to the technical field of image processing and computer vision, provides an indoor activity action detection algorithm based on an adaptive fusion CBAM (adaptive fusion CBAM, AF-CBAM for short) attention mechanism, is mainly applied to a feature extraction part of a deep learning network, and aims to improve the action classification precision and detection capability of indoor activities. By introducing an AF-CBAM attention mechanism into a 2D CNN branch of a YOWO model, a traditional independent operation mode of space and channel attention is improved. Specifically, the weights of the space attention and the channel attention are dynamically adjusted through an adaptive fusion mechanism, so that the network can weight and enhance the spatial-temporal characteristics more flexibly, and the detection capability of actions in a complex indoor environment is improved. According to the method, a space and channel attention mechanism is added in a feature extraction process of a 2D CNN branch, important information of different regions and different channels can be effectively concerned, and the capability of capturing details is enhanced. Compared with a traditional single attention mechanism, the adaptive fusion attention mechanism can better adapt to different input data and task requirements, and the robustness and accuracy of the model in a complex dynamic environment are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and computer vision, and in particular relates to an indoor activity action detection algorithm based on an improved CBAM attention mechanism. Background Art

[0002] Indoor activity detection is an important research topic in computer vision and is widely used in smart home, health monitoring, smart security and other fields. This technology aims to automatically identify and classify various human actions in indoor environments through video analysis, which has important social and economic value. However, the task of indoor activity detection faces the challenges of complex environmental factors, such as lighting changes, object occlusion, and interference from multiple people, which often lead to a decrease in the accuracy of traditional methods.

[0003] With the development of deep learning technology, end-to-end models based on convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have gradually become mainstream, which can automatically extract spatial and temporal features in videos. Although these methods have improved the detection accuracy to a certain extent, existing models still face certain limitations in fine-grained action recognition and diverse indoor scenes, especially in capturing key features. To solve this problem, the attention mechanism was introduced into deep learning models. As an effective attention mechanism, the convolutional block attention module (CBAM) combines spatial and channel attention, which can enhance the model's attention to important areas and features and improve performance. However, in some scenarios, the original CBAM mechanism has dynamic effects on spatial and channel attention, and the traditional independent processing method may not be able to give full play to the complementary effects of the two.

[0004] Therefore, an adaptive fusion-based CBAM (AF-CBMA for short) attention mechanism is proposed. By introducing the adaptive fusion module, the weights of spatial and channel attention can be dynamically adjusted according to different tasks and input features, thereby enhancing the flexibility and robustness of the model. This improvement can effectively improve the accuracy of indoor activity detection and better adapt to complex environments in practical applications. Summary of the invention

[0005] In order to solve the shortcomings of existing indoor activity detection algorithms in feature extraction and action classification accuracy, this paper proposes an indoor activity detection algorithm based on adaptive fusion CBAM (AF-CBAM) attention mechanism. The algorithm is based on a deep convolutional neural network and improves the traditional CBAM model by introducing an adaptive fusion mechanism, so that it can dynamically adjust the fusion weights of spatial attention and channel attention, and improve the extraction capability and detection accuracy of spatiotemporal features.

[0006] In this algorithm, the continuous frames and key frames of the RGB video are first input into the temporal detection module and the spatial detection module. The temporal detection module extracts spatiotemporal features through a 2D convolutional network and combines the AF-CBAM mechanism to enhance the sensitivity to temporal information. In this process, AF-CBAM first calculates the attention of space and channels, and then dynamically adjusts the weights of space and channel attention through the adaptive fusion module. According to the difference in input features, the system can automatically choose to enhance the attention of space or channels, thereby optimizing the feature representation and improving the effect of spatiotemporal feature extraction.

[0007] In the spatial detection module, Darknet-19 is used to extract the spatial features of key frames, and the spatiotemporal features and spatial features are fused. Finally, the detection results are output through the bounding box regression and classification network, including action category, duration and location information. In order to achieve the above purpose, the technical solution of the present invention is as follows.

[0008] The indoor activity detection algorithm based on adaptive fusion CBAM attention mechanism includes the following steps.

[0009] Step 1: Obtain the indoor activity action recognition video dataset, and modify the indoor activity action dataset according to the AVA dataset annotation format so that it can run successfully in the YOWO network structure.

[0010] Step 2: In the time series detection module, a deep convolutional network is used to extract spatiotemporal features, and 2D convolution is combined to extract time series information. The AF-CBAM module is introduced to improve the model's sensitivity to complex actions by adaptively adjusting the fusion weights of spatial and channel attention.

[0011] Step 3: In the spatial detection module, Darknet-19 is used to extract key frame features, and AF-CBAM is used for adaptive fusion to further enhance the representation ability of spatiotemporal features and improve the accuracy of action recognition.

[0012] Step 4: Through the bounding box regression and classification network, the detection results of indoor activity actions are output, including action category, duration and location information.

[0013] The specific situation in step 1 is as follows: (1) Obtain a public indoor activity dataset (Charades_v1_480) and divide the video to be detected into 30 consecutive frames; (2) Use YOLOV5 to identify the target person in each frame and delete the annotation boxes of irrelevant people; (3) Use the VIA annotation tool to manually annotate the target person's actions, annotate the target person's actions and spatial position information in each frame; process the action labels annotated by experts as network labels. According to the expert annotations, twelve basic actions are classified as: Stand, Lie-down, Sit, Tidy, Cook, Play-phone, Call, Eat-Drink, Watch-TV, Open-Close-door, Hand-Shakes-Hugs, and Fall-Down; (4) The images cropped from the video and the coordinates detected by YOLOV5 are sent to Deepsort for detection to associate the ID of the target person.

[0014] The specific situation in the step 2 is as follows: In the temporal detection module, a 2D convolutional network is used to extract the spatiotemporal features of continuous video frames. In order to improve the sensitivity to channel features, the present invention introduces an adaptive fusion CBAM (AF-CBAM) attention mechanism.

[0015] (1) First, spatial attention and channel attention are calculated. Spatial attention adjusts the weight of the feature map according to the importance of each position, while channel attention adjusts the weight of each channel according to the contribution of the channel; (2) Using the adaptive fusion module, the fusion ratio of spatial and channel attention is dynamically adjusted according to the current task and input characteristics, so that the model can automatically enhance spatial or channel information according to actual needs.

[0016] The specific situation in step three is as follows: The 2D CNN branch is used to extract spatial features, and the spatial information of the target person is analyzed through the Darknet-19 network model. The Darknet-19 network model achieves effective learning of features by alternating between 3×3 and 1×1 filters. The initial feature size of the input is C*H*W, and features are extracted through multiple convolutional modules consisting of 32, 64, 128, 256, 512, 1024 and a specific output filter number of 4. The maximum pooling layer, normalization operation BN and activation function LeakyReLU are embedded between the convolutional modules.

[0017] The specific situation in the step 4 is as follows: First, the spatiotemporal features and spatial features extracted by the temporal detection module and the spatial detection module are fused through the channel stacking technology to ensure that the outputs of the two-dimensional and three-dimensional networks have the same shape in the last two dimensions. Then, the fused feature map is input into the adaptive fusion CBAM (AF-CBAM) module, and the self-attention mechanism is used to highlight the relationship between the previous and next frames and enhance the distinguishability of the features. Finally, through the border regression idea of ​​YOLO, a 1x1 convolution kernel is used to generate the number of target channels, and the coordinate box, behavior category and confidence of the target are output to achieve action detection and positioning.

[0018] Compared with the prior art, the technical solution of the present invention has the following beneficial effects: (1) By introducing the adaptive fusion CBAM (AF-CBAM) mechanism and dynamically adjusting the fusion weights of spatial and channel attention, the temporal information and spatial features can be captured more accurately, thereby effectively improving the model's ability to extract spatiotemporal features in complex indoor activity scenes.

[0019] (2) The channel stacking and 1x1 convolution kernel processing methods enable the model to reduce the amount of calculation and improve the detection speed while ensuring high detection accuracy, thus adapting to the application requirements of real-time monitoring and resource-constrained environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is the overall flow chart of the present invention.

[0021] Figure 2 This is the improved YOWO network model used in the present invention.

[0022] Figure 3 The improved adaptive fusion CBAM attention mechanism module used in the present invention. DETAILED DESCRIPTION

[0023] For those skilled in the art, some well-known structures and their descriptions in the drawings may be omitted. The technical solution of the present invention is further described below in conjunction with the drawings and embodiments.

[0024] The present invention provides an indoor activity action detection algorithm based on an adaptive fusion CBAM attention mechanism. This method significantly improves the ability of the 2D CNN branch to process complex data, enhances the spatial perception ability of the model, and also improves the model's ability to extract shallow and deep features.

[0025] Figure 1This is the flow chart of the method of the present invention. First, the video is preprocessed to modify the Charades_v1_480 dataset into the input and annotation files required by the network; then an improved YOWO network model is constructed and used to train the video data. Secondly, the weight files that appear in the training are evaluated, and the network weights with the best effect are saved to realize video action detection.

[0026] Figure 2 The YOWO network model of the present invention includes two main branches: 2D CNN branch and 3D CNN branch. Among them, the 2D CNN branch introduces an improved adaptive fusion CBAM (AF-CBAM) mechanism, which enhances the ability to extract spatial features and improves the performance of the model in complex indoor activity detection.

[0027] Figure 3 The improved adaptive fusion CBAM attention mechanism module of the present invention adaptively fuses the spatial and channel attentions and dynamically adjusts their fusion weights to enhance the expression of important spatial and channel information in the feature map. The module first highlights important areas through the spatial attention mechanism, and then increases the weights of key channels through the channel attention mechanism, thereby improving the model's ability to recognize complex indoor activities.

[0028] The specific implementation steps are: Step 1.1 Obtain a public indoor activity dataset (Charades_v1_480) and divide the video to be detected into 30 consecutive frames; Step 1.2 Use YOLOV5 to identify the target person in each frame and delete the annotation boxes of irrelevant people; Step 1.3 Use VIA annotation tools to manually annotate the target person's actions, annotate the target person's actions and spatial position information in each frame; process the action labels annotated by experts as network labels. According to the expert annotations, the twelve basic actions are classified as: Stand, Lie-down, Sit, Tidy, Cook, Play-phone, Call, Eat-Drink, Watch TV, Open-Close-door, HandShakes-Hugs, and Fall-Down; Step 1.4 Send the image cropped from the video and the coordinates detected by YOLOV5 to Deepsort for detection to associate the ID of the target person.

[0029] Step 2.1 uses 2D CNN branches to extract spatial features, and uses the improved adaptive fusion CBAM (AF-CBAM) module to enhance the extraction capability of spatial features and improve the model's recognition effect on complex indoor activity scenes; Step 2.2 The fusion of spatial attention and channel attention is dynamically adjusted through an adaptive mechanism, so that the model can adaptively select the optimal fusion method according to different tasks and input features to enhance the distinguishability of features; Step 2.3 The processed spatiotemporal features and spatial features are fused by channel stacking to ensure that the shapes of the two-dimensional and three-dimensional features are consistent in the last two dimensions, and then the fused features are passed to the CFAM module for further processing.

[0030] Step 3.1 Use 2D CNN branches to extract spatial features and analyze the spatial information of the target person through the Darknet-19 network model; Step 3.2 The Darknet-19 network model achieves effective feature learning by alternating between 3×3 and 1×1 filters. The initial feature size of the input is C*H*W. Features are extracted through multiple convolutional modules consisting of 32, 64, 128, 256, 512, 1024 and a specific number of output filters, and the maximum pooling layer, normalization operation BN and activation function LeakyReLU are embedded between the convolutional modules.

[0031] Step 4.1 uses the channel stacking feature to fuse the two feature maps, as shown in Formula 1: (1) Among them, A is the feature map of 2D CNN and 3D CNN fusion. However, since such fusion has no connection relationship, it is necessary to connect the feature maps together through a convolutional layer, as shown in Formula 2: (2) Among them, B is the newly generated feature map, and F is the reshaped tensor, that is, the feature vector of each channel is quantized into one dimension: The information is then passed as input to the CFAM module. This module maps the dependencies between channels based on the Gram matrix. The self-attention mechanism can highlight the relationship between the previous and next frames and enhance the distinguishability of features, as shown in Formula 3. (3) Among them, G is the Gram matrix, G ij represents the inner product between the vectorized feature maps i and j.

[0032] Step 4.2 follows YOLO’s idea of ​​bounding box regression and uses a 1*1 convolution kernel in the last layer to generate the target channel number. Finally, the coordinate box information, behavior category and confidence of the target person are output.

Claims

1. An indoor activity detection algorithm based on the Adaptive Fusion CBAM (AF-CBAM) attention mechanism, characterized by: The following steps are involved: Step 1: Obtain the indoor activity action recognition video dataset, and modify the indoor activity dataset according to the AVA dataset annotation format so that it can run successfully in the YOWO network structure; Step 2: In the time series detection module, a deep convolutional network is used to extract spatiotemporal features, and 2D convolution is combined to extract time series information. The AF-CBAM module is introduced to adaptively adjust the fusion weights of spatial and channel attention; Step 3: In the spatial detection module, Darknet-19 is used to extract key frame features, and AF-CBAM is used for adaptive fusion to further enhance the representation ability of spatiotemporal features; Step 4: Through the bounding box regression and classification network, the detection results of indoor activity actions are output, including action category, duration and location information.

2. The indoor activity detection algorithm based on the AF-CBAM attention mechanism according to claim 1 is characterized in that: The specific process in Step 1 is as follows: Step 1.1 Obtain a public indoor activity dataset (Charades_v1_480) and divide the video to be detected into 30 consecutive frames; Step 1.2 Use YOLOV5 to identify the target person in each frame and delete the annotation boxes of irrelevant people; Step 1.3 Use the VIA annotation tool to manually annotate the target person’s actions, annotate the action and spatial position information of each frame of the target person; process the action labels annotated by experts as network labels; Step 1.4 Send the cropped image from the video and the coordinates detected by YOLOV5 to Deepsort for detection to associate the ID of the target person.

3. The figure skating action detection algorithm based on the improved 3DSE attention mechanism according to claim 1 is characterized in that: The specific process in Step 2 is as follows: Step 2.1 uses 2D CNN branches to extract spatial features, and uses the improved AF-CBAM attention mechanism module to enhance the extraction capability of spatial features and improve the model's recognition effect on complex indoor activity scenes; Step 2.2 dynamically adjusts the fusion of spatial attention and channel attention through an adaptive mechanism, and fuses the processed spatiotemporal features and spatial features through channel stacking to ensure that the shapes of two-dimensional and three-dimensional features are consistent in the last two dimensions. The fused features are then passed to the CFAM module for further processing.

4. The indoor activity detection algorithm based on the AF-CBAM attention mechanism according to claim 1 is characterized in that: The specific process in Step 3 is as follows: Step 3.1 uses 2D CNN branches to extract spatial features and analyzes the spatial information of the target person through the Darknet-19 network model; Step 3.2 The Darknet-19 network model achieves effective feature learning by alternating between 3×3 and 1×1 filters. The initial feature size of the input is C*H*W. Features are extracted through multiple convolutional modules consisting of 32, 64, 128, 256, 512, 1024 and a specific number of output filters, and the maximum pooling layer, normalization operation BN and activation function LeakyReLU are embedded between the convolutional modules.

5. The indoor activity detection algorithm based on AF-CBAM attention mechanism according to claim 1 is characterized in that: The specific process in Step 4 is as follows: Step 4.1 The outputs of the 2D and 3D networks have the same shape in the last two dimensions, and the channel stacking feature is used to fuse the two feature maps; Step 4.2 The self-attention mechanism can highlight the relationship between the previous and next frames and enhance the distinguishability of features; Step 4.3 follows YOLO’s idea of ​​bounding box regression. The last layer uses a 1*1 convolution kernel to generate the target channel number and outputs the target person’s coordinate box information, behavior category, and confidence.

Images