A fast behavior detection method based on attention mechanism

Abstract

The present invention belongs to the technical fields of image processing and computer vision, and specifically relates to a fast behavior detection method based on an attention mechanism. The method comprises an end-to-end real-time YOWO network, wherein continuous frames of RGB video and RGB keyframe information are respectively input into a temporal detection module and a spatial detection module for processing. In the spatial detection module, a feature extractor with a residual module is added to output corresponding features. The features are processed through a spatial domain attention mechanism to obtain more fine-grained attention mechanism features. The features of the two branches are concatenated and fused through a convolutional layer. Finally, the fused features are detected to obtain the category, duration, and location information of the target behavior in the video. The method solves the problem of small gaps between behavior classes while ensuring real-time performance and a small size, and effectively improves the accuracy of spatiotemporal behavior detection.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and computer vision technology, and specifically relates to a fast behavior detection method based on an attention mechanism. Background Art

[0002] Spatiotemporal behavior detection, a key research area in computer vision, primarily involves detecting the target's behavior category in a video and locating the time and location of the behavior. With the widespread adoption of surveillance cameras and mobile photography, the demand for spatiotemporal behavior detection is increasing, and it is widely used in various scenarios and fields, including abnormal behavior detection, quality control, and airports. However, the difficulty in acquiring long-term temporal information from videos, coupled with the increasing volume of video, has led to current mainstream spatiotemporal behavior detection methods being large and slow.

[0003] Existing methods for spatiotemporal behavior detection are primarily divided into two-stream networks and networks based on three-dimensional convolution, and have achieved significant improvements in recent years. However, the optical flow in two-stream networks only contains short-term temporal information, which is not ideal for modeling long-term videos and cannot effectively distinguish between behavior categories with small inter-class differences. While three-dimensional convolution has achieved significant improvements in extracting temporal information, it consumes a large amount of computing resources, has a large model size, and has a slow detection speed, making it difficult to widely apply. Improving the accuracy and speed of spatiotemporal behavior detection while reducing the network size is of great research significance. Summary of the Invention

[0004] To solve the above technical problems, the present invention proposes a fast behavior detection method based on an attention mechanism, comprising the following steps:

[0005] S1: Obtain the RGB video to be detected containing interactive information, and divide the RGB video to be detected into segments of K consecutive frames;

[0006] S2: Building a fast behavior detection model using the end-to-end spatiotemporal behavior detection network YOWO;

[0007] The fast behavior detection model includes: a time sequence detection module, a space detection module and an interaction processing module;

[0008] The key frame is the first frame of the continuous video frames;

[0009] S3: The time series detection module of the fast behavior detection model performs time series feature detection on the interaction information and K consecutive frames to obtain time series features;

[0010] S4: A residual module is added to the DarkNet-19 feature extractor in the spatial detection module of the fast behavior detection model to obtain an improved DarkNet-19 feature extractor. The improved DarkNet-19 feature extractor is used to extract features from interaction information and key frames to obtain a feature map of multi-scale target information.

[0011] S5: Implement the attention mechanism in the spatial domain based on the feature map of multi-scale target information to obtain the attention mechanism features in the spatial domain;

[0012] S6: Treat the temporal features and the attention mechanism features of the spatial domain as n non-overlapping feature images, and perform splicing and fusion to obtain the fused features;

[0013] S7: The interaction processing module of the fast behavior detection model processes the target behavior information and its interaction information to obtain the second-order relationship between the target information, the interaction information and the environment information;

[0014] S8: Detect the second-order relationship between the fused features, target information, interaction information and environmental information to obtain the spatiotemporal behavior detection results of the target in the video.

[0015] Beneficial effects of the present invention:

[0016] The present invention adopts a dual-stream end-to-end real-time spatiotemporal behavior detection YOWO network, which achieves a good trade-off between speed and accuracy. The residual module effectively solves the problem of poor long-term modeling effect of the dual-stream network while retaining the original advantage of the YOWO network's smaller size. The spatial domain attention mechanism module locates the target more quickly and accurately, effectively improving the accuracy of the network. At the same time, through the feature fusion mechanism of continuous frames and key frames, and using interaction information to obtain second-order information of the target and the environment, the features can obtain information in the spatial and temporal domains, which has a better identification effect on similar behaviors. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a process framework diagram of the present invention;

[0018] Figure 2 Schematic diagram of the spatial domain attention mechanism structure of the present invention. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0020] A fast behavior detection method based on attention mechanism, such as Figure 1 Shown, including:

[0021] S1: Obtain the RGB video to be detected containing interactive information, and divide the RGB video to be detected into segments of K consecutive frames;

[0022] S2: Building a fast behavior detection model using the end-to-end spatiotemporal behavior detection network YOWO;

[0023] The fast behavior detection model includes: a time sequence detection module, a space detection module and an interaction processing module;

[0024] The key frame is the first frame of the continuous video frames;

[0025] S3: The time series detection module of the fast behavior detection model performs time series feature detection on the interaction information and K consecutive frames to obtain time series features;

[0026] S4: A residual module is added to the DarkNet-19 feature extractor in the spatial detection module of the fast behavior detection model to obtain an improved DarkNet-19 feature extractor. The improved DarkNet-19 feature extractor is used to extract features from interaction information and key frames to obtain a feature map of multi-scale target information.

[0027] S5: Implement the attention mechanism in the spatial domain based on the feature map of multi-scale target information to obtain the attention mechanism features in the spatial domain;

[0028] S6: Treat the temporal features and the attention mechanism features of the spatial domain as n non-overlapping feature images, and perform splicing and fusion to obtain the fused features;

[0029] S7: The interaction processing module of the fast behavior detection model processes the target behavior information and its interaction information to obtain the second-order relationship between the target information, the interaction information and the environment information;

[0030] S8: Detect the second-order relationship between the fused features, target information, interaction information and environmental information to obtain the spatiotemporal behavior detection results of the target in the video.

[0031] For ease of explanation, this example uses a 224×224 size. For the original RGB video, all video frames are proportionally resized to 224×224. K frames are fed into the temporal detection module as a segment, with the first frame of the K frames serving as the keyframe for that segment and feeding it into the spatial detection module. The entire video segment is traversed using a sliding window with a step size of n, which serves as the network input.

[0032] Figure 2This is a schematic diagram of the spatial domain attention mechanism structure of this embodiment. Figure 2 As shown in Figure 3, the spatial domain attention mechanism consists of a generation attention mechanism parameter branch and an original feature branch, thereby obtaining more accurate location information and accelerating the training convergence process.

[0033] The time series detection module of the fast behavior detection model performs time series feature detection on the interaction information and K consecutive frames to obtain time series features, including:

[0034] The time series detection module uses 3D-ResNet for feature extraction. The 3D-ResNet structure is based on the combination of ResNet and 3D convolutional network, and consists of multiple 3D convolutional layers and residual blocks;

[0035] Using the annotation information contained in the interaction information, K consecutive frames corresponding to the interactive objects are intercepted, and the interaction information and K consecutive frames are stacked to form K×W×H×3 three-dimensional information, which is input into 3D-ResNet for feature extraction to obtain the corresponding temporal feature map.

[0036] The improved DarkNet-19 feature extractor includes:

[0037] The DarkNet-19 feature extractor consists of an 11-layer structure, including 6 convolutional layers and 5 maximum pooling layers. Four residual modules are added to the DarkNet-19 feature extractor, which respectively add the features output by the second maximum pooling layer to the features output by the third convolutional layer; add the features output by the third maximum pooling layer to the features output by the fourth convolutional layer; add the features output by the fourth maximum pooling layer to the features output by the fifth convolutional layer; and add the features output by the fifth maximum pooling layer to the features output by the sixth convolutional layer; thereby obtaining an improved DarkNet-19 feature extractor that can extract more shallow fine-grained information and multi-scale target information.

[0038] The improved DarkNet-19 feature extractor is used to extract features from interaction information and key frames, and a feature map of multi-scale target information is obtained, including:

[0039] S41: Using the bounding box information included in the interaction information, extracting the interactive object image;

[0040] S42: The key frame image and the matching interactive item image are formed into a two-dimensional information of W×H×3, which is input into the improved DarkNet-19 feature extractor to output a multi-scale target information feature map of the key frame and interactive information.

[0041] The attention mechanism is implemented in the spatial domain based on the feature map of multi-scale target information, and the attention mechanism features of the spatial domain are obtained, including:

[0042] S51: Position encoding of the feature map of multi-scale target information using a sine function;

[0043] S52: The position code is input into the fully connected layer for convolution operation with a kernel size of 1×1, and the convolution result is output through the ReLU activation function. The output result is subjected to logistic regression using the softmax layer to obtain the spatial domain attention mechanism parameter θ;

[0044] S53: According to the spatial domain attention mechanism parameters θ and the original feature map before position encoding, bilinear sampling is performed to obtain the feature map V of the input spatial domain attention mechanism features.

[0045] The feature map of multi-scale target information is positionally encoded using a sine function, including:

[0046]

[0047] Where H and W represent the height and width of the feature respectively.

[0048] The temporal features and the attention mechanism features of the spatial domain are regarded as n non-overlapping feature images, which are spliced ​​and fused to obtain the fused features, including:

[0049] S61: Treat the temporal features and the spatial domain attention mechanism features as n non-overlapping images to generate a two-dimensional input table, set the input feature size to (H, W), and the image resolution to (l, l), so that all patches have the same projection matrix, and obtain the respective weights;

[0050] S62: First, operate on each column and share the weights of all columns; then operate on each row and share the weights of all rows, so as to further extract features and obtain multiple relatively independent feature images;

[0051] S63: Use the convolution layer to perform feature fusion on relatively independent feature images to obtain fused features.

[0052] The various weights obtained include:

[0053] W=W j z j , j = 1, 2, ..., n

[0054] Among them, Z j represents the jth image, W j represents the weight of each column or row of the jth image, and n represents the number of feature images.

[0055] Use the convolution layer to fuse relatively independent feature images to obtain fused features, including:

[0056]

[0057] Among them, PE represents the fused features, W k represents the weight of the kth feature image, Z k represents the kth and jth feature images, and n represents the number of feature images.

[0058] The interaction processing module of the fast behavior detection model processes the target behavior information and its interaction information to obtain the second-order relationship with the environment information, including:

[0059] S71: Encode the target information and interaction information, copy and connect the target feature information to all H×W spatial positions of the interaction feature information, and obtain the feature map of the behavior connection. Among them, x and y represent the position of target i in space;

[0060] S72: As the first-order relationship between target information and interaction information, all the first-order relationships between target information and interaction information are input into the interaction processing module to obtain the second-order relationship with the environmental information.

[0061] The first-order relationship between all target information and interaction information is input into the interaction processing module to obtain the second-order relationship with the environment information, including:

[0062]

[0063] in, Represents the second-order relationship between target information, interaction information and environmental information, Represents the characteristics of the target information, Represents the corresponding feature weight, x, y represent the position of target i in space.

[0064] The fused features and second-order relationships are tested to obtain the spatiotemporal behavior detection results of the target in the video, including:

[0065] The second-order relationship between the fused features, target information, interaction information and environmental information is predicted. The maximum number of targets in the segment is set to 32. The target is located through regression convolution to generate the link channel of the segment. The channel features are classified to obtain the behavior category.

[0066] The batch size of network training is set to 16, the total number of iterations is 100 epochs, and the learning rate is 10 -3 The weight decay factor is set to 0.0005 and the momentum factor is set to 0.9.

[0067] During training, the network automatically calculates the spatiotemporal IoU ratio (IoU) between the link channel and the ground truth. This is the IoU ratio between the region and start time where the behavior occurred and the ground truth. The matching principle for link channels is as follows: for each ground truth in a segment, the link channel with the largest spatiotemporal IoU ratio is found. This link channel is matched with the ground truth and is considered a positive sample. Conversely, if a link channel does not match any ground truth, it matches the background and is considered a negative sample. For the remaining unmatched link channels, if the spatiotemporal IoU ratio of a ground truth is greater than the threshold of 0.5, then this link channel is also matched with this ground truth.

[0068] The loss function of the network includes regression loss and classification loss:

[0069]

[0070] Among them, N is the number of positive samples of the link channel, c is the category confidence prediction value, l is the predicted value, and g is the location parameter of the ground truth. α is the weight coefficient, which is set to 1.

[0071] The test process automatically performs forward propagation based on the network parameters to predict the target's location, category, and duration. Since there may be multiple link channels predicted for the same target, the NMS algorithm is used to filter out duplicate link channels. The NMS threshold can be adjusted according to the actual scenario and is set to 0.35 for reference.

[0072] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A fast behavior detection method based on attention mechanism, characterized in that: include: S1: Obtain the RGB video to be detected containing interactive information, and divide the RGB video to be detected into segments of K consecutive frames; S2: Building a fast behavior detection model using the end-to-end spatiotemporal behavior detection network YOWO; The fast behavior detection model includes: a time sequence detection module, a space detection module and an interaction processing module; S3: The time series detection module of the fast behavior detection model performs time series feature detection on the interaction information and K consecutive frames to obtain time series features; S4: adding a residual module to the DarkNet-19 feature extractor in the spatial detection module of the fast behavior detection model to obtain an improved DarkNet-19 feature extractor, performing feature extraction on the interaction information and key frames using the improved DarkNet-19 feature extractor to obtain a feature map of multi-scale target information; the key frame is the first frame of the continuous video frames; The improved DarkNet-19 feature extractor includes: The DarkNet-19 feature extractor consists of an 11-layer structure, including 6 convolutional layers and 5 maximum pooling layers. Four residual modules are added to the DarkNet-19 feature extractor, which respectively add the features output by the second maximum pooling layer to the features output by the third convolutional layer; add the features output by the third maximum pooling layer to the features output by the fourth convolutional layer; add the features output by the fourth maximum pooling layer to the features output by the fifth convolutional layer; and add the features output by the fifth maximum pooling layer to the features output by the sixth convolutional layer. This results in an improved DarkNet-19 feature extractor that can extract more shallow fine-grained information and multi-scale target information. S5: Implement the attention mechanism in the spatial domain based on the feature map of multi-scale target information to obtain the attention mechanism features in the spatial domain; The attention mechanism is implemented in the spatial domain based on the feature map of multi-scale target information, and the attention mechanism features of the spatial domain are obtained, including: S51: Position encoding of the feature map of multi-scale target information using a sine function; S52: The position code is input into the fully connected layer for convolution operation with a kernel size of 1×1, and the convolution result is output through the ReLU activation function. The output result is subjected to logistic regression using the softmax layer to obtain the spatial domain attention mechanism parameter θ; S53: Obtain a feature map V of the input spatial domain attention mechanism feature through bilinear sampling based on the spatial domain attention mechanism parameter θ and the original feature map before position encoding; S6: Treat the temporal features and the attention mechanism features of the spatial domain as n non-overlapping feature images, and perform splicing and fusion to obtain the fused features; S7: The interaction processing module of the fast behavior detection model processes the target behavior information and its interaction information to obtain the second-order relationship between the target information, the interaction information and the environment information; S8: Detect the second-order relationship between the fused features, target information, interaction information and environmental information to obtain the spatiotemporal behavior detection results of the target in the video.

2. A fast behavior detection method based on attention mechanism according to claim 1, characterized in that: The time series detection module of the fast behavior detection model performs time series feature detection on the interaction information and K consecutive frames to obtain time series features, including: The time series detection module uses 3D-ResNet for feature extraction. The 3D-ResNet structure is based on the combination of ResNet and 3D convolutional network, and consists of multiple 3D convolutional layers and residual blocks; Using the annotation information contained in the interaction information, K consecutive frames corresponding to the interactive objects are intercepted, and the interaction information and K consecutive frames are stacked to form K×W×H×3 three-dimensional information, which is input into 3D-ResNet for feature extraction to obtain the corresponding temporal feature map.

3. The fast behavior detection method based on the attention mechanism according to claim 1 is characterized in that: The improved DarkNet-19 feature extractor is used to extract features from interaction information and key frames, and a feature map of multi-scale target information is obtained, including: S41: Using the bounding box information included in the interaction information, extracting the interactive object image; S42: The key frame image and the matching interactive item image are formed into a two-dimensional information of W×H×3, which is input into the improved DarkNet-19 feature extractor to output a multi-scale target information feature map of the key frame and interactive information.

4. The method for rapid behavior detection based on attention mechanism according to claim 1, characterized in that: The feature map of multi-scale target information is positionally encoded using a sine function, including: Where H and W represent the height and width of the feature respectively.

5. The method for rapid behavior detection based on attention mechanism according to claim 1, characterized in that: The temporal features and the attention mechanism features of the spatial domain are regarded as n non-overlapping feature images, which are spliced ​​and fused to obtain the fused features, including: Among them, PE represents the fused features, W k represents the weight of the kth feature image, Z k represents the kth and jth feature images, and n represents the number of feature images.

6. The method for rapid behavior detection based on attention mechanism according to claim 1, characterized in that: The interaction processing module of the fast behavior detection model processes the target behavior information and its interaction information to obtain the second-order relationship with the environment information, including: S71: Encode the target information and interaction information, copy and connect the target feature information to all H×W spatial positions of the interaction feature information, and obtain the feature map of the behavior connection. Among them, x and y represent the position of target i in space; S72: As the first-order relationship between target information and interaction information, all the first-order relationships between target information and interaction information are input into the interaction processing module to obtain the second-order relationship with the environmental information.

7. The method for rapid behavior detection based on attention mechanism according to claim 6, characterized in that: The first-order relationship between all target information and interaction information is input into the interaction processing module to obtain the second-order relationship with the environment information, including: in, Represents the second-order relationship between target information, interaction information and environmental information, Represents the characteristics of the target information, Represents the corresponding feature weight, x, y represent the position of target i in space.

8. The method for rapid behavior detection based on attention mechanism according to claim 1, characterized in that: The fused features and second-order relationships are tested to obtain the spatiotemporal behavior detection results of the target in the video, including: The second-order relationship between the fused features, target information, interaction information and environmental information is predicted. The maximum number of targets in a segment is set to 32. The targets are located through regression convolution to generate the link channel of the segment. The channel features are classified to obtain the behavior category.

Images