Action recognition methods, systems, devices, and media based on spatiotemporal scale transformation
By employing target detection and spatiotemporal scale transformation methods, the problem of inconsistency between target spatial and temporal scales in behavior recognition is solved, enabling accurate identification of behavior regions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2021-12-31
- Publication Date
- 2026-05-26
Smart Images
Figure CN116434318B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of behavior recognition technology, specifically a behavior recognition method that combines target detection with multi-scale feature fusion. Background Technology
[0002] Action recognition research in video applications has broad practical applications such as intelligent video surveillance and is a popular research direction in computer vision. However, existing action recognition algorithms still face the following difficulties and challenges, resulting in areas where detection results need improvement:
[0003] Detection of behavioral regions. In video images, the region where the actual behavior is performed is usually a local area centered on the human body and combined with a certain scene. Mainstream behavior recognition methods typically use the entire image for behavior recognition or directly crop a fixed central region, ignoring the semantics of the subject of the behavior, i.e., the human body, thus resulting in certain redundant regions.
[0004] The spatiotemporal scale of the actors' targets is inconsistent. For the same type of behavior, the distance between the actors and the camera varies across different samples, leading to inconsistencies in spatial scale. Furthermore, the speed at which different actors perform their actions also differs, resulting in inconsistencies in temporal scale.
[0005] Currently, mainstream methods for behavior recognition can be mainly divided into two categories: representation-based methods and deep network-based methods.
[0006] In representation-based methods, behavioral representations are divided into two categories: global representation, which is based on the extraction of the overall representation of human body structure, shape and movement, and local representation, which mines local information in videos.
[0007] Global representation utilizes the video's global motion energy image (MEI) and motion history image (MHI). The MEI uses a binary image to represent the location of motion in the video, while the MHI uses a grayscale image to represent changes in the motion image; the grayscale value of each pixel in the MHI is a function of the motion information at that point. Both MEI and MHI contain important semantic features of the video; for example, the gradient information in the MHI can be used to filter out complex backgrounds.
[0008] Local representation can compensate for the lack of local detail mining in global features. Local representation methods typically follow this framework: detection of Space-Time Interest Points (STIPs); extraction of local descriptors; and fusion of local descriptors. Key methods include extending 2D Harris detection to 3D; and Gabor filtering in both time and space dimensions to address the problem of insufficient STIPs. Furthermore, feature extraction of motion trajectories to capture local motion information has attracted widespread attention. Dense Trajectory (DT) performs dense sampling frame-by-frame on the video and tracks the trajectories of the sampled points, extracting trajectory feature descriptors. It also extracts HOG, HOF, and MBH features from the regions near the trajectory and finally encodes these features for behavior classification. Improved Dense Trajectory (iDT) improves upon DT by using projection transformations between consecutive frames to estimate camera motion, removing its influence on trajectory estimation, and optimizing feature normalization and encoding methods, achieving further performance improvements compared to DT.
[0009] Deep network-based methods can be mainly divided into three categories: multi-stream network models, spatiotemporal network models, and human information-based network models. Multi-stream network models utilize video to extract motion information such as optical flow, specifically extracting motion features from moving regions within the video. Spatiotemporal network models use 3D convolutional networks to perform 3D convolution on video image sequences to extract spatiotemporal features, or perform 2D convolution on each frame and then fuse the features from each frame. Human information-based network models extract the skeleton joint positions of pedestrians in videos and use the skeleton information as features to assist in behavior recognition and classification.
[0010] Multi-stream network models and spatiotemporal network models both use the entire image as input for behavior recognition, failing to utilize methods such as object detection to focus on the region where the behavior occurs. Furthermore, network models based on human information only retain skeletal features, lacking scene mining of the behavior's location. Utilizing object detection to locate pedestrians and cropping their areas could help the model more effectively focus on the behavior's location.
[0011] Existing behavior recognition methods can effectively identify behaviors performed by some large targets, but in practical applications, targets are often far from the camera and occupy a small portion of the frame, making it difficult for behavior recognition models to focus on distant targets and affecting the accuracy of behavior recognition.
[0012] Feature 1: Inconsistent target spatial scale. Due to the varying distances of targets from the camera, the target features extracted by the behavior recognition model also exhibit inconsistent scales. Targets farther from the camera often only have a portion of their features extracted, making behavior recognition difficult.
[0013] Feature Two: Inconsistent Time Scales of Behaviors. The same behavior can be performed at different times and speeds due to different people's habits. Simply using a fixed frame rate to extract features and identify behaviors will be insufficient for recognizing behaviors with significant temporal variations. Summary of the Invention
[0014] In view of the above-mentioned shortcomings of existing behavior recognition methods, the purpose of this invention is to propose a behavior recognition method based on spatiotemporal scale transformation.
[0015] This invention is achieved through the following technical solution: A behavior recognition method based on spatiotemporal scale transformation, comprising:
[0016] The image sequence is subjected to target detection based on the target detector to obtain the corresponding target bounding box region;
[0017] Perform spatial multi-scale transformation on each target region to obtain multi-scale target regions;
[0018] The multi-scale target region is fused to obtain fused features;
[0019] Based on the fused features, behavior classification is performed to identify the behavior.
[0020] Preferably, the step of performing target detection based on the target detector to obtain the corresponding target bounding box region includes:
[0021] Different targets yield corresponding bounding box positions;
[0022] Expand the scene horizontally and vertically at the location of the target box;
[0023] After expansion, cropping is performed to obtain the target bounding box area.
[0024] Preferably, the step of performing spatial multi-scale transformation on each of the target regions to obtain multi-scale target regions includes:
[0025] The target bounding box region is scaled to (H, W) using bilinear interpolation.
[0026] By scaling by 1x, 2x, and 4x, we obtain multi-scale regions of (H,W), (2H,2W), and (4H,4W).
[0027] Preferably, the step of fusing features in the multi-scale target region to obtain fused features includes:
[0028] Spatial consistency feature extraction is performed on multi-scale target regions to obtain feature maps with consistent spatial scales.
[0029] Perform temporal multi-scale transformation on the multiple feature maps with consistent spatial scales to obtain features at multiple scales.
[0030] The multiple scale features are fused together.
[0031] Preferably, the multi-scale target region undergoes spatial consistency feature extraction to obtain multiple feature maps with consistent spatial scales, including:
[0032] The multi-scale target regions obtained by multi-scale transformation are fed into a two-dimensional convolutional network to extract features, resulting in feature maps (H1,W1), (H2,W2) and (H3,W3) at different spatial scales.
[0033] Construct a feature pyramid, where (H1,W1), (H2,W2), and (H3,W3) correspond to the high-level, middle-level, and low-level features, respectively.
[0034] Where (H1,W1) is the feature map corresponding to (H,W), which is the feature map with the smallest spatial size;
[0035] By performing a convolution operation on (H2,W2) and (H3,W3), a feature map of size (H1,W1) is obtained, resulting in features with consistent spatial scale.
[0036] Then, the corresponding feature maps of each layer of each frame are average pooled and concatenated to form the spatial consistency features of each frame.
[0037] Preferably, the step of performing a temporal multi-scale transformation on multiple feature maps with consistent spatial scales to obtain multiple scale features includes:
[0038] Multiple sampling frame rates can be set, including 1 / 2 sampling and 1 / 4 sampling;
[0039] The spatial-temporal consistency features are temporally sampled based on the aforementioned sampling frame rate.
[0040] Preferably, the fusion of the multiple time-scale features includes:
[0041] By performing behavior recognition on features at the same time scale, their respective behavior classification probability values are obtained;
[0042] The behavior recognition probability values at each time scale are averaged to obtain the final behavior recognition result.
[0043] According to a second aspect of the present invention, a behavior recognition system based on spatiotemporal scale transformation is provided, comprising:
[0044] The target detection module performs target detection on the video, obtains the corresponding target box position for different targets, expands the target box area in the horizontal and vertical directions, and then crops out the respective target areas.
[0045] A spatial multi-scale transformation module performs scale normalization processing on the target region and performs spatial multi-scale transformation to form target regions of multiple scales.
[0046] The spatiotemporal multi-scale transformation module feeds the target regions at various scales into a two-dimensional convolutional network to extract features, thereby obtaining multi-scale feature maps; it then performs a spatial consistency transformation on the multi-scale feature maps to obtain spatially scale-consistent feature maps, and finally performs a temporal multi-scale transformation to obtain spatially consistent features.
[0047] The feature fusion module fuses the spatial consistency features to perform behavior classification.
[0048] According to a third aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can be used to perform any of the methods described above, or to run the system described above.
[0049] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the above-described method or to run the above-described system.
[0050] Compared with the prior art, the present invention has the following beneficial effects:
[0051] This invention combines target detection with behavior recognition, enabling the behavior recognition model to effectively focus on the area where the behavior actually occurs.
[0052] Furthermore, to address the issue of inconsistent target spatial scales, this invention utilizes upsampling and downsampling to achieve scale uniformity, thereby better ensuring the scale consistency of the target.
[0053] Meanwhile, to address the temporal differences among different samples, this method proposes a multi-frame-rate sampling operation to fuse features from multiple time scales, enabling effective feature extraction for behaviors of different durations. Attached Figure Description
[0054] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0055] Figure 1 This is a flowchart of the target area cropping process in one embodiment of the present invention;
[0056] Figure 2 This is a flowchart of multi-layer feature fusion in one embodiment of the present invention. Detailed Implementation
[0057] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.
[0058] This invention provides an embodiment of a behavior recognition method based on spatiotemporal scale transformation, comprising:
[0059] S1, Target detection is performed based on the target detector to obtain the corresponding target bounding box region;
[0060] S2, perform spatial multi-scale transformation on each target region to obtain multi-scale target regions;
[0061] S3, which performs feature fusion on multi-scale target regions to obtain fused features;
[0062] S4, based on fused features, performs behavior classification and identifies behaviors.
[0063] This embodiment combines the target detection module to focus on the target's motion region and fuses multi-scale features through multi-scale spatial and temporal transformations, thereby alleviating the behavior recognition problem caused by the inconsistency of spatiotemporal scales and improving the model's robustness to target scale and behavior speed.
[0064] like Figure 1 As shown, the present invention provides a preferred embodiment in which S1 is executed, and the target detector is used to detect the region where the behavior occurs. Figure 1 As shown, it includes:
[0065] S101, using a target detector, detects pedestrian targets in the video frame by frame;
[0066] S102, after taking the union of the regions of each frame, expand the region by a certain number of pixels in the horizontal and vertical directions and then crop out the region as the region where the behavior occurs.
[0067] The expanded pixel range can be adjusted according to the original size of the input image, focusing on covering the pedestrian's movement area. Specifically, for an input image with a width of W and a height of H, the horizontal expansion pixel range is located in the range [W / 3, 2W / 3], and the vertical expansion pixel range is located in the range [H / 3, 2H / 3]. For example, for an input image with a size of 1920*1080, the horizontal expansion pixel range can be set to [640, 1280] and the vertical expansion pixel range to [360, 720].
[0068] like Figure 2 The diagram shown is a flowchart of a preferred embodiment of the multi-layer feature fusion method provided by the present invention. As can be seen from the diagram, it includes:
[0069] S2 utilizes a scale normalization module to perform spatiotemporal scale normalization on the target, specifically including:
[0070] S201, given dimensions (H, W);
[0071] S202, each input target box region (behavior occurrence region) is scaled to (H,W), (2H,2W), (4H,4W) using bilinear interpolation, and then sent to the feature fusion module for feature fusion;
[0072] S3 utilizes a spatiotemporal multi-scale module to extract spatiotemporal multi-scale features, specifically including:
[0073] S301, Images at various scales are fed into a two-dimensional convolutional network ResNet50 to extract features, and feature maps at scales such as (H1,W1), (H2,W2), and (H3,W3) of the output features of the res5 layer are obtained respectively.
[0074] S302, then the residual network convolution operation in ResNet is used to align the spatial scale of the three-layer features, so that the low-level and middle-level features are converted to be consistent with the scale of the high-level features, and a feature map with consistent spatial scale is obtained.
[0075] S303, then average pooling is performed on the features of each layer to obtain features with a scale of D1;
[0076] S304 concatenates the features of each layer to form a spatially consistent feature with a scale of D2;
[0077] S305. In order to ensure the multi-scale nature of the temporal features, temporal sampling operations are performed on the features of each layer. Each layer of features is sampled frame by frame, 1 / 2 sampled, and 1 / 4 sampled to obtain 8 frames of spatiotemporal features, 4 frames of spatiotemporal features, and 2 frames of spatiotemporal features.
[0078] Average pooling represents the entire feature map by its mean, reducing feature dimensionality, computational complexity, and the risk of overfitting. Feature concatenation fuses features from different spatial scales, improving the model's ability to represent multi-scale features.
[0079] S4 utilizes multi-scale features for behavior classification, specifically including:
[0080] S401, the obtained multi-scale features are fed into the fully connected layer to obtain the behavior classification results;
[0081] S402, compare with the actual behavior results, calculate the loss value, backpropagate, and train the network.
[0082] This invention provides an application example for behavior recognition based on spatiotemporal scale transformation.
[0083] First, the video with a size of 1280*720 is uniformly sampled to obtain an 8-frame image sequence {I1,I2,…}. After object detection, the target bounding box positions {B1,B2,…} are obtained, and the target region {I1',I2',…} is cropped.
[0084] Given a size of (224, 224), the target region is scaled using bilinear interpolation to form the sequence {C1, C2, ...} at (224, 224), the sequence {D1, D2, ...} at (448, 448), and the sequence {E1, E2, ...} at (996, 996).
[0085] Each frame of {C1,C2,…}, {D1,D2,…}, and {E1,E2,…} is fed into a two-dimensional convolutional network to extract features, forming a feature pyramid. The features formed by {E1,E2,…} constitute the bottom layer features, the features formed by {D1,D2,…} constitute the middle layer features, and the features formed by {C1,C2,…} constitute the high layer features.
[0086] The three layers of features have different spatial scales. Therefore, convolution is used to align the spatial scales of the three layers of features, so that the bottom and middle layer features are converted to be consistent with the scale of the high layer features. Then, the corresponding feature maps of the three layers of features in each frame are averaged and concatenated to form the spatial consistency features {S1,S2,…} of each frame.
[0087] Temporal sampling is performed on the spatial consistency features {S1,S2,…} of each frame image. Frame-by-frame sampling, half-sample, and quarter-sample are performed respectively to obtain 8-frame, 4-frame, and 2-frame spatiotemporal features. Finally, the 8-frame, 4-frame, and 2-frame features are fed into the fully connected layer of the neural network for behavior recognition. The behavior probability results obtained at different time scales are averaged to obtain the final classification probability, and the category with the highest probability value is taken as the classification result.
[0088] In some embodiments of the present invention, a multi-task learning mechanism can be adopted, in which the parameters of each layer of the two-dimensional convolutional network are not shared, and each layer of features is used to perform behavior recognition, and each layer calculates its own loss value to train the parameters in the two-dimensional convolutional network. In this way, the features of multiple layers can be extracted and multi-task end-to-end training can be performed to obtain an end-to-end behavior recognition model.
[0089] Addressing the challenges of target detection in reality, the behavior recognition method based on spatiotemporal scale transformation in videos, as described in the above embodiments of this invention, proposes first detecting the target region using target detection, then expanding the behavior region to represent the behavior occurrence region. The behavior occurrence region is then cropped, and a spatial feature pyramid is constructed using multi-scale scaling features. Finally, multi-scale temporal sampling is employed to fuse features from different time scales, resulting in behavior classification. This behavior recognition method based on spatiotemporal scale transformation, addressing the aforementioned problems, effectively solves the identification of distant target behaviors and behaviors with significant temporal differences through designs such as spatial scale consistency transformation and multi-scale temporal feature fusion.
[0090] The method proposed in the above embodiments of the present invention includes target behavior region cropping, behavior region spatial scale consistency transformation, temporal multi-scale sampling, and spatiotemporal feature fusion; the entire network is weighted and summed using the loss functions of each layer's feature behavior recognition as the loss function of the entire network, achieving end-to-end network training; or a multi-task learning mechanism is used to recognize behaviors and calculate losses using features from each layer, training each layer of the network separately.
[0091] During model training, there are two training modes. The first is a multi-layer co-training mode, where the 2D convolutional networks extracting features from each layer share parameters. After fusing the features from each layer, subsequent behavior recognition is performed to obtain the final behavior classification result. This result is then compared with the true label, the loss value is calculated, and the result is backpropagated to train the convolutional network parameters. The second is a multi-task training mode, where each 2D convolutional network extracting features operates independently. The recognition result of each layer is compared with the true label, the loss value is calculated, and the result is backpropagated to train the parameters of its respective convolutional network. During testing, the recognition results of all layers are averaged to obtain the final behavior classification result.
[0092] The loss value is calculated as shown in the formula. It is a loss function based on cross-entropy and is used to determine the difference between the recognition result and the true result.
[0093]
[0094] Where M is the number of categories, y c p is a sign function that takes the value 1 when the true class of the sample equals c, and 0 otherwise. c This represents the probability that the model classifies the sample as class c.
[0095] This invention constructs a spatiotemporal multi-scale transformation model, first accurately detects the target region as the behavior region, and then performs a spatial consistency transformation on the target region, thereby effectively addressing the problem of inconsistent scale of behavior targets in videos. At the same time, by utilizing temporal multi-scale sampling, features can be effectively extracted from behaviors across different time spans, thus enabling more accurate behavior identification.
[0096] Based on the same concept as the above embodiments, another embodiment provides a behavior recognition system based on spatiotemporal scale transformation, including:
[0097] The target detection module performs target detection on the video, obtains the corresponding target bounding box position for different targets, expands the target bounding box area in the horizontal and vertical directions of the scene, and then crops out the respective target areas.
[0098] The spatial multi-scale transformation module performs scale normalization on the target region and performs spatial multi-scale transformation to form target regions of multiple scales.
[0099] The spatiotemporal multi-scale transformation module feeds target regions of various scales into a two-dimensional convolutional network to extract features, resulting in multi-scale feature maps. The multi-scale feature maps are then subjected to spatial consistency transformation to obtain spatially scale-consistent feature maps, and finally subjected to temporal multi-scale transformation to obtain temporal multi-scale features.
[0100] The feature fusion module performs behavior recognition based on multi-scale features in the time domain, obtains the respective behavior classification probability values, and averages the behavior recognition probability values at each time scale to obtain the final behavior recognition result.
[0101] Based on the same concept as the above embodiments, other embodiments provide a terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to perform the above-described method or run the above-described system.
[0102] Based on the same concept as the above embodiments, other embodiments provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform any of the above methods, or to run the above system.
[0103] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.
Claims
1. A behavior recognition method based on spatiotemporal scale transformation, characterized in that, include: The image sequence is subjected to target detection based on the target detector to obtain the corresponding target bounding box region; Perform spatial multi-scale transformation on each of the target bounding boxes to obtain multi-scale target regions; The multi-scale target region is fused to obtain fused features; Based on the fused features, behavior classification is performed to identify the behavior; The step of fusing features from the multi-scale target regions to obtain fused features includes: The multi-scale target region is subjected to spatial consistency feature extraction to obtain multiple feature maps with consistent spatial scales. Perform temporal multi-scale transformation on the multiple feature maps with consistent spatial scales to obtain multiple temporal scale features; Integrate the features of the multiple time-domain scales; The fusion of features across multiple time scales includes: By performing behavior recognition on features at the same time scale, their respective behavior classification probability values are obtained; The behavior classification probability values at each time scale are averaged to obtain the final behavior recognition result.
2. The behavior recognition method based on spatiotemporal scale transformation according to claim 1, characterized in that, The step of performing target detection based on the target detector to obtain the corresponding target bounding box region includes: Target detection is performed on the image sequence, and the corresponding bounding box positions are obtained for different targets; Expand the scene horizontally and vertically at the location of the target box; After expansion, cropping is performed to obtain the target bounding box area.
3. The behavior recognition method based on spatiotemporal scale transformation according to claim 1, characterized in that, The step of performing spatial multi-scale transformation on each of the target regions to obtain multi-scale target regions includes: The target bounding box region is scaled to (H, W) using bilinear interpolation. By scaling by 1x, 2x, and 4x, we obtain multi-scale target regions of (H,W), (2H,2W), and (4H,4W).
4. The behavior recognition method based on spatiotemporal scale transformation according to claim 1, characterized in that, The multi-scale target region undergoes spatial consistency feature extraction to obtain multiple feature maps with consistent spatial scales, including: The multi-scale target regions obtained by multi-scale transformation are fed into a two-dimensional convolutional network to extract features, resulting in feature maps (H1,W1), (H2,W2) and (H3,W3) at different spatial scales. Construct a feature pyramid, where (H1,W1), (H2,W2), and (H3,W3) correspond to the high-level, middle-level, and low-level features, respectively. Where (H1,W1) is the feature map corresponding to (H,W), which is the feature map with the smallest spatial size; By performing convolution operations on (H2,W2) and (H3,W3) again, a feature map of size (H1,W1) is obtained, resulting in a feature map with consistent spatial scale. The feature maps corresponding to each layer of features in each frame are average pooled and concatenated to form spatial consistency features for each frame.
5. The behavior recognition method based on spatiotemporal scale transformation according to claim 1, characterized in that, The step of performing a multi-scale temporal transformation on the multiple feature maps with consistent spatial scales to obtain multiple temporal scale features includes: Multiple sampling frame rates can be set, including 1 / 2 sampling and 1 / 4 sampling; The spatial-temporal consistency features are temporally sampled based on the sampling frame rate.
6. A behavior recognition system based on spatiotemporal scale transformation, used to implement the method according to any one of claims 1-5, characterized in that, include: The target detection module performs target detection on the video, obtains corresponding target bounding boxes for different targets, expands the target bounding boxes in the horizontal and vertical directions, and then crops out their respective target regions. A spatial multi-scale transformation module performs scale normalization processing on the target region and performs spatial multi-scale transformation to form target regions of multiple scales. The spatiotemporal multi-scale transformation module feeds the target regions at various scales into a two-dimensional convolutional network to extract features, thereby obtaining multi-scale feature maps; it then performs a spatial consistency transformation on the multi-scale feature maps to obtain spatially scale-consistent feature maps, and finally performs a temporal multi-scale transformation to obtain temporal multi-scale features. The feature fusion module performs behavior recognition based on the multi-scale features in the time domain, obtains the respective behavior classification probability values, and averages the behavior recognition probability values at each time scale to obtain the final behavior recognition result.
7. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-5.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, this program can be used to perform the method of any one of claims 1-5.