A method and system for extracting video frames for bone behavior recognition in multiple time scales
By employing a method of uniform sampling of video frames and multi-scale feature fusion, the problem of poor performance of skeletal behavior recognition models on long videos was solved, achieving higher classification accuracy and understanding capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUBEI UNIV OF TECH
- Filing Date
- 2023-04-27
- Publication Date
- 2026-05-29
AI Technical Summary
Existing skeletal behavior recognition models perform poorly on long videos, especially 3D-CNN, which lacks classification accuracy on long videos.
By uniformly sampling video frames and dividing them into low-frequency, mid-frequency, and high-frequency channels, parallel convolution processing is performed, combined with multi-scale feature fusion, and feature understanding is gradually deepened, learning video features from coarse-grained to fine-grained.
It improved the model's performance on long videos, enhanced its ability to understand long videos, and ensured classification accuracy.
Smart Images
Figure CN116469172B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to multiple technical fields, including deep learning, behavior detection, and skeletal behavior recognition. Specifically, it relates to a method and system for extracting video frames for skeletal behavior recognition across multiple time scales. Background Technology
[0002] Action recognition has become a crucial task in the field of computer vision over time. For action detection, various detection methods have been developed, including using optical flow features, RGB representation, and skeletal features as multimodal methods. Among these, skeletal action detection has received increasing attention in recent years due to its better robustness against interference from other information in the environment. Skeletal action recognition first extracts key human body points through a network, then uses the detected skeletal positions of these key points as a sequence as network input. This allows the recognition process to focus more on the positional information of the key points.
[0003] Action recognition differs from traditional computer vision tasks in that it requires extracting not only spatial features from images but also temporal information. In other words, action recognition relies not only on the image itself within a single frame but also on changes in keypoint information over time. Skeletal action recognition can be broadly categorized into two types. One type treats the coordinates of skeletal keypoints as a graph structure and extracts keypoint information using graph convolutional networks (GCNs). GCNs compare the differences in keypoints between pairs of frames to obtain information about the temporal dimension, while simultaneously convolving the temporal and spatial dimensions.
[0004] Another type of skeletal behavior recognition method uses 3D-CNN for feature extraction. 3D-CNN adds temporal dimension information to the convolutional kernel of CNN to form a new convolutional kernel. That is, by reading multiple images frames at once, temporal information is introduced into feature extraction to enable the network to understand time and space.
[0005] Both GCN and 3D-CNN require processing of the temporal dimension. However, due to the limitation of the convolution kernel size, GCN and 3D-CNN often perform worse on long videos than on short videos.
[0006] Therefore, how to improve the model's performance on long videos while ensuring classification accuracy has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0007] The purpose of this invention is to provide a method and system for extracting video frames for skeletal behavior recognition at multiple time scales, which can improve the model's performance on long videos while ensuring classification accuracy.
[0008] According to a first aspect of the present invention, a method for extracting video frames for skeletal behavior recognition at multiple time scales is provided, comprising the following steps:
[0009] Step 1: Input the original image, select the video characters in the original image, extract the 2D human pose key points in the video, and convert the obtained key points into heat maps or coordinate information and save them.
[0010] Step 2: If the data is saved as coordinate information, it is converted into a heatmap using a mapping function. The heatmaps are then stacked to form the corresponding human body key point heatmap.
[0011] Step 3: Perform uniform sampling on the stacked human body key point heatmap to obtain the uniformly sampled frame as the raw input for the next stage of sampling;
[0012] Step 4: Process the uniformly sampled frames and fuse the low-frequency and mid-frequency sampling channels. After fusion, the number of frames in the low-frequency and mid-frequency channels is the same as that in the high-frequency channel.
[0013] Step 5: Convolve the channels in parallel for frames of different frequencies at multiple scales to obtain the corresponding feature information;
[0014] Step 6: The feature information is judged and normalized. Based on the probability results after recognition, the final output is the behavior recognition category.
[0015] Based on the above technical solution, the present invention can also be improved as follows.
[0016] Optionally, the step of selecting video characters in the original image includes:
[0017] The object detection algorithm is used to select people in the video, and the output is the processed original image containing the candidate boxes.
[0018] Optionally, the extraction of 2D human pose key points from the video includes:
[0019] A top-down human pose estimation model is used to estimate the pose of the selected human body. Subnets are gradually added from high resolution to low resolution to form more stages, and the multi-resolution subnets are connected in parallel. Throughout the process, information is repeatedly exchanged on the parallel multi-resolution subnets to perform multi-scale repeated fusion. The final output of the network serves as the key point for the overall estimation.
[0020] Optionally, the process of uniformly sampling the stacked human body key point heatmap includes: dividing the video into segments of equal length and then uniformly sampling from each segment.
[0021] Optionally, the processing of the uniformly sampled frame includes: processing the sampled frame using a low-frequency sampling channel, a mid-frequency sampling channel, and a high-frame-rate sampling channel; wherein, the channel division under multiple scales includes, but is not limited to, the above three sampling channels, including multiple low-frequency channels, mid-frequency channels, and high-frequency channels.
[0022] Optionally, the process of fusing the low-frequency frame acquisition channel and the intermediate-frequency frame acquisition channel is substituted into the following formula:
[0023]
[0024] In the formula, T l T represents the low frame rate sampling channel. m T represents the medium frame sampling frequency channel. h This indicates the high frame rate sampling channels, and num represents the total number, meaning that the number of low frame rate sampling channels and medium frame rate sampling channels, after being concatenated by features, is consistent with the number of high frame rate sampling channels.
[0025] Optionally, the step of using parallel convolution channels for different frequency frames at multiple scales to obtain corresponding feature information includes: learning the low-frequency and mid-frequency frame acquisition channels, and increasing the number of input frames on this basis, continuously deepening from coarse-grained to fine-grained, and fusing the learned multi-segment features.
[0026] Optionally, the learned multi-segment features are fused and substituted into the following formula.
[0027] F(n)=g φ (f l +f m )+f h
[0028] In the formula, F(n) represents the final fused feature, and n is the number of input layers; f l f represents the characteristics of the low frame rate sampling channel. m Indicates the channel characteristics of medium frame sampling frequency, f h denoted by , gφ is an aggregation function that fuses the low-frame-rate and medium-frame-rate channels.
[0029] Optionally, the final output of the behavior recognition category based on the probability results after recognition includes:
[0030] The behavior detection results are analyzed and evaluated. Based on the characteristics of the dataset and actual needs, appropriate performance indicators are selected for evaluation. Based on the experimental results, the algorithm is optimized and adjusted. The appropriate performance indicators include, but are not limited to, accuracy, recall, and low frame rate frame channel feature values.
[0031] According to a second aspect of the present invention, a system for extracting video frames for skeletal behavior recognition at multiple scales is provided, the system comprising:
[0032] The video information acquisition module is used to input the original image, select the video characters in the original image, extract the 2D human pose key points in the video, and convert the acquired key points into heat maps or coordinate information for storage; if saved as coordinate information, it is converted into heat maps through a mapping function, and the heat maps are stacked to form the corresponding human key point heat map.
[0033] The video information input module is used to uniformly sample the stacked human key point heatmap and obtain the uniformly sampled frame as the raw input for the next stage of sampling.
[0034] The video information processing module is used to process uniformly sampled frames, fuse low-frequency and mid-frequency sampling channels, and after fusion, the number of frames in the low-frequency and mid-frequency channels is the same as that in the high-frequency channel; and convolve the channels in parallel for different frequency frames at multiple scales to obtain the corresponding feature information.
[0035] The video information output module is used to process the feature information after judgment and normalization, and finally output the behavior recognition category based on the probability results after recognition.
[0036] The technical effects and advantages of this invention are as follows:
[0037] This invention provides a method and system for extracting video frames for skeletal behavior recognition across multiple time scales. It improves the model's performance on long videos while maintaining classification accuracy. By further sampling the uniformly sampled frames at low, medium, and high frequencies, the model can learn features at different video scales hierarchically, from coarse-grained to fine-grained, thereby enhancing the 3D-CNN's ability to understand long videos. Therefore, this invention solves the problem of traditional 3D-CNN's poor performance on long videos.
[0038] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description
[0039] Figure 1A flowchart illustrating the steps of a method for extracting video frames for skeletal behavior recognition at multiple time scales, provided in an embodiment of the present invention;
[0040] Figure 2 This is a flowchart of a multi-timescale downsampling process based on an improved 3D-CNN algorithm, provided for an embodiment of the present invention. Detailed Implementation
[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0042] Understandably, given the deficiencies in the background technology, this invention proposes a method for extracting video frames for skeletal behavior recognition across multiple time scales, specifically as follows: Figure 1 As shown, it includes the following steps:
[0043] Step 1: Input the original image, select the video characters in the original image, extract the 2D human pose key points in the video, and convert the obtained key points into heat maps or coordinate information and save them.
[0044] Specifically, the original image includes a video file.
[0045] The step of selecting video characters in the original image includes: performing target detection algorithm recognition on the video in the original image to obtain the position information of the characters in the video.
[0046] The object detection algorithm uses the YOLO algorithm to select bounding boxes for human figures, outputting a processed image containing candidate boxes. The YOLO algorithm employs a single CNN model for end-to-end object detection. Its core idea is to use the entire image as network input and directly regress the location and category of the bounding box at the output layer. The goal is to select people within the video.
[0047] The process of extracting 2D human pose key points from the video and converting the acquired key points into heatmaps or coordinate information for storage includes:
[0048] The 2D human pose key points in the video are extracted using the HRNet (Deep High-Resolution Representation Learning for Human Pose Estimation) 2D human pose estimation model, and then saved as a heatmap or key point coordinates.
[0049] Specifically, extracting 2D human pose keypoints from videos using the HRNet model involves: estimating the pose of selected human figures using a top-down Human Keypoint Detection (HRNet) model; gradually adding high-to-low resolution subnetworks to create more stages; and connecting these multi-resolution subnetworks in parallel. Throughout this process, information is repeatedly exchanged between the parallel multi-resolution subnetworks for multi-scale fusion, and the final output of the network serves as the overall estimated keypoints.
[0050] The HRnet model is used to obtain key points of the human skeleton from the selected image.
[0051] Step 2: If saved as coordinate information, convert it into a heatmap using a mapping function, and stack the key point heatmaps to form the corresponding human body key point heatmap;
[0052] Furthermore, stacking keypoint heatmaps to form corresponding human keypoint heatmaps includes: stacking T K*W*H heatmaps to form a T*K*W*H channel heatmap, where T heatmaps represent T frames, K represents K human keypoints, and W and H are the width and height of the heatmap. Stacking heatmaps allows the model to learn temporal dimension features.
[0053] Step 3: Perform uniform sampling on the stacked human body key point heatmap to obtain the uniformly sampled frame as the original input for the next stage of sampling.
[0054] The process of uniformly sampling the stacked human body key point heatmap includes: dividing the video into equal-length segments and then uniformly sampling from each segment.
[0055] In this embodiment, uniform sampling specifically includes: using a certain number of frames as the total input frames, and uniformly sampling n frames from the entire total input frames.
[0056] Step 4: Process the uniformly sampled frames, including fusing the low-frequency and mid-frequency sampling channels. After fusion, the number of frames in the low-frequency and mid-frequency channels is the same as that in the high-frequency channel.
[0057] like Figure 2The diagram shows a flowchart of the multi-timescale downsampling process based on the improved 3D-CNN algorithm. The processing of the uniformly sampled frames specifically includes the following steps:
[0058] To improve the understanding of long videos, new features are formed through frame fusion. Three frame sampling methods are used to process uniformly sampled frames: low-frequency sampling channel, mid-frequency sampling channel, and high frame rate sampling channel.
[0059] (1) Low-frequency channel frame sampling
[0060] Low-frequency channels have fewer samples, typically no more than n / 4 frames, and are represented as channels for coarse feature analysis of the entire video.
[0061] (2) Intermediate frequency channel frame sampling
[0062] The intermediate frequency channel has a larger number of samples, usually no more than n / 3 frames, and is used as a feature analysis channel to further understand the overall video.
[0063] (3) High-frequency channel frame sampling
[0064] The high-frequency channel has the most samples, usually no more than n / 2 frames, and is represented as the feature analysis channel for a detailed understanding of the overall video.
[0065] The fusion processing of the low-frequency acquisition channel and the medium-frequency acquisition channel includes:
[0066] The low-frequency and mid-frequency frame acquisition channels are fused, resulting in the same frame count for both channels as the high-frequency channel. The fusion function for the low-frequency and mid-frequency frame acquisition channels is as follows:
[0067]
[0068] In the formula, T l T represents the low frame rate sampling channel. m T represents the medium frame sampling frequency channel. h This indicates the high frame rate frame acquisition channel.
[0069] Step 5: Convolve the channels in parallel for frames of different frequencies at multiple scales to obtain the corresponding feature information;
[0070] The parallel convolution of different frequency frames at multiple scales includes: multi-scale frame convolution and multi-scale frame feature fusion. Specifically, multi-scale frame convolution involves fusing the uniformly sampled low-frequency and mid-frequency frame channels at multiple scales using parallel convolution. Each channel uses an independent 3D convolution kernel and is connected via a skip connection. In detail, by first learning the low-frequency and mid-frequency frame channels to gain a basic understanding of the entire video, the number of input frames is increased, progressively deepening the granularity from coarse to fine, thereby obtaining a more detailed feature understanding than the original direct input. The learned multi-segment features are then fused, resulting in more effective learning of time-series information.
[0071] Multi-scale frame convolution also includes: using parallel convolution of high-frequency sampling channels after uniform sampling, using independent 3D convolution kernels between channels and using residual connections (skip-connection) for feature exchange, and increasing the number of input frames on this basis, continuously deepening from coarse-grained to fine-grained, and fusing the learned multi-segment features.
[0072] The channel division at multiple scales includes, but is not limited to, three frame acquisition channels, and may include multiple low-frequency channels, mid-frequency channels, and high-frequency channels.
[0073] Multi-scale frame feature fusion includes: substituting into the multi-segment feature fusion formula, as shown below:
[0074] F(n)=g φ (f l +f m )+f h
[0075] In the formula, F(n) represents the final fused feature, n is the number of input layers, and f l f represents the characteristics of the low frame rate sampling channel. m Indicates the channel characteristics of medium frame sampling frequency, f h denoted by , gφ is an aggregation function that fuses the low-frame-rate and medium-frame-rate channels.
[0076] In this embodiment, PoseC3D is used for behavioral feature extraction. PoseC3D improves upon SlowFast for pose and RGB optical flow features, proposing two 3D-CNNs: Pose-SlowOnly and RGBPose-SlowFast. RGBPose-SlowFast comprises two branches, processing RGB and skeletal modalities respectively. The RGB branch has a low frame rate and a larger network width, while the skeletal branch has a high frame rate and a smaller network width. Bidirectional connections exist between the two branches to promote feature fusion between modalities. PoseC3D often performs worse on long videos than on short videos. To address this issue, we propose a multi-temporal-scale network. By inputting frames at different temporal scales, we improve temporal convolution to achieve better understanding of the temporal dimension.
[0077] Step 6: The feature information is judged and normalized. Based on the probability results after recognition, the final output is the behavior recognition category.
[0078] The process of outputting the behavior recognition category based on the probability results after identification includes: analyzing and evaluating the behavior detection results, and optimizing and adjusting the algorithm based on experimental results to improve the accuracy and efficiency of behavior detection. Specifically, appropriate performance metrics are selected for evaluation based on the characteristics of the dataset and actual needs, including but not limited to accuracy, recall, and F1 score.
[0079] In summary, this invention provides a method for extracting video frames for skeletal behavior recognition across multiple time scales. This method improves the performance of the model on long videos while maintaining classification accuracy. By further sampling the uniformly sampled frames at low, medium, and high frequencies, the model can learn features at different video scales hierarchically, from coarse-grained to fine-grained, thereby enhancing the 3D-CNN's ability to understand long videos. Therefore, this invention solves the problem of traditional 3D-CNN's poor performance on long videos.
[0080] This invention also provides a system for extracting video frames for skeletal behavior recognition at multiple scales, the system comprising:
[0081] The video information acquisition module is used to input the original image, select the video characters in the original image, extract the 2D human pose key points in the video, and convert the acquired key points into heat maps or coordinate information for storage; if saved as coordinate information, it is converted into heat maps through a mapping function, and the heat maps are stacked to form the corresponding human key point heat map.
[0082] The video information input module is used to uniformly sample the stacked human key point heatmap and obtain the uniformly sampled frame as the raw input for the next stage of sampling.
[0083] The video information processing module is used to process uniformly sampled frames, fuse low-frequency and mid-frequency sampling channels, and after fusion, the number of frames in the low-frequency and mid-frequency channels is the same as that in the high-frequency channel; and convolve the channels in parallel for different frequency frames at multiple scales to obtain the corresponding feature information.
[0084] The video information output module is used to process the feature information after judgment and normalization, and finally output the behavior recognition category based on the probability results after recognition.
[0085] It is understood that the skeletal behavior recognition video frame extraction system provided by the present invention corresponds to the skeletal behavior recognition video frame extraction method provided in the foregoing embodiments. The relevant technical features of the skeletal behavior recognition video frame extraction system can be referred to the relevant technical features of the skeletal behavior recognition video frame extraction method, and will not be repeated here.
[0086] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for extracting video frames for skeletal behavior recognition at multiple time scales, characterized in that, Includes the following steps: Input the original image, select the video characters in the original image, extract the 2D human pose key points in the video, and convert the obtained key points into heat maps or coordinate information and save them. If saved as coordinate information, it is converted into a heatmap through a mapping function, and the heatmaps are stacked to form the corresponding human body key point heatmap. The stacked human key point heatmap is uniformly sampled. The video is divided into equal-length segments, and a certain number of frames from each segment are used as the total input frames. n frames are uniformly sampled from the total input frames to obtain the uniformly sampled frames as the original input for the next stage of sampling. The uniformly sampled frames are fused, including: dividing the uniformly sampled frames into low-frequency, mid-frequency, and high-frame-rate sampling channels according to image resolution; fusing the sampled frames from the low-frequency and mid-frequency channels so that the number of frames in the fused low-frequency and mid-frequency channels is the same as that in the high-frequency channel; and substituting the fused sampled frames from the low-frequency and mid-frequency channels into the following formula: In the formula, T l T represents the low frame rate sampling channel. m T represents the medium frame sampling frequency channel. h This indicates the high frame rate sampling channels, and num represents the total number, meaning that the number of low frame rate sampling channels and medium frame rate sampling channels, after feature concatenation, is consistent with the number of high frame rate sampling channels. Parallel convolution channels are used for frames of different frequencies at multiple scales, including: using parallel convolution channels, using independent 3D convolution kernels between channels and using residual connections for feature exchange, learning low-frequency and mid-frequency frame channels, increasing the number of input frames, deepening from coarse-grained to fine-grained, fusing the learned multi-segment features, and obtaining the corresponding feature information. The feature information is judged and normalized, and the final output is the behavior recognition category based on the probability results after recognition.
2. The method for extracting video frames for skeletal behavior recognition at multiple time scales according to claim 1, characterized in that, The step of selecting video characters in the original image includes: The object detection algorithm is used to select people in the video, and the output is the processed original image containing the candidate boxes.
3. The method for extracting video frames for skeletal behavior recognition at multiple time scales according to claim 1, characterized in that, The extracted 2D human pose key points from the video include: A top-down human pose estimation model is used to estimate the pose of the selected human body. Subnets are gradually added from high resolution to low resolution to form more stages, and the multi-resolution subnets are connected in parallel. Throughout the process, information is repeatedly exchanged on the parallel multi-resolution subnets to perform multi-scale repeated fusion. The final output of the network serves as the key point for the overall estimation.
4. The method for extracting video frames for skeletal behavior recognition at multiple time scales according to claim 1, characterized in that, The learned multi-segment features are then fused and substituted into the following formula: In the formula, F(n) represents the final fused feature, and n is the number of input layers; f l f represents the characteristics of the low frame rate sampling channel. m Indicates the channel characteristics of medium frame sampling frequency, f h Indicates the characteristics of high frame rate sampling channels, g It is an aggregation function that fuses frames captured from low frame rate and medium frame rate channels.
5. The method for extracting video frames for skeletal behavior recognition at multiple time scales according to claim 1, characterized in that, The final output of the behavior recognition category based on the probability results after recognition includes: The behavior detection results are analyzed and evaluated. Based on the characteristics of the dataset and actual needs, appropriate performance indicators are selected for evaluation. Based on the experimental results, the algorithm is optimized and adjusted. The appropriate performance indicators include, but are not limited to, accuracy, recall, and low frame rate frame channel feature values.
6. A system for extracting video frames for skeletal behavior recognition across multiple time scales, characterized in that, The system includes: The video information acquisition module is used to input the original image, select the video characters in the original image, extract the 2D human pose key points in the video, and convert the acquired key points into heat maps or coordinate information for storage; if saved as coordinate information, it is converted into heat maps through a mapping function, and the heat maps are stacked to form the corresponding human key point heat map. The video information input module is used to uniformly sample the stacked human key point heatmap. After dividing the video into equal-length segments, a certain number of frames are taken from each segment as the total input frames. Then, n frames are uniformly sampled from the total input frames to obtain the uniformly sampled frames as the original input for the next stage of sampling. The video information processing module is used to perform fusion processing on uniformly sampled frames, including: dividing the uniformly sampled frames into low-frequency, mid-frequency, and high-frame-rate sampling channels according to the image resolution; fusing the sampled frames of the low-frequency and mid-frequency sampling channels, so that the number of frames in the low-frequency and mid-frequency channels is the same as that in the high-frequency channel after fusion; and using parallel convolution channels for different frequency frames at multiple scales, including: using parallel convolution channels, with independent 3D convolution kernels between each channel and residual connections for feature exchange; learning the low-frequency and mid-frequency sampling channels, increasing the number of input frames, deepening from coarse-grained to fine-grained, and fusing the learned multi-segment features to obtain the corresponding feature information. The video information output module is used to process the feature information after judgment and normalization, and finally output the behavior recognition category based on the probability results after recognition.