A method for extracting key frames from motion videos based on fusion features of human posture recognition

Through human posture recognition and spatial graph convolution fusion features, combined with self-attention mechanism and reinforcement learning, the missed detection and false detection problem in the extraction of keyframes of motion videos is solved, which improves the accuracy and completeness of extraction, and reduces the calculation and storage burden.

CN114724058BActive Publication Date: 2024-11-15SHANDONG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210245767.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-14
Publication Date
2024-11-15
Estimated Expiration
2042-03-14

AI Technical Summary

Technical Problem

运动类视频的关键帧提取存在漏检和误检问题,且传统方法在计算和存储上存在较大负担,难以有效节省空间。

Method used

The fusion feature method based on human posture recognition is adopted, and the motion features are extracted through the residual network and the spatial graph convolution is extracted, and linear weighting is performed, combining the self-attention mechanism and reinforcement learning to extract and optimize keyframes.

Benefits of technology

Improve the accuracy and completeness of keyframe extraction, reduce the amount of calculation, avoid missed detection and misdetection, and effectively save storage space.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114724058B_ABST
    Figure CN114724058B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for extracting key frames of fusion feature motion video based on human posture recognition, comprising the following steps: S1. segmenting the target video segment frame by frame; S2. extracting static features using a residual network, dimensionalizing the data, and obtaining static features of the video frame; S3. abstracting the skeleton data of the human body in three-dimensional space, extracting motion features of the video frame, and obtaining motion features S d ; S4. Perform linear weighted processing on the extracted static features and motion features according to the weights; S5. Extract global features from the fused features through the self-attention mechanism, and then calculate the importance of the video frame, extract the corresponding action key frames through the Bernoulli function, and use reinforcement learning to optimize the result set. Its advantage is that the result set can be optimized, effectively improving the problems of motion target feature loss and key frame missed detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of video processing, and in particular to a method for extracting key frames from motion videos by integrating features. Background Art

[0002] For a video, it is a sequence of images. Its content is much richer than a single image, and it has strong expressiveness and large amount of information. Generally speaking, the analysis of a video is performed after decomposing the video into video frames. However, there is usually a lot of redundancy in video frames. Extracting the key frames of the video for analysis can effectively reduce the computing time.

[0003] With the development of the Internet, multimedia information retrieval has an increasingly greater impact on all areas of society. Traditional video retrieval methods use image retrieval methods to search frame by frame. This method requires processing a large amount of image information, which places a heavy burden on information transmission and calculation. In addition, with the popularity of home video equipment today, we often need to save a surveillance video, but the storage of video information requires a lot of storage space. Using video keyframes to store video information can not only maintain the authenticity of the video information, but also save space to a great extent.

[0004] For sports videos, the states of moving objects change frequently. Due to the diversity of moving targets and the similarity of actions, it is easy to miss detections if only motion features are considered, and the deviation of feature extraction may be large. Therefore, the present invention studies the key frame extraction technology for sports videos by fusing features. Summary of the invention

[0005] To solve the above problems, the present application provides a method for extracting key frames from motion videos with fusion features. To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] A method for extracting key frames from fusion feature motion videos based on human posture recognition comprises the following steps:

[0007] S1. segment the target video segment frame by frame, and divide the video into a series of video frames;

[0008] S2. Use the residual network to extract static features and reduce the dimension of the data to obtain the static features of the video frame S s =[S s1 ,S s2 ,…,S sT ];

[0009] S3. Abstract the skeleton data of the human body in three-dimensional space, extract the motion features of the video frame, and obtain the motion feature S d =[S d1 ,Sd2 ,…,S dT ];

[0010] S4. Extract the static features S s and sports characteristics S d Perform linear weighting processing according to the weight size, S = mS s +nS d , m and n are the weight factors of static features and motion features respectively;

[0011] S5. The global features are extracted from the fused features through the self-attention mechanism, and then the importance of the video frame is calculated. The corresponding action key frames are extracted through the Bernoulli function, and reinforcement learning is used to optimize the result set.

[0012] Further preferably, the specific method of step S3 is as follows:

[0013] S31. Extract the human skeleton for each frame in the video and use lightweight HRNet to perform human posture analysis;

[0014] S32. Taking the coordinates and confidence of the skeleton key points identified in each frame of the video as input, constructing a topological map according to the physical connection between the bones, and then performing batch normalization processing on it;

[0015] S33. The processed data is passed through multiple S-GCN units for feature extraction, and different weight coefficients are assigned to different torsos to obtain the feature representation S of the video. d ={S d1 ,S d2 ,…,S dT}.

[0016] Further preferably, the specific method of step S31 is as follows:

[0017] S311. The subnet of each branch in each stage includes two residual blocks and a multi-resolution fusion module;

[0018] S312. All residual blocks in the original network are replaced by the Shuffle module of ShuffleNet. The Shuffle module divides the channel into two parts, one part is passed directly without any convolution operation, and the other part needs to be subjected to depthwise separable convolution;

[0019] S313. Replace the convolution in the depthwise separable convolution with channel weighting, downsample by average pooling, and adjust to the same size as the minimum resolution. Perform channel addition feature fusion on the feature maps of different resolutions of the i branches after processing, and use the SE module to obtain the weight matrix W t , the weight matrix W tEach branch is upsampled to restore its original size and the channels are weighted.

[0020] Further preferably, the specific method of step S5 is as follows:

[0021] S51. Modeling position information between video frames by bidirectional mask;

[0022] S52. After obtaining the global context information of the video sequence, the feature matching degree is calculated based on the global correlation feature, and then the importance score of the video frame is predicted using the fully connected layer;

[0023] S53. After obtaining the frame score of each video frame, a key frame is selected for the corresponding action through Bernoulli distribution. t ~B(Y), a t Expressed as the probability of taking the current frame as a key frame;

[0024] S54. Use reinforcement learning to judge the quality of the extracted key frame result set. This paper uses the state-action value to represent the sum of the importance and diversity of the result set. The key frame set's ability to cover the complete video information is used to evaluate the importance of the result set. The difference in the feature space between the selected frames is used to evaluate the diversity of the result set.

[0025] Further preferably, the specific method of step S51 is as follows:

[0026] S511. The forward mask indicates that the attention weight is related to the calculation result before the current position, and the reverse mask indicates that the weight of the current position is related to the calculation result after that;

[0027] S512. Input T frames of video X = {x i |i=1,…,T}, each frame contains N key points, and the correlation coefficient can be calculated through the self-attention mechanism

[0028]

[0029] Where t,i∈[0,T), U and V are the weight matrices of the two frames, M is the position encoding matrix, the forward mask retains the upper triangular information, and the reverse mask retains the lower triangular information. λ is the eigenvalue of the fusion feature matrix, s t is the fusion feature of the current frame, s i It is the fusion feature of the frames before and after this frame;

[0030] S513. Correlation coefficient Combined with the relative position information of the frame (representing the position relationship with the previous and next frames), we get And merge the positive and negative directions, Mapping back to the original video frame sequence, we get a sequence c containing context information = {c t |t=1,…,T}.

[0031] Beneficial Effects

[0032] (1) The present invention provides a motion video key frame extraction technology with fused features through human posture recognition, spatial graph convolution and feature fusion, which meets the requirements for key frame extraction accuracy and completeness.

[0033] (2) The present invention proposes a method for extracting video frame features, which fuses the static features extracted by human posture recognition and the motion features extracted by spatial graph convolution as the final video frame features, and performs importance analysis on them, which can effectively avoid the problems of missed detection and false detection.

[0034] (3) The present invention improves HRNet by replacing the residual module and adding an attention mechanism, which greatly reduces the amount of computation without losing accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of the stages of a method for extracting key frames from motion videos using fusion features disclosed in an embodiment of the present invention;

[0036] Figure 2 A specific schematic diagram of a human posture recognition module of a method for extracting key frames from motion videos using fusion features disclosed in an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of key frame extraction results of a method for extracting key frames from motion videos using fusion features disclosed in an embodiment of the present invention; DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0039] The present invention provides a method for extracting key frames of motion videos by integrating features, such as Figure 1 As shown, this method combines the static features extracted by the lightweight human posture recognition algorithm and the motion features extracted by the spatial graph convolution, thereby improving the accuracy and completeness of key frame detection. The specific embodiments are as follows:

[0040] (1) Segment the target video segment frame by frame and divide the video into a series of video frames.

[0041] (2) In order to better preserve the original information in the input image and reduce the loss, the residual network ResNet50 is used to extract static features and reduce the data dimension to 256 dimensions. The obtained video frame static features are represented as S s =[S s1 ,S s2 ,…,S sT ].

[0042] (3) The skeleton data of the human body in three-dimensional space is abstracted, and the lightweight HRNet algorithm is used to analyze the human posture. Then, the ST-GCN network is used to extract the motion features of the video frame to obtain the motion feature S d =[S d1 ,S d2 ,…,S dT ].

[0043] The specific method of step (3) is as follows:

[0044] (3.1) The human skeleton is extracted for each frame in the video. In order to improve the accuracy without increasing the computational burden, this application uses a lightweight HRNet for human posture analysis.

[0045] like Figure 2 As shown, the specific method of step (3.1) is as follows:

[0046] (3.1.1) For HRNet, the accuracy is much higher than other bottom-up algorithms, but the obvious drawback is that the number of parameters is large and the operation speed is slow. Therefore, this paper makes lightweight improvements to this obvious shortcoming and establishes a method to speed up the analysis.

[0047] (3.1.2) In order to make the model as lightweight as possible, the depth and width of the original HRNet network are first reduced, and the subnetwork of each branch in each stage is reduced to two residual blocks and a multi-resolution fusion module.

[0048] (3.1.3) All residual blocks in the original network are replaced by the Shuffle module of ShuffleNet. This module divides the channel into two parts, one part passes directly without any convolution operation, and the other part needs to undergo depthwise separable convolution.

[0049] (3.1.4) The 1×1 convolution in the depthwise separable convolution is replaced by channel weighting, which also achieves the effect of information exchange, but the time complexity is much lower than that of 1×1 convolution. Through average pooling downsampling, the feature map is adjusted to the same size as the minimum resolution, and the feature maps of different resolutions of the i branches are added (channel addition feature fusion) after processing, and then the SE module (including Squeeze and Excitation) is used to obtain the weight matrix W t , the weight matrix W t Each branch is upsampled to restore its original size and the channels are weighted.

[0050] (3.2) The coordinates and confidence of the skeleton key points identified in each frame of the video are used as input, and a topological map is constructed based on the physical connection between the bones. Then, batch normalization is performed on it to unify the scattered data.

[0051] (3.3) The processed data is passed through 9 S-GCN units for feature extraction, and different weight coefficients are assigned to different torsos to obtain the feature representation S of the video. d ={S d1 ,S d2 ,…,S dT}.

[0052] (4) Perform linear weighting on the extracted static features and motion features according to their weights, S = mS s +nS d , m and n are the weight factors of static features and motion features respectively;

[0053] (5) The global features are extracted from the fused features through the self-attention mechanism, and then the importance of the video frame is calculated. The corresponding action key frames are extracted through the Bernoulli function, and reinforcement learning is used to optimize the result set.

[0054] The specific method of step (5) is as follows:

[0055] (5.1) By modeling the position information between video frames with bidirectional masks, it can be ensured that the importance of the current video frame is affected not only by the previous video frame, but also by the subsequent video frame.

[0056] The specific method of step (5.1) is as follows:

[0057] (5.1.1) The forward mask indicates that the attention weight is related to the calculation result before the current position, and the reverse mask indicates that the weight of the current position is related to the calculation result after that.

[0058] (5.1.2) Input T frames of video X = [x i|i=1,…,T}, each frame contains N key points, and the correlation coefficient can be calculated through the self-attention mechanism

[0059]

[0060] Where t,i∈[0,T), U and V are the weight matrices of the two frames respectively, M is the position encoding matrix, the forward mask retains the upper triangular information, and the reverse mask retains the lower triangular information.

[0061] (5.1.3) The correlation coefficient Combined with the relative position information of the frame (representing the position relationship with the previous and next frames), we get And merge the positive and negative directions, Mapping back to the original video frame sequence, we get a sequence c containing context information = {c t |t=1,…,T}.

[0062] (5.2) After obtaining the global context information of the video sequence, the feature matching degree is calculated based on the global correlation feature, and then the importance score of the video frame is predicted using the fully connected layer.

[0063] (5.3) After obtaining the frame score of each video frame, the key frame is selected for the corresponding action through Bernoulli distribution. t ~B(Y), a t It is expressed as the probability of taking the current frame as a key frame.

[0064] (5.4) Reinforcement learning is used to judge the quality of the extracted key frame result set. This paper uses the state-action value to represent the sum of the importance and diversity of the result set. The key frame set’s ability to cover the complete video information is used to evaluate the importance of the result set. The difference in the feature space between the selected frames is used to evaluate the diversity of the result set.

[0065] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles disclosed herein.

Claims

1. A method for extracting key frames from motion video using fusion features based on human posture recognition, characterized in that: The following steps are involved: S1. segment the target video segment frame by frame, and divide the video into a series of video frames; S2. Use the residual network to extract static features and reduce the dimension of the data to obtain the static features of the video frame S s =[S s1 ,S s2 ,…,S sT ]; S3. Abstract the skeleton data of the human body in three-dimensional space, extract the motion features of the video frame, and obtain the motion feature S d =[S d1 ,S d2 ,…,S dT ]; S4. Extract the static features S s and motion characteristics S d Perform linear weighting processing according to the weight size, S = mS s +nS d , m and n are the weight factors of static features and motion features respectively; S5. Extract global features from the fused features through the self-attention mechanism, then calculate the importance of the video frame, extract key frames for the corresponding actions through the Bernoulli function, and use reinforcement learning to optimize the result set; the specific method is as follows: S51. Modeling position information between video frames by bidirectional mask; S52. After obtaining the global context information of the video sequence, the feature matching degree is calculated based on the global correlation feature, and then the importance score of the video frame is predicted using the fully connected layer; S53. After obtaining the frame score of each video frame, a key frame is selected for the corresponding action through Bernoulli distribution. t ~B(Y), a t Expressed as the probability of taking the current frame as a key frame; S54. Use reinforcement learning to judge the quality of the extracted key frame result set, use the state-action value to represent the sum of the importance and diversity of the result set, use the key frame set's ability to cover the complete video information to evaluate the importance of the result set, and use the difference in the feature space between the selected frames to evaluate the diversity of the result set.

2. The method for extracting key frames of motion video based on fusion feature recognition of human posture according to claim 1, characterized in that: The specific method of step S3 is as follows: S31. Extract the human skeleton for each frame in the video and use lightweight HRNet to perform human posture analysis; S32. Taking the coordinates and confidence of the skeleton key points identified in each frame of the video as input, constructing a topological map according to the physical connection between the bones, and then performing batch normalization processing on it; S33. The processed data is passed through multiple S-GCN units for feature extraction, and different weight coefficients are assigned to different torsos to obtain the feature representation S of the video. d ={S d1 ,S d2 ,…,S dT }.

3. The method for extracting key frames of motion video based on fusion feature recognition of human posture according to claim 2, characterized in that: The specific method of step S31 is as follows: S311. The subnet of each branch in each stage includes two residual blocks and a multi-resolution fusion module; S312. All residual blocks in the original network are replaced by the Shuffle module of ShuffleNet. The Shuffle module divides the channel into two parts, one part is passed directly without any convolution operation, and the other part needs to be subjected to depthwise separable convolution; S313. Replace the convolution in the depthwise separable convolution with channel weighting, downsample by average pooling, and adjust to the same size as the minimum resolution. Perform channel addition feature fusion on the feature maps of different resolutions of the i branches after processing, and use the SE module to obtain the weight matrix W t , the weight matrix W t Each branch is upsampled to restore its original size and the channels are weighted.

4. The method for extracting key frames of motion video based on fusion feature recognition of human posture according to claim 1, characterized in that: The specific method of step S51 is as follows: S511. The forward mask indicates that the attention weight is related to the calculation result before the current position, and the reverse mask indicates that the weight of the current position is related to the calculation result after that; S512. Input T frames of video X = {x i |i=1,…,T}, each frame contains N key points, and the correlation coefficient is calculated through the self-attention mechanism Where t,i∈[0,T), U and V are the weight matrices of the two frames, M is the position encoding matrix, the forward mask retains the upper triangular information, and the reverse mask retains the lower triangular information. λ is the eigenvalue of the fusion feature matrix, s t is the fusion feature of the current frame, s i It is the fusion feature of the frames before and after this frame; S513. Correlation coefficient Combined with the relative position information of the frame, we get And merge the positive and negative directions, Mapping back to the original video frame sequence, we get a sequence c containing context information = {c t |t=1,…,T}.