Action segmentation method based on human skeleton and action boundary regression optimization
Through the action segmentation method based on human body skeleton and action boundary regression optimization, the problem of difficult fine-grained semantic information processing with fast movement changes in the prior art is solved, and the accuracy and robustness of action segmentation are improved.
Patent Information
- Application Number
- CN202411836891.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-13
- Publication Date
- 2025-04-15
AI Technical Summary
Existing action segmentation methods are difficult to process fine-grained semantic information with fast action changes, resulting in low accuracy of action segmentation.
The action segmentation method based on the optimization of human skeleton and action boundary regression is adopted. The human skeleton features are extracted through the ST-GCN network to build an action segmentation network, and the action segmentation results are optimized using the LFE module, action segmentation branch network and boundary regression branch network.
It improves the accuracy of action segmentation, can better process rapidly changing action information, reduces dependence on video content scenes and object information, and enhances robustness.
Smart Images
Figure CN119832631A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of action segmentation, and in particular to an action segmentation method based on human skeleton and action boundary regression optimization. Background Art
[0002] Action segmentation is a prerequisite for video action recognition, which requires accurately identifying the start and end frames of an action in the video stream. In video surveillance systems, action segmentation can be used to detect abnormal behaviors and improve the security and reliability of the system. In the field of human-computer interaction, action segmentation can help machines better understand human action commands and achieve more natural and efficient human-computer interaction.
[0003] However, existing action segmentation methods have the following problems:
[0004] 1. Existing models are relatively easy to handle coarse-grained temporal action segmentation tasks; however, for the more challenging frame-level action classification, existing models are difficult to meet the application requirements of fine-grained semantics;
[0005] 2. In most temporal action segmentation datasets, scenes, tools, and objects contain more direct action category features, which are sometimes more important than the action itself; however, in practical applications, more attention should be paid to action features; in addition, task-driven temporal action segmentation datasets cannot show the complete human body, so it is difficult to extract more modal features to perform temporal action segmentation tasks;
[0006] 3. Task-driven temporal action segmentation datasets usually have simple action content categories, and the speed differences between different actions are small, which is difficult to cause changes in frame-level features and poses less challenges to the temporal action segmentation task. Summary of the invention
[0007] In view of the shortcomings of the existing methods, the present invention solves the problem that the existing action segmentation methods rely on the scene and object information of the video content and cannot process the fine-grained semantic information of fast-changing actions, thus resulting in low action segmentation accuracy.
[0008] The technical solution adopted by the present invention is: an action segmentation method based on human skeleton and action boundary regression optimization, comprising the following steps:
[0009] Step 1: Segment the video into actions and build an action video dataset;
[0010] As a preferred implementation of the present invention, the action segmentation is performed by retaining the moving human body, moving the moving human body to the center of the video, and filling the surrounding of the video with black pixels.
[0011] Step 2: Extract human skeleton features of each frame using action video;
[0012] As a preferred implementation of the present invention, the ST-GCN network is used to extract the human skeleton features of each frame.
[0013] Step 3: construct an action segmentation network consisting of an LFE module, an action segmentation branch network, and a boundary regression branch network, input human skeleton features into the LFE module to output shared action features;
[0014] As a preferred embodiment of the present invention, the LFE module includes Conv1d and 10 stacked dilated residual layers.
[0015] As a preferred embodiment of the present invention, the formula for expanding the residual layer is:
[0016]
[0017]
[0018] In the formula, H l represents the output of the lth dilated residual layer, is the intermediate state of the output, · represents the convolution operation, W1 and b1 are the weight and bias of the dilated convolution, and W2 and b2 are the weight and bias of the 1×1 convolution.
[0019] As a preferred implementation of the present invention, the formula for sharing action features is:
[0020]
[0021] In the formula, Conv 1d represents one-dimensional convolution, DRL represents dilated residual layer; X represents human skeleton features.
[0022] Step 4: Input the shared action features into the action segmentation branch network and the boundary regression branch network respectively;
[0023] As a preferred embodiment of the present invention, the action segmentation branch network includes: a first Conv1d, three TCN modules are cascaded, the TCN module is cascaded by an LFE module and a Conv1d, and the first Conv1d, the first and second TCN modules are connected with Softmax.
[0024] As a preferred embodiment of the present invention, the boundary regression branch network replaces the first Conv1d of the action segmentation branch network with conv_bound, and replaces Softmax with Sigmoid.
[0025] Step 5: Use the action boundary predicted by the boundary regression branch network to optimize the action category of the action segmentation branch network and output the action segmentation result;
[0026] As a preferred implementation of the present invention, step five specifically includes:
[0027] Step 51: Construct the square error loss function L s , the formula is;
[0028]
[0029] Where T is the number of frames in the video, N is the number of action categories, and y t,c is the probability of the predicted category c of the action segmentation branch network at time t, y t,b,c is the probability of predicting category c after optimizing the action boundary at time t;
[0030] Step 52: Use the boundary regression loss function to optimize the action segmentation. The formula is:
[0031]
[0032] In the formula, λ s YesL s The weight, L ce is the cross entropy loss, L GS-TMSE is Gaussian similarity weighted TMSE loss;
[0033] Step 53: Calculate the loss L of the action segmentation branch asb , the formula is:
[0034]
[0035] Where N as The number of predictions for the action segmentation branch;
[0036] Step 54: Calculate the loss L of the boundary regression branch brb , the formula is:
[0037] Where N br is the number of predictions of the boundary regression branch, L bl is the binary logistic regression loss;
[0038] Step 55: Add the losses of the action segmentation branch and the boundary regression branch to obtain the total loss of the action segment refinement network, which is: L = L asb +λL brb (13)
[0039] Where λ is the weight of the boundary regression branch loss.
[0040] As a preferred embodiment of the present invention, an action segmentation method and system based on human skeleton and action boundary regression optimization includes: a memory for storing instructions executable by a processor; a processor for executing instructions to implement an action segmentation method based on human skeleton and action boundary regression optimization.
[0041] Beneficial effects of the present invention:
[0042] 1. The present invention focuses on fast-moving independent human actions, accurately models action features through human skeleton information, and enables the network to focus on fast-changing motion information instead of relying on scene and object information of video content, thereby improving the accuracy of human action segmentation;
[0043] 2. The present invention extracts human skeleton features instead of RGB visual features, and by classifying actions with similar skeleton movements, more detailed fine-grained semantic requirements can be constructed;
[0044] 3. The present invention only extracts the motion features of the human skeleton as the judgment of the action category, and does not rely on elements such as scenes, tools and objects. The network pays more attention to the human motion features in the video;
[0045] 4. The present invention demonstrates good robustness in the task of motion segmentation. Compared with the existing methods, the present invention has a simple structure, is easy to use, and only extracts the motion features of the human skeleton, with higher detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 It is a flow chart of the action segmentation method based on human skeleton and action boundary regression optimization of the present invention;
[0047] Figure 2 is a schematic diagram of the dilated residual layer structure of the present invention;
[0048] Figure 3 is the training loss graph of the present invention;
[0049] Figure 4 is the action segmentation visualization graph of the present invention; DETAILED DESCRIPTION
[0050] The present invention is further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore it only shows the components related to the present invention.
[0051] like Figure 1 As shown in FIG, the action segmentation method based on human skeleton and action boundary regression optimization includes:
[0052] Step 1: Segment the video into actions and build an action video dataset;
[0053] The input video resolution is set to 640*480, the video frame rate is set to 30FPS, irrelevant people in the video are removed, only the key moving people are retained, and the moving people are moved to the center of the video, and the video is filled with black pixels around. The action video is segmented, and 70% of the action video dataset is divided into a training set, 30% into a test set, and 20% of the training set is divided into a validation set.
[0054] Step 2: Input the action video into the ST-GCN network and extract the human skeleton features X = [x1,...,x T ]∈R T ×D ;
[0055] Where T is the number of frames in the video and D is the dimension of the feature.
[0056] That is, extract 25 key points of the complete human skeleton, and each key point saves three-dimensional data: X coordinate, Y coordinate, visibility;
[0057] Among them, visibility is divided into 0, 1, and 2; 0 means out of the field of view, 1 means within the field of view but blocked and invisible, and 2 means within the field of view and visible.
[0058] When the human skeleton features are input into the ST-GCN network, the 0th and 1st dimensions of the features are merged, and the dimension of the merged skeleton features is (75, T).
[0059] Step 3: Construct an action segmentation network consisting of an LFE module (Long-term Feature Extractor), an action segmentation branch network, and a boundary regression branch network; input human skeleton features into the long-term feature extractor to expand the temporal receptive field in the skeleton features, capture the long-term dependencies between actions, and provide shared action features for the action segmentation branch and the boundary regression branch;
[0060] The LFE module includes 1 one-dimensional convolution for feature dimension conversion and 10 stacked dilated residual layers; that is, it includes Conv1d and dilated residual layers*10.
[0061] Figure 2: is a schematic diagram of the dilated residual layer structure of the present invention, the dilated residual layer consists of a one-dimensional dilated convolution (Dilated Conv) with a convolution kernel size of 3, a ReLU activation function, a one-dimensional convolution with a convolution kernel size of 1, and a Dropout layer, and the output and input features of the Dropout layer are residually connected; wherein, the number of input feature map channels and the number of output feature map channels of the dilated convolution and 1*1 convolution are both 64; the padding and dilation of the dilated convolution are set to the current layer dilation rate to ensure that the size of the output feature map after the convolution operation is the same as the input; the discard rate of the Dropout layer is set to 0.5, and the dilation rate of each dilated residual layer is doubled compared to the previous layer (1,2,4,....,512), and the dilated residual layer can be expressed as:
[0062]
[0063]
[0064] In the formula, H l represents the output of the lth dilated residual layer, is the intermediate state of the output, · represents the convolution operation, W1 and b1 are the weight and bias of the dilated convolution, and W2 and b2 are the weight and bias of the 1×1 convolution.
[0065] The dilated residual layer can obtain the long-term dependencies in the video human skeleton features X, so that the network has a larger temporal receptive field and extracts richer shared action features X':
[0066] X'=Conv 1d (DRL *10 (X)) (3)
[0067] In the formula, Conv 1d represents a one-dimensional convolution, which is used to convert the feature dimension from 2048 to 64, DRL represents the dilated residual layer, X'∈R T*D' , D' is the feature dimension.
[0068] Step 4: Input the shared action features into the action segmentation branch network and the boundary regression branch network respectively;
[0069] Among them, the action segmentation branch predicts the action category C = [c1,...,c T ], and the boundary regression branch outputs the action boundary probability B = [b1,...,b T ];
[0070] The action boundary predicted by the boundary regression branch network is used to optimize the classification results of the action segmentation branch. The principle is:
[0071] The predicted action boundary B∈{0,1} whose local maximum value in the statistical action boundary probability P is greater than a specific threshold (boundary_th) T , and divide the video into different action segments according to these predicted action boundaries B; assuming that each action segment contains only a single action type, count the action category that appears most frequently in the segment according to the prediction results of the action segmentation branch, and update the action category label of the entire segment to this majority category; if multiple categories appear the same number of times, select the category with the largest total probability as the action category of the segment according to the probability value output by the action segmentation branch.
[0072] The action segmentation branch network receives the feature X' output by the long-term LFE module and predicts the action category C of each frame; the action segmentation branch consists of a conv_cls layer, a Softmax activation function and three stacked single-stage TCN structures. The main components and layer names of the action segmentation branch network are shown in Table 1.
[0073] The network structure of the boundary regression branch is the same as that of the action segmentation branch, but the conv_cls layer is replaced by the conv_bound layer, the number of output channels is replaced by 1, and the multi-classification Softmax activation function is replaced by the binary Sigmoid activation function.
[0074] Table 1 Main components and layer names of action segmentation branch
[0075]
[0076]
[0077] Step 5: Use the action boundary predicted by the boundary regression branch network to optimize the action category of the action segmentation branch network and improve the accuracy of action segmentation;
[0078] Step 51, using the squared difference loss to measure the overall difference between the action category output by the action segmentation branch network and the action category of the action segmentation branch network optimized by the action boundary of the boundary regression branch network, as shown in the following formula;
[0079]
[0080] Where T is the number of frames in the video, N is the number of action categories, and y t,c is the probability of the predicted category c of the action segmentation branch network at time t, y t,b,c is the probability of predicting category c after optimizing the action boundary at time t.
[0081] Step 52: After adding the squared error loss, the network uses boundary regression prediction to continuously optimize the segmentation results of action segmentation during training, and uses boundary regression results to further strengthen the action segmentation branch during inference to improve the network's recognition accuracy for continuous actions.
[0082] In the action segmentation branch network after action boundary optimization, the loss function of each prediction is:
[0083] L as =L ce +L GS-TMSE +λ s L s (5)
[0084] In the formula, λ s YesL s The weight is 0.1, L ce is the cross entropy loss, and the formula is:
[0085]
[0086] L GS-TMSE is the Gaussian similarity weighted TMSE loss, the formula is:
[0087]
[0088] In the formula, x t is the similarity index of frame t, σ is the variance, and its size is 1, Yes t,c The truncation result indicates the probability change of category c at time t.
[0089]
[0090] Where τ is the probability conversion threshold, which is 4; Δ t,c It represents the absolute value of the logarithmic difference of the probability of predicting category c at two adjacent moments (time t and time t-1), which is used to measure the change in probability.
[0091] Step 53: By averaging all predicted losses, the loss of the action segmentation branch is obtained, and the formula is:
[0092]
[0093] Where N as is the number of predictions for the action segmentation branch, with a size of 4.
[0094] In the boundary regression branch, each prediction uses binary logistic regression as the loss function, the formula is:
[0095]
[0096] In the formula, y t and p t are the true boundary and predicted action boundary probabilities of frame t respectively; since the number of action boundary frames is much less than other frames, weight w is applied to positive samples p , whose size is equal to the inverse of the ratio of the number of action boundary frames in the training data to the total number of frames.
[0097] Step 54: Average all predicted losses to obtain the loss of the boundary regression branch, which is:
[0098]
[0099] Where N br is the number of predictions for the boundary regression branch, of size 4.
[0100] Step 55, add the losses of the action segmentation branch and the boundary regression branch to obtain the total loss of the action segment refinement network, which is:
[0101] L=L asb +λL brb (13)
[0102] Where λ is the weight of the boundary regression branch loss, which is 0.1.
[0103] The action segmentation network model was trained using 5-fold cross validation.
[0104] Experimental process:
[0105] The test platform and experimental environment of the present invention are: Windows 10 Professional operating system, Intel(R) Xeon(R) Gold 6248R CPU, 128GB memory, NVIDIARTX 3090GPU, Windows 10 operating system, Pytorch 1.10.2, CUDA 11.4, and Python 3.8.12.
[0106] The training parameter settings of the network are shown in Table 2, using 5-fold cross validation.
[0107] Table 2 Training parameters
[0108]
[0109] Figure 3 is the training loss curve of the network. From the training loss curve of the network, we can see that the network reached the lowest training loss of 0.8917 in the 91st epoch. Therefore, the model weights generated in the 91st epoch are used to evaluate the network performance. The boundary regression performance of the network is shown in Table 3.
[0110] Table 3 Network boundary regression performance
[0111]
[0112] The action segmentation performance of the network is shown in Table 4. It can be seen from Table 4 that the use of boundary regression optimization can effectively improve the action segmentation performance of the network. The network has improved by 1.2% in frame-by-frame accuracy, 9.49% in segment edit distance, and up to 16.50% in segment F1 score. It can also be observed from the table that when the network uses a larger overlap threshold k on the segment F1 score, the improvement using boundary regression optimization is also greater: F1s@0.1 optimization improves by 14.79%, while F1s@0.5 optimization improves by 16.50%. This verifies that the boundary regression branch can effectively improve the recognition accuracy of the action segmentation branch for action segments that highly overlap with the real segment by predicting the start and end positions of the action.
[0113] Table 4 Network action segmentation performance
[0114]
[0115] Figure 4 This is the action segmentation visualization diagram of the present invention. It can be seen that the network readjusts the prediction results of the action segmentation branch according to the action boundary information, and corrects some misclassified frames. For example, the black area in the action segmentation prediction result is shortened, and the purple area on its right is expanded. At the same time, the purple part in the green area is deleted, making it closer to the real action segmentation result.
[0116] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.
Claims
1. An action segmentation method based on human skeleton and action boundary regression optimization, characterized in that: The following steps are involved: Step 1: Segment the video into actions and build an action video dataset; Step 2: Extract human skeleton features of each frame using action video; Step 3: construct an action segmentation network consisting of an LFE module, an action segmentation branch network, and a boundary regression branch network, input human skeleton features into the LFE module to output shared action features; Step 4: Input the shared action features into the action segmentation branch network and the boundary regression branch network respectively; Step 5: Use the action boundary predicted by the boundary regression branch network to optimize the action category of the action segmentation branch network and output the action segmentation result.
2. The action segmentation method based on human skeleton and action boundary regression optimization according to claim 1 is characterized in that: The LFE module consists of Conv1d and 10 stacked dilated residual layers.
3. The action segmentation method based on human skeleton and action boundary regression optimization according to claim 2 is characterized in that: The formula for the dilated residual layer is: In the formula, H l represents the output of the lth dilated residual layer, is the intermediate state of the output, · represents the convolution operation, W1 and b1 are the weight and bias of the dilated convolution, and W2 and b2 are the weight and bias of the 1×1 convolution.
4. The action segmentation method based on human skeleton and action boundary regression optimization according to claim 3 is characterized in that: The formula for shared action features is: X'=Conv 1d (DRL *10 (X)) (3) In the formula, Conv 1d represents one-dimensional convolution, DRL represents dilated residual layer; X represents human skeleton features.
5. The action segmentation method based on human skeleton and action boundary regression optimization according to claim 1, characterized in that: The action segmentation branch network includes: the first Conv1d, three TCN modules cascaded, the TCN module is cascaded by the LFE module and Conv1d, and the first Conv1d, the first and second TCN modules are connected with Softmax.
6. The action segmentation method based on human skeleton and action boundary regression optimization according to claim 1, characterized in that: The boundary regression branch network replaces the first Conv1d of the action segmentation branch network with conv_bound and replaces Softmax with Sigmoid.
7. The action segmentation method based on human skeleton and action boundary regression optimization according to claim 1, characterized in that: Step 5 specifically includes: Step 51: Construct the square error loss function L s , the formula is; Where T is the number of frames in the video, N is the number of action categories, and y t,c is the probability of the predicted category c of the action segmentation branch network at time t, y t,b,c is the probability of predicting category c after optimizing the action boundary at time t; Step 52: Use the boundary regression loss function to optimize the action segmentation. The formula is: L as =L ce +L GS-TMSE +λ s L s (5) In the formula, λ s YesL s The weight, L ce is the cross entropy loss, L GS-TMSE is Gaussian similarity weighted TMSE loss; Step 53: Calculate the loss L of the action segmentation branch asb , the formula is: Where N as The number of predictions for the action segmentation branch; Step 54: Calculate the loss L of the boundary regression branch brb , the formula is: Where N br is the number of predictions of the boundary regression branch, L bl is the binary logistic regression loss; Step 55: Add the losses of the action segmentation branch and the boundary regression branch to obtain the total loss of the action segment refinement network, which is: L = L asb +λL brb (13) Where λ is the weight of the boundary regression branch loss.
8. The action segmentation method based on human skeleton and action boundary regression optimization according to claim 1, characterized in that: The ST-GCN network is used to extract the human skeleton features of each frame.
9. The method for action segmentation based on human skeleton and action boundary regression optimization according to claim 1, characterized in that: Action segmentation is achieved by retaining the moving human body, moving it to the center of the video, and filling the surrounding area of the video with black pixels.
10. An action segmentation method and system based on human skeleton and action boundary regression optimization, characterized in that: include: a memory for storing instructions executable by a processor; A processor, configured to execute instructions to implement the action segmentation method based on human skeleton and action boundary regression optimization as described in any one of claims 1 to 9.