An action segmentation network optimization method based on feature similarity

Abstract

The application discloses a motion segmentation network optimization method based on feature similarity. First, the video of the data set is obtained, the original video is subjected to frame processing to obtain time sequence pictures, and corresponding labels are obtained. Then, a general feature extraction network is used to extract features of the data set to obtain corresponding features, and then representative features are calculated according to the input features. Then, feature similarity calculation is performed according to the representative features, then filtering processing is performed, and then the filtered feature similarity is input into a loss function to update parameters through back propagation. The application obtains the representative features of the motion in the video, combines the feature similarity, and assists the training of the network to achieve better motion classification results, reduces over-segmentation errors, and has wide application range, high model accuracy and strong robustness.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to a label smoothing strategy based on feature similarity, specifically a method for optimizing action segmentation networks based on feature similarity. Background Technology

[0002] In recent years, Temporal Convolutional Networks (TCNs) have achieved remarkable performance in video action segmentation. However, most TCNs tend to overconfide in training samples, which significantly affects their generalization ability to test samples. To address this issue, many researchers have borrowed label smoothing techniques from the image domain to improve the generalization ability of TCNs.

[0003] However, directly using label smoothing is not well-suited for video action segmentation tasks. Traditional label smoothing utilizes soft labels, averaging between hard labels and a uniform distribution of labels. While this soft labeling provides strong regularization, preventing the learned model from becoming overconfident, it treats non-target classes equally by assigning fixed, uniform probabilities. In traditional video action segmentation tasks, there are many similar label classes. Treating non-target classes equally often has a negative effect on the task. For example, with label smoothing, the action boundary of "cutting tomatoes" might resemble "placing tomatoes" more than "cutting tomatoes." Therefore, for video action segmentation, the probability assignment for non-target classes should highly consider their similarity to a given action. Treating every non-target class equally may weaken the effectiveness of label smoothing and limit model performance. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a label smoothing method applicable to the field of action segmentation. This invention provides an action segmentation network optimization method based on feature similarity.

[0005] This invention first requires acquiring the video dataset, segmenting the original video into frames to obtain time-series images, and then obtaining the corresponding labels. Next, a general feature extraction network is used to extract features from the dataset, obtaining the corresponding features, and then representative features are calculated based on the input features. Feature similarity is then calculated based on the representative features, followed by filtering. Finally, the filtered feature similarity is input into a loss function, and parameters are updated via backpropagation.

[0006] The technical solution adopted by this invention to solve its technical problem is as follows:

[0007] Step 1: Obtain the dataset video, perform frame segmentation on the video, and obtain images and labels.

[0008] Step 2: Extract features using an existing feature extraction network and input them into the action segmentation network.

[0009] Step 3: Calculate representative features using the extracted features.

[0010] Step 4: Calculate feature similarity based on representative features.

[0011] Step 5: Process the obtained feature similarity sequence using the Savitzky-Golay filter.

[0012] Step 6: Generate a supervision matrix based on the processed similarity sequence.

[0013] Step 7: Calculate the joint loss and backpropagate to update the parameters.

[0014] Furthermore, the specific method for step 3 is as follows:

[0015] Suppose that in the current training phase, the long video input into the action segmentation network is V. Then, the video feature dimension extracted from the long video V by the feature extraction network is T×N, where T is the video length and N is the extracted feature dimension. Assuming that the long video V consists of K actions, this long video is divided into video segments T1, T2, ..., T. k ,…,T K k = 1, 2, ..., K. Each video segment has a corresponding feature range, and the representative features of each video segment are calculated.

[0016] The K-Means algorithm is used to calculate representative features for each video segment, and video segment T is then... k The features within the corresponding feature range are input into the K-Means algorithm. The K-Means algorithm is set to have two cluster categories. After the K-Means algorithm achieves cluster convergence, the cluster center of the largest cluster among the two cluster categories is selected to obtain T. k Corresponding representative feature T γk Traverse all video segments to obtain the representative feature sequence T corresponding to video V. γ1 ,T γ2 ,…,T γk ,…,T γK .

[0017] Furthermore, the specific method for step 4 is as follows:

[0018] The representative feature sequence T obtained in step 3 γ1 ,T γ2 ,…,T γk,…,T γK The representative features in the video segment are compared with other features within the same video segment to calculate their feature similarity. The formula for calculating the similarity is as follows:

[0019]

[0020] Where Z represents the total number of video frames in a single video segment, and T k,z This represents the z-th feature of the k-th video segment. This represents the feature similarity of the z-th feature in the k-th video segment within this segment. This yields all similarity sequences within the k-th video segment.

[0021] Iterate through all video segments to obtain the similarity sequence of all video segments in video V. Combine the similarity sequences of all video segments to obtain the similarity sequence of all frames in video V.

[0022] Furthermore, the specific method for step 5 is as follows:

[0023] The similarity sequence of all frames in video V obtained in step 4 is smoothed using a Savitzky-Golay filter, with a filter window set to 5 and a filter polynomial fitting order set to 4. The resulting processed sequence is:

[0024]

[0025] Furthermore, the specific method for step 6 is as follows:

[0026] Based on the similarity sequence processed in step 5, generate an L×T all-zero matrix M, where L represents the classification category, and then assign values ​​to it:

[0027]

[0028] In this context, r represents the row number of the all-zero matrix M, c represents the column number of the all-zero matrix M, and L... i The index representing the true category of the i-th frame (e.g., if the current video V has L classes, and these L classes are set as a one-dimensional matrix according to the label order, then the L-th frame...). i (This represents the position of the true class of the i-th frame within this 1xL matrix).

[0029] Furthermore, the specific method for step 7 is as follows:

[0030] To better supervise the neural network, a traditional hard-label loss (i.e., loss calculated using the original labels) is used along with a constructed supervision matrix to build a loss to supervise the action segmentation network. The total loss L... total Divided into two parts Lhard and L soft And a hyperparameter 'a' is used to reconcile the two losses, where L hard It is the traditional cross-entropy loss, while L soft It is a soft loss constructed from the supervision matrix.

[0031]

[0032] Where p(l|x) i y represents the probability of predicting class l. i x represents the true label of the input sample. i Representing the input sample, q(l|x) i ) represents sample x i Is it of type l? (If yes, set to 1; otherwise, set to 0).

[0033]

[0034] Where i represents the i-th column of the supervision matrix generated in step 6, T represents the total number of frames, and x i p(l|x) represents the input sample. i ) represents the similarity of predicted class l in the similarity matrix generated in the i-th column, q(l|x i ) represents sample x i Is it of type l? (If yes, set to 1; otherwise, set to 0).

[0035] Total loss:

[0036] L total =aL hard +(1-a)L soft ;

[0037] Where 'a' is the adjusted hyperparameter.

[0038] The parameters of the action segmentation network are updated by backpropagation based on the calculated joint loss, thereby optimizing the action segmentation network.

[0039] The beneficial effects of this invention are as follows:

[0040] This invention does not require the design of a new neural network. It can be directly used on existing action segmentation networks (such as MS-TCN and C2F-TCN). By obtaining representative features of actions in the video and combining feature similarity, the network training is assisted to achieve better action classification results, reduce oversegmentation errors, and has a wide range of applications, high model accuracy, and strong robustness. Attached Figure Description

[0041] Figure 1 Flowchart of the method of this invention. Detailed Implementation

[0042] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] like Figure 1 As shown, an action segmentation network optimization method based on feature similarity includes the following steps:

[0044] Step 1: Obtain the dataset video (the video needs to contain multiple actions of varying lengths). In this example, the 50Salads and Gtea datasets are used. The video is processed by frame segmentation to obtain images and labels.

[0045] Step 2: Extract features using an existing feature extraction network and input them into the action segmentation network. In this embodiment, the feature extraction network used is I3D, and the action segmentation network used is MS-TCN.

[0046] Step 3: Calculate representative features using the extracted features.

[0047] Suppose that in the current training phase, the long video input into the action segmentation network is V (training batch is 1). Then, the video feature dimension extracted from the long video V by the feature extraction network is T×N (T is the video length, and N is the extracted feature dimension). Assuming that the long video V consists of K actions, this long video is divided into video segments T1, T2, ..., T. k ,…,T K k = 1, 2, ..., K. Each video segment has a corresponding feature range, and we calculate the representative features of each video segment.

[0048] The K-Means algorithm is used to calculate representative features for each video segment, and video segment T is then... k The features within the corresponding feature range are input into the K-Means algorithm. The K-Means algorithm is set to have two cluster categories. After the K-Means algorithm achieves cluster convergence, the cluster center of the largest cluster among the two cluster categories is selected to obtain T. k Corresponding representative feature T γk Traverse all video segments to obtain the representative feature sequence T corresponding to video V. γ1 ,T γ2 ,…,T γk ,…,T γK .

[0049] Step 4: Calculate feature similarity based on representative features.

[0050] The representative feature sequence T obtained in step 3 γ1 ,T γ2 ,…,T γk ,…,T γKThe representative features in it and the video segment to which it belongs (such as T) γ1 For other features within segment T1, feature similarity is calculated using the following formula:

[0051]

[0052] Where Z represents the total number of video frames in a single video segment, and T k,z This represents the z-th feature of the k-th video segment. This represents the feature similarity of the z-th feature in the k-th video segment within this segment. This yields all similarity sequences within the k-th video segment.

[0053] Iterate through all video segments to obtain the similarity sequence of all video segments in video V. Combine the similarity sequences of all video segments to obtain the similarity sequence of all frames in video V.

[0054] Step 5: The similarity sequence of all frames in video V obtained in Step 4 is not smooth, and the similarity between adjacent features fluctuates significantly, which is detrimental to the final training. Therefore, a Savitzky-Golay filter is applied to the similarity sequence of all frames in video V obtained in Step 4 to make it smoother. The filter window is set to 5, and the order of the filter polynomial fitting is set to 4. The processed sequence is obtained as follows:

[0055]

[0056] Step 6: Generate a supervision matrix based on the similarity sequence processed in Step 5.

[0057] Based on the similarity sequence processed in step 5, generate an L×T all-zero matrix M, where L represents the classification category, and then assign values ​​to it:

[0058]

[0059] In this context, r represents the row number of the all-zero matrix M, c represents the column number of the all-zero matrix M, and L... i The index representing the true category of the i-th frame (e.g., if the current video V has L classes, and these L classes are set as a one-dimensional matrix according to the label order, then the L-th frame...). i (This represents the position of the true class of the i-th frame within this 1xL matrix).

[0060] Step 7: Calculate the joint loss and backpropagate to update the parameters of the Action Segmentation Network (MS-TCN).

[0061] To better supervise the neural network, we don't just use the supervision matrix to construct the loss, as this is too biased. Instead, we also use the traditional hard-label loss (i.e., calculating the loss using the original labels) along with the constructed supervision matrix to build the loss for supervising the action segmentation network (MS-TCN). Our total loss L total Divided into two parts L hard and L soft And a hyperparameter 'a' is used to reconcile the two losses, where L hard It is the traditional cross-entropy loss, while L soft This is the soft loss we construct from the supervision matrix. The purpose of this is to supervise the neural network because our model has difficulty converging due to the initial random parameter initialization and lack of hard labels.

[0062]

[0063] Where p(l|x) i y represents the probability of predicting class l. i x represents the true label of the input sample. i Representing the input sample, q(l|x) i ) represents sample x i Is it of type l? (If yes, set to 1; otherwise, set to 0).

[0064]

[0065] Where i represents the i-th column of the supervision matrix generated in step 6, T represents the total number of frames, and x i p(l|x) represents the input sample. i ) represents the similarity of predicted class l in the similarity matrix generated in the i-th column, q(l|x i ) represents sample x i Is it of type l? (If yes, set to 1; otherwise, set to 0).

[0066] Our total loss:

[0067] L total =aL hard +(1-a)L soft ;

[0068] Where 'a' is the adjusted hyperparameter.

[0069] The parameters of the action segmentation network (MS-TCN) are updated by backpropagation based on the calculated joint loss, thereby optimizing the action segmentation network.

[0070] Step 8: Evaluate the trained model using metrics such as Acc accuracy, F1 score, and Edit score;

[0071] Indicator Explanation:

[0072] Accuracy refers to the proportion of correctly predicted labels across all frames. The calculation method is as follows:

[0073]

[0074] Where p i and g i Let i represent the predicted class and the label class of frame i, respectively, and N be the total number of frames in the video.

[0075] F1: F1@{10, 25, 50} represents the comparison threshold. If the IOU between the predicted segment and the labeled segment is greater than the threshold, the segmentation is considered correct.

[0076]

[0077] Edit: By solving the left Venn distance between two action sequences, let the truth value segmentation sequence be G = G1, G2, G3, ..., G M The predicted segmentation sequence is P = P1, P2, P3, ..., P M We calculate the segment edit distance as follows:

[0078]

[0079] The experimental results are shown in the table below:

[0080]

[0081] Table 1

[0082]

[0083] Table 2

[0084] Table 1 shows the comparison between our method and some commonly used label smoothing strategies on the 50salads dataset. Improvement represents the comparison with the OLS method. Table 2 shows the comparison of our method on the Gtea dataset. Our results are bolded. Compared with some commonly used methods, our method has some improvements, especially on short datasets like Gtea, where our F1 score is significantly improved.

[0085] The above description, in conjunction with specific / preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. Those skilled in the art can make various substitutions or modifications to these described embodiments without departing from the inventive concept, and all such substitutions or modifications should be considered within the scope of protection of the present invention.

[0086] The parts of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A method for optimizing action segmentation networks based on feature similarity, characterized in that, The steps include the following: Step 1: Obtain the dataset video, perform frame segmentation on the video, and obtain images and labels; Step 2: Extract features using an existing feature extraction network and input them into the action segmentation network; Step 3: Calculate representative features using the extracted features; Step 4: Calculate feature similarity based on representative features; Step 5: Process the obtained feature similarity sequence using the Savitzky-Golay filter; Step 6: Generate a supervision matrix based on the processed similarity sequence; Step 7: Calculate the joint loss and backpropagate to update the parameters; The specific method for step 3 is as follows: Let V be the long video input to the action segmentation network during the current training phase. Then, the video feature dimension extracted from the long video V by the feature extraction network is T×N, where T is the video length and N is the extracted feature dimension. Assuming that the long video V consists of K actions, this long video is divided into video segments T1, T2, ..., T. k ,…,T K k=1,2,...,K; each video segment has a corresponding feature range, and the representative features of each video segment are calculated; The K-Means algorithm is used to calculate representative features for each video segment, and video segment T is then... k The features within the corresponding feature range are input into the K-Means algorithm. The K-Means algorithm is set to have two clustering categories. After the K-Means algorithm achieves cluster convergence, the cluster center of the category with the most clusters among the two clustering categories is taken, resulting in T. k Corresponding representative feature T γk ; Traverse all video segments to obtain the representative feature sequence T corresponding to video V. γ1 ,T γ2 ,…,T γk ,…,T γK ; Step 4 is explained in detail below: The representative feature sequence T obtained in step 3 γ1 ,T γ2 ,…,T γk ,…,T γK The representative features in the video segment are compared with other features within the same video segment to calculate their feature similarity. The formula for calculating the similarity is as follows: ； in, This represents the total number of video frames in a single video segment. This represents the z-th feature of the k-th video segment. This represents the feature similarity of the z-th feature in the k-th video segment within this segment; thus, all similarity sequences within the k-th video segment are obtained. , ,…, ,…, ; Iterate through all video segments to obtain the similarity sequence of all video segments in video V. Combine the similarity sequences of all video segments to obtain the similarity sequence of all frames in video V. , ,…, ,…, ,…, ,…, ,…, ; Step 6 is explained in detail below: Based on the similarity sequence processed in step 5, generate an L×T all-zero matrix M, where L represents the classification category, and then assign values ​​to it: ； Where r represents the row number of the all-zero matrix M, c represents the column number of the all-zero matrix M, and L i The index representing the true category of the i-th frame. This represents the feature similarity of the z-th feature in the k-th smoothed video segment within this segment; Step 7 is explained in detail below: To better supervise neural networks, a loss is constructed using traditional hard-label loss and a built-in supervision matrix to supervise the action segmentation network; the total loss... Divided into two parts and And using a hyperparameter To coordinate the two losses, among which It is the traditional cross-entropy loss, while It is a soft loss constructed from the supervision matrix; ； in For prediction The probability, Represents the true label of the input sample. Represents the input sample. Representative sample Is it the first? If the class is correct, set the value to 1; otherwise, set the value to 0. ； in The first step represents the supervision matrix generated in step 6. The column, T represents the total number of frames. Represents the input sample. Representing the The predicted class in the similarity matrix generated by the column is similarity, Representative sample Is it the first? kind; Total loss: ； in For adjusting hyperparameters; The parameters of the action segmentation network are updated by backpropagation based on the calculated joint loss, thereby optimizing the action segmentation network.

2. The action segmentation network optimization method based on feature similarity according to claim 1, characterized in that, Step 5 is explained in detail below: The similarity sequence of all frames in video V obtained in step 4 is smoothed using a Savitzky-Golay filter, with the filter window set to 5 and the filter polynomial fitting order set to 4; the processed sequence is obtained as follows: { , …, ,…, ,…, ,…, ,…, 。

Images