Action segmentation network optimization method based on feature similarity

Abstract

The invention discloses an action segmentation network optimization method based on feature similarity. The method comprises the following steps: firstly, acquiring a video of a data set, carrying out framing processing on the original video to obtain a time sequence picture, and then obtaining a corresponding label; then, a universal feature extraction network is used for carrying out feature extraction on the data set to obtain corresponding features, and then representative features are calculated according to the input features; and then feature similarity calculation is carried out according to the representative features, then filtering processing is carried out, the filtered feature similarity is input into a loss function, and back propagation is carried out to update parameters. According to the method, the representative features of the actions in the video are obtained, the feature similarity is combined, network training is assisted, a better action classification result is achieved, over-segmentation errors are reduced, and the method is wide in application range, high in model accuracy and high in robustness.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and relates to a label smoothing strategy based on feature similarity, and specifically to an action segmentation network optimization method based on feature similarity. Background Art

[0002] In recent years, Temporal Convolutional Networks (TCN) have achieved remarkable performance in the field of video action segmentation. However, most TCN networks tend to be overconfident when training samples, which greatly affects their generalization ability for test samples. To solve this problem, many researchers have borrowed the label smoothing method from the image field to improve the generalization ability of TCN networks.

[0003] However, directly using label smoothing is not very compatible with the video action segmentation task. Traditional label smoothing uses soft labels by taking the average between hard labels and uniform distribution on labels. Although this soft label can provide strong regularization and prevent the learned model from being overconfident, it treats non-target categories equally by assigning fixed equal probabilities. There are many similar label categories in traditional video action segmentation tasks. If non-target categories are treated equally, this will often have a counterproductive effect on the video action segmentation task. For example, if label smoothing is used, the action at the boundary of the "cut tomatoes" action will be more similar to "put tomatoes" rather than "cut tomatoes". Therefore, for the video action segmentation task, the assigned probability of non-target categories should highly consider their similarity to the given action. Treating each non-target category equally may weaken the ability of label smoothing and limit the performance of the model. Summary of the invention

[0004] The purpose of the present invention is to provide a universal label smoothing method in the field of action segmentation in view of the shortcomings of the prior art. The present invention provides an action segmentation network optimization method based on feature similarity.

[0005] The present invention first needs to obtain the video of the data set, divide the original video into frames, obtain the time-series picture, and then obtain the corresponding label. Then, a universal feature extraction network is used to extract the features of the data set to obtain the corresponding features, and then the representative features are calculated based on the input features. Then, the feature similarity is calculated based on the representative features, and then filtering is performed. After that, the filtered feature similarity is input into the loss function, and the parameters are updated by back propagation.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] Step 1: Get the dataset video, divide the video into frames, and obtain images and labels.

[0008] Step 2: Use the existing feature extraction network to extract features and input them into the action segmentation network.

[0009] Step 3: Use the extracted features to calculate representative features.

[0010] Step 4: Calculate feature similarity based on representative features.

[0011] Step 5: Process the obtained feature similarity sequence using the Savitzky-Golay filter.

[0012] Step 6: Generate a supervision matrix based on the processed similarity sequence.

[0013] Step 7: Calculate the joint loss and backpropagate updated parameters.

[0014] Furthermore, the specific method of step 3 is as follows:

[0015] Assume that in the current training stage, the long video input to the action segmentation network is V. The dimension of the video features extracted by the feature extraction network from the long video V is T×N, where T is the video length and N is the extracted feature dimension. Assuming that the long video V consists of K actions, the long video is divided into video segments T 1 ,T 2 ,…,T k ,…,T K , k = 1, 2, ..., K. Each video segment has a corresponding feature range, and the representative features of each video segment are calculated.

[0016] The K-Means algorithm is used to calculate the representative features of each video segment and the video segment T k The features within the corresponding feature range are input into the K-Means algorithm, and the K-Means algorithm cluster category is set to 2. After the K-Means algorithm clustering converges, the cluster center with the most clusters in the two cluster categories is taken to obtain T k The corresponding representative feature T γk Traverse all video segments and obtain the representative feature sequence T corresponding to video V γ1 ,T γ2 ,…,T γk ,…,T γK .

[0017] Furthermore, the specific method of step 4 is as follows:

[0018] The representative feature sequence T obtained in step 3 γ1 ,Tγ2 ,…,T γk ,…,T γK The representative features in the video are used to calculate the feature similarity with other features in the video segment to which they belong. The calculation formula for the similarity is:

[0019]

[0020] Where Z represents the total number of video frames in a single video segment, T k,z represents the zth feature of the kth video segment, Represents the feature similarity of the zth feature in the kth video segment in this segment. Thus, all similarity sequences in the kth video segment are obtained.

[0021] Traverse all video segments to obtain the similarity sequences of all video segments of video V, combine the similarity sequences of all video segments together to obtain the similarity sequences of all frames in video V

[0022] Furthermore, the specific method of step 5 is as follows:

[0023] The similarity sequence of all frames in the video V obtained in step 4 is filtered using the Savitzky-Golay filter to make it smoother. The filter window is set to 5 and the filter polynomial fitting order is set to 4. The processed sequence is obtained:

[0024]

[0025] Furthermore, the specific method of step 6 is as follows:

[0026] According to the similarity sequence processed in step 5, generate an L×T all-zero matrix M, where L is the classification category, and then assign it:

[0027]

[0028] In fact, r represents the row number of the all-zero matrix M, c represents the column number of the all-zero matrix M, and L i Represents the subscript corresponding to the true category of the i-th frame (for example, the current video V has L categories, and these L categories are set as a one-dimensional matrix according to the label order, then the L-th i is the position of the true category of the i-th frame in this 1xL matrix).

[0029] Furthermore, the specific method of step 7 is as follows:

[0030] In order to better supervise the neural network, the traditional hard label loss (i.e., the loss is calculated using the original label) is used together with the constructed supervision matrix to construct the loss to supervise the action segmentation network.total Divided into two parts hard and L soft , and use a hyperparameter a to coordinate the two losses, where L hard is the traditional cross entropy loss, and L soft It is a soft loss constructed by the supervision matrix.

[0031]

[0032] where p(l|x i ) is the probability of predicting class l, y i represents the true label of the input sample, x i represents the input sample, q(l|x i ) represents the sample x i Is it the lth category (if yes, take 1, otherwise take 0).

[0033]

[0034] Where i represents the i-th column of the supervision matrix generated in step 6, T represents the total number of frames, and x i represents the input sample, p(l|x i ) represents the similarity of the predicted class l in the similarity matrix generated by the i-th column, q(l|x i ) represents the sample x i Is it the lth category (if yes, take 1, otherwise take 0).

[0035] Total loss:

[0036] L total =aL hard +(1-a)L soft ;

[0037] Where a is the tuned hyperparameter.

[0038] Back-propagation is performed based on the calculated joint loss to update the action segmentation network parameters and achieve action segmentation network optimization.

[0039] The beneficial effects of the present invention are as follows:

[0040] The present invention does not require the design of a new neural network, and can be directly used on the existing action segmentation network (such as MS-TCN, C2F-TCN). By obtaining the representative features of the action in the video and combining the feature similarity, the network training is assisted to achieve better action classification results, reduce over-segmentation errors, and has a wide range of applications, high model accuracy, and strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Flowchart of execution of the method of the present invention. DETAILED DESCRIPTION

[0042] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0043] like Figure 1 As shown, a method for optimizing an action segmentation network based on feature similarity includes the following steps:

[0044] Step 1: Get a dataset video (the video needs to contain multiple actions of varying lengths). This embodiment uses the 50Salads and Gtea datasets to perform frame processing on the video to obtain images and labels.

[0045] Step 2: Use the existing feature extraction network to extract features and input them into the action segmentation network. In this embodiment, the feature extraction network uses I3D, and the action segmentation network uses MS-TCN.

[0046] Step 3: Use the extracted features to calculate representative features.

[0047] Assume that in the current training stage, the long video input to the action segmentation network is V (the training batch is 1), then the video feature dimension extracted by the feature extraction network of the long video V is T×N (T is the video length, N is the extracted feature dimension). Assuming that the long video V consists of K actions, this long video is divided into video segments T 1 ,T 2 ,…,T k ,…,T K , k = 1, 2, ..., K. Each video segment has a corresponding feature range, and we calculate the representative features of each video segment.

[0048] The K-Means algorithm is used to calculate the representative features of each video segment and the video segment T k The features within the corresponding feature range are input into the K-Means algorithm, and the K-Means algorithm cluster category is set to 2. After the K-Means algorithm clustering converges, the cluster center with the most clusters in the two cluster categories is taken to obtain T k The corresponding representative feature T γk Traverse all video segments and obtain the representative feature sequence T corresponding to video V γ1 ,T γ2 ,…,T γk ,…,T γK .

[0049] Step 4: Calculate feature similarity based on representative features.

[0050] The representative feature sequence T obtained in step 3 γ1 ,Tγ2 ,…,T γk ,…,T γK The representative features in the video segment to which they belong (such as T γ1 Corresponding to T 1 The feature similarity calculation is performed on other features in the segment, and the calculation formula corresponding to the similarity is:

[0051]

[0052] Where Z represents the total number of video frames in a single video segment, T k,z represents the zth feature of the kth video segment, Represents the feature similarity of the zth feature in the kth video segment in this segment. Thus, all similarity sequences in the kth video segment are obtained.

[0053] Traverse all video segments to obtain the similarity sequences of all video segments of video V, combine the similarity sequences of all video segments together to obtain the similarity sequences of all frames in video V

[0054] Step 5: The similarity sequence of all frames in the video V obtained in step 4 is not smooth, and the similarity between adjacent features will have large jumps, which is not conducive to the final training. Therefore, the similarity sequence of all frames in the video V obtained in step 4 is filtered using the Savitzky-Golay filter to make it smoother. The filter window is set to 5, and the filter polynomial fitting order is set to 4. The processed sequence is obtained:

[0055]

[0056] Step 6: Generate a supervision matrix based on the similarity sequence processed in step 5.

[0057] According to the similarity sequence processed in step 5, generate an L×T all-zero matrix M, where L is the classification category, and then assign it:

[0058]

[0059] In fact, r represents the row number of the all-zero matrix M, c represents the column number of the all-zero matrix M, and L i Represents the subscript corresponding to the true category of the i-th frame (for example, the current video V has L categories, and these L categories are set as a one-dimensional matrix according to the label order, then the L-th i is the position of the true category of the i-th frame in this 1xL matrix).

[0060] Step 7: Calculate the joint loss and back-propagate to update the action segmentation network (MS-TCN) parameters.

[0061] In order to better supervise the neural network, we not only use the supervision matrix to construct the loss, which is too biased, but we also use the traditional hard label loss (that is, the loss is calculated using the original label) together with the constructed supervision matrix to construct the loss to supervise the action segmentation network (MS-TCN). Our total loss L total Divided into two parts hard and L soft , and use a hyperparameter a to coordinate the two losses, where L hard is the traditional cross entropy loss, and L soft It is the soft loss we constructed from the supervision matrix. The purpose of doing this is that due to the random parameter initialization and the lack of hard labels at the beginning, our model is difficult to converge, so we use this joint loss to supervise the neural network.

[0062]

[0063] where p(l|x i ) is the probability of predicting class l, y i represents the true label of the input sample, x i represents the input sample, q(l|x i ) represents the sample x i Is it the lth category (if yes, take 1, otherwise take 0).

[0064]

[0065] Where i represents the i-th column of the supervision matrix generated in step 6, T represents the total number of frames, and x i represents the input sample, p(l|x i ) represents the similarity of the predicted class l in the similarity matrix generated by the i-th column, q(l|x i ) represents the sample x i Is it the lth category (if yes, take 1, otherwise take 0).

[0066] Our total loss:

[0067] L total =aL hard +(1-a)L soft ;

[0068] Where a is the tuned hyperparameter.

[0069] Back-propagation is performed based on the calculated joint loss to update the parameters of the action segmentation network (MS-TCN) to achieve action segmentation network optimization.

[0070] Step 8: Evaluate the trained model using indicators such as Acc, F1, and Edit;

[0071] Indicator explanation:

[0072] Acc accuracy refers to the proportion of correctly predicted labels for all frames. The calculation method is:

[0073]

[0074] where p i and g i They represent the predicted category and label category of frame i respectively, and N is the total number of frames in the video.

[0075] F1: F1@{10, 25, 50} represents the comparison threshold. If the IOU between the predicted segmented segment and the labeled segment is greater than the threshold, the segmentation is considered correct.

[0076]

[0077] Edit: By solving the left-Wen distance of two action sequences, let the true value segmentation sequence be G = G 1 ,G 2 ,G 3 ,...,G M , the predicted segmentation sequence is P = P 1 ,P 2 ,P 3 ,...,P M , we calculate the split edit distance as:

[0078]

[0079] The experimental results are shown in the following table:

[0080]

[0081] Table 1

[0082]

[0083] Table 2

[0084] Table 1 shows the comparison between our method and some commonly used label smoothing strategies on the 50salads dataset, Improvement shows the comparison between our method and the OLS method, and Table 2 shows the comparison between our method and the Gtea dataset. Our results are bolded. Compared with some commonly used methods, our method has some improvements, especially on short datasets such as Gtea, where our F1 score has improved significantly.

[0085] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0086] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A method for optimizing an action segmentation network based on feature similarity, characterized in that: The steps include: Step 1: Get the dataset video, divide the video into frames, and get images and labels; Step 2: Use the existing feature extraction network to extract features and input them into the action segmentation network; Step 3: Calculate representative features using the extracted features; Step 4: Calculate feature similarity based on representative features; Step 5: Process the obtained feature similarity sequence using the Savitzky-Golay filter; Step 6: Generate a supervision matrix based on the processed similarity sequence; Step 7: Calculate the joint loss and backpropagate updated parameters.

2. The method for optimizing an action segmentation network based on feature similarity according to claim 1, characterized in that: Step 3: Suppose that in the current training stage, the long video input to the action segmentation network is V, then the video feature dimension extracted by the feature extraction network of the long video V is T×N, T is the video length, N is the extracted feature dimension, assuming that the long video V consists of K actions, then this long video is divided into video segments T1, T2, …, T k ,…,T K , k = 1, 2, ..., K; each video segment has a corresponding feature range, and the representative features of each video segment are calculated; The K-Means algorithm is used to calculate the representative features of each video segment and the video segment T k The features within the corresponding feature range are input into the K-Means algorithm, and the K-Means algorithm cluster category is set to 2. After the K-Means algorithm clustering converges, the cluster center with the most clusters in the two cluster categories is taken to obtain T k The corresponding representative feature T γk ; Traverse all video segments and obtain the representative feature sequence T corresponding to video V γ1 ,T γ2 ,…,T γk ,…,T γK .

3. The method for optimizing an action segmentation network based on feature similarity according to claim 2, characterized in that: Step 4: The representative feature sequence T obtained in step 3 γ1 ,T γ2 ,…,T γk ,…,T γK The representative features in the video are used to calculate the feature similarity with other features in the video segment to which they belong. The calculation formula for the similarity is: Where Z represents the total number of video frames in a single video segment, T k,z represents the zth feature of the kth video segment, Represents the feature similarity of the zth feature in the kth video segment in this segment; thus, all similarity sequences in the kth video segment are obtained Traverse all video segments to obtain the similarity sequences of all video segments of video V, combine the similarity sequences of all video segments together to obtain the similarity sequences of all frames in video V 4. The method for optimizing an action segmentation network based on feature similarity according to claim 3, characterized in that: Step 5: The similarity sequence of all frames in the video V obtained in step 4 is filtered using the Savitzky-Golay filter to make it smoother. The filter window is set to 5 and the filter polynomial fitting order is set to 4. The processed sequence is obtained:

5. The method for optimizing an action segmentation network based on feature similarity according to claim 4, characterized in that: Step 6: According to the similarity sequence processed in step 5, generate an L×T all-zero matrix M, where L is the classification category, and then assign it: In fact, r represents the row number of the all-zero matrix M, c represents the column number of the all-zero matrix M, and L i Represents the subscript corresponding to the true category of the i-th frame.

6. The method for optimizing an action segmentation network based on feature similarity according to claim 5, characterized in that: Step 7: In order to better supervise the neural network, the traditional hard label loss is used together with the constructed supervision matrix to construct the loss to supervise the action segmentation network; the total loss L total Divided into two parts hard and L soft , and use a hyperparameter a to coordinate the two losses, where L hard is the traditional cross entropy loss, and L soft It is the soft loss constructed by the supervision matrix; where p(l|x i ) is the probability of predicting class l, y i represents the true label of the input sample, x i represents the input sample, q(l|x i ) represents the sample x i Whether it is the lth category (if yes, it takes 1, otherwise it takes 0); Where i represents the i-th column of the supervision matrix generated in step 6, T represents the total number of frames, and x i represents the input sample, p(l|x i ) represents the similarity of the predicted class l in the similarity matrix generated by the i-th column, q(l|x i ) represents the sample x i Whether it is the first category; Total loss: L total =aL hard +(1-a)L soft ; Where a is the adjusted hyperparameter; Back-propagation is performed based on the calculated joint loss to update the action segmentation network parameters and achieve action segmentation network optimization.

Images