Dual-branch network video segmentation method based on chi-square filtering

Abstract

The invention discloses a dual-branch network video segmentation method based on chi-square filtering. Firstly, an existing 3D feature extraction network is used for carrying out feature extraction on a data set, and then extracted features are reprocessed; the reprocessed features are input into the double-branch neural network, in order to combine the features of the two-branch network, a boundary branch pays attention to boundary recognition, a main branch pays attention to action subject recognition, and network parameters output in the double-branch stage are combined in a voting mode; and finally, performing back propagation to update parameters. Aiming at the overfitting problem caused by the fact that a neural network uses noise-containing features for training, the invention provides a feature preprocessing mode, and non-processed features are optimized by using chi-square filtering and a genetic algorithm. According to the method, the preprocessing part is combined with the double-branch neural network, so that the prediction accuracy is improved, and an excellent result is obtained on 50Slads and Gtea data sets.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, mainly relates to the problem of feature preprocessing and video action segmentation, and specifically relates to a double-branch network video segmentation method based on chi-square filtering. Background Art

[0002] Existing research on action video segmentation usually consists of many ways, including using diffusion models and using TCN networks. The most typical TCN is to complete this task by using two stages: first, extract spatial or temporal features using feature extraction networks such as 2D CNN, two-stream neural network or 3DCNN, and then use RNN or temporal convolutional network to classify the extracted features into actions. These models have better results than earlier methods on datasets with a small number of action classes and video clips. However, using feature extraction networks to extract spatial and spatiotemporal features as input to neural networks often relies too much on feature extraction networks. The features extracted by existing feature extraction networks often contain too much noise, which is often unfavorable for neural network training and will greatly affect the accuracy of the final prediction. Moreover, if a single network is used to classify the extracted features for video actions, these single networks are often too biased and will confuse the main part and the boundary part of the action, especially when identifying the same action in non-identical clips. We generally call this an over-segmentation error. If it is on a large dataset with different action categories, this will lead to more significant over-segmentation errors. Summary of the invention

[0003] The purpose of the present invention is to address the deficiencies of existing methods and provide a dual-branch network video segmentation method based on chi-square filtering. The present invention solves the problem of low prediction accuracy of existing temporal convolutional networks (TCNs) when the training data contains a lot of interference.

[0004] The present invention first obtains the time-series images and corresponding labels in the original data set; then uses the existing 3D convolutional network (I3D or C3D) and other feature extraction networks to extract features, so as to facilitate the training of the neural network and obtain more action information and action boundary information. Through the chi-square filtering preprocessing module, the features extracted by the feature extraction network are further optimized.

[0005] The optimized features are then input into our dual-branch network for prediction to obtain the final prediction result. In our dual-branch network, by sharing the network parameters of each branch at each stage, the over-segmentation error caused by excessive bias of a single network is reduced, and the accuracy of the task of video action segmentation is improved.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] Step 1: Use the existing 3D feature extraction network (such as I3D, C3D, etc.) to extract features from the dataset.

[0008] Step 2: Reprocess the features extracted from the dataset video.

[0009] Step 3: Input the reprocessed features into the two-branch neural network. In order to combine the characteristics of the two-branch network, the boundary branch focuses on identifying the boundary, and the trunk branch focuses on identifying the action subject. The network parameters output by the two-branch stage are combined in the form of voting.

[0010] Step 4: Back propagate to update parameters.

[0011] Step 5: Test the trained network model and evaluate the trained model.

[0012] Furthermore, the specific method of step 1 is as follows:

[0013] The existing 3D feature extraction network is used to extract features from the dataset. The size of the extracted feature matrix is ​​T×N, where T is the number of frames in a single video and N is the extracted dimension. The dataset is divided into a training set and a test set. Then, representative actions are determined based on the video segmentation task, and the labels of representative actions are divided into a significant label set k. c middle.

[0014] In one embodiment, in step 1, according to the characteristics of the data set, the action with the largest number of labels in the data set is determined as a representative action, and the labels of the representative actions are divided into a significant label set k. c among.

[0015] Furthermore, the specific method of step 2 is as follows:

[0016] For a video D with K actions train =(x i ,y i ), where x i Represents the features extracted from the i-th frame image in the video, y i Represents the corresponding label. For the entire video, the extracted features are divided into K video segments SEG (seg1, seg2, ..., seg j ,…,seg K ), and then the chi-square value is calculated for the features in each video segment. The specific calculation formula is as follows:

[0017]

[0018] in represents the value of the feature of the zth frame of the jth video segment, represents the average value of the features of all frames in the jth video segment, Represents the video segment seg j The chi-square value of the z-th frame feature, z=1…Z, where Z represents the number of frames in the video segment.

[0019] According to the above operation, the video segment seg is traversed. j The chi-square value of all frames within The extracted features are ranked according to the size of these chi-square values ​​to obtain a ranked feature sequence.

[0020] For the ranked feature sequence, the last 10% of the features in this video segment are selected and optimized using a crossover method similar to the genetic algorithm. Specifically, the positions of the last 10% of the features in the video segment in the original feature sequence are found, and then 2 frames are detected before and after, and the feature at the top of the sequence after the ranking is found in the 4 detected frames. This feature is then crossovered with the feature to be optimized using the genetic algorithm, and the new feature is used to replace the original feature to obtain the reprocessed feature of the video segment.

[0021] Finally, the reprocessed features of all video segments are combined to obtain the reprocessed feature sequence of the video:

[0022]

[0023] Furthermore, the specific method of step 3 is as follows:

[0024] The reprocessed feature sequence SEG of the video cross , respectively input into the boundary branch and trunk branch of the two-branch neural network.

[0025] The boundary branch of the two-branch neural network consists of four modules with the same structure. A single module first matches the dimension through a 1x1 convolution, then processes the feature sequence through four fixed convolution kernels, and finally matches the output dimension through an N×K convolution to obtain the output of a single module of the boundary branch.

[0026] The trunk branch uses the original structure of ASFormer, which also includes 4 modules, corresponding to the boundary branches.

[0027] The input of the subsequent stage modules of the boundary branch and the trunk branch is judged by the voting module Vnote_Compare. The voting module uses a voting-like form to combine the outputs of the modules of each stage of the two branches. Specifically, let the output of the current stage module of the boundary branch be x1, and the output of the current stage module of the trunk branch be x2. The final output of the current stage of the two-branch neural network is determined by the following formula:

[0028]

[0029] in Indicates that the label in the input x1 is k c The number of Represents the total number of inputs of x1. Indicates that the label in the input x2 is k c The number of Represents the total number of inputs to x2.

[0030] where k c It is the set of salient labels pre-defined in step 1.

[0031] The final output of the current stage of the two-branch neural network determined by the voting module is used as the input of the next stage module of the boundary branch and the trunk branch. The final output of the last stage of the boundary branch and the trunk branch is the final prediction matrix. After the prediction matrix is ​​normalized by SoftMax, the final prediction sequence can be obtained.

[0032] Furthermore, the specific method of step 4 is as follows:

[0033] The two-branch neural network is supervised by combining the standard cross-entropy loss function in multiple stages.

[0034]

[0035] Where K represents the number of actions in the video, S is the total number of two-branch neural network stages used, is the probability of predicting class l, and y i represents the true label of the input sample, x i represents the input sample, q(l|x i ) represents the sample x i Is it the lth category (if yes, take 1, otherwise take 0).

[0036] The beneficial effects of the present invention are as follows:

[0037] To address the overfitting problem caused by using noisy features for training neural networks, we proposed a feature preprocessing method that uses chi-square filtering and genetic algorithms to optimize unprocessed features. This method is suitable for tasks that use extracted features as input to neural networks when processing video segmentation tasks. To address the over-segmentation and boundary prediction problems caused by a single network, we designed a neural network based on boundary prediction and trunk prediction. The boundary branch network is mainly used to identify the boundaries between different actions in the video, and the trunk branch mainly predicts the category to which each video action belongs. A voting method based on ensemble learning is used between the two branches to combine the advantages and disadvantages of the two neural networks. Based on this, we designed a two-branch neural network. By combining the preprocessing part with the two-branch neural network, we improved the accuracy of the prediction and achieved excellent results on the 50Salads and Gtea datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 The present invention is an embodiment of the method execution flow chart.

[0039] Figure 2 A schematic diagram of the overall network structure adopted by an embodiment of the present invention.

[0040] Figure 3 This is a comparison of the effects of the present invention on the 50salads and Gtea datasets. DETAILED DESCRIPTION

[0041] The specific embodiments of the present invention will be further described below in conjunction with the accompanying drawings.

[0042] like Figure 1 As shown, a dual-branch network video segmentation method based on chi-square filtering includes the following steps:

[0043] Step 1: Use the existing 3D feature extraction network (I3D) to extract features from the dataset (50Salads, Gtea). The size of the extracted feature matrix is ​​(T×N) (T is the number of frames in a single video, and N is the extracted dimension). The dataset is divided into a training set and a test set, with a specific division ratio of 4:1. Then, representative actions are determined based on the video segmentation task, and the labels of representative actions are divided into a significant label set k c The purpose of doing this is to determine the representative action of this dataset based on the initial label ratio of the dataset, and use this as a basis to optimize the neural network. (For the 50Salads dataset, the label containing the add action accounts for the highest proportion in the dataset, so the labels of actions such as (add_oil, add_vinegar) are taken as the labels of representative actions, and Gtea is the label containing Stir as the label of representative action a).

[0044] In one embodiment, in step 1, according to the characteristics of the data set, the action with the largest number of labels in the data set is determined as a representative action, and the labels of the representative actions are divided into a significant label set k. c among.

[0045] Step 2: Reprocess the features extracted from the dataset video (corresponding to Figure 2 The chi-square filtering part) is used to obtain the reprocessed feature sequence of the video.

[0046] For a video D with K actions train =(x i ,y i ), where x i Represents the features extracted from the i-th frame image in the video, y i Represents the corresponding label. For the entire video, the extracted features are divided into K video segments SEG (seg1, seg2, ..., seg j ,…,seg K ), and then calculate the chi-square value for the features in each video segment. The specific calculation formula is as follows:

[0047]

[0048] in represents the value of the feature of the zth frame of the jth video segment, represents the average value of the features of all frames in the jth video segment, Represents the video segment seg j The chi-square value of the z-th frame feature, z=1…Z, where Z represents the number of frames in the video segment.

[0049] According to the above operation, the video segment seg is traversed. j The chi-square value of all frames within We rank the extracted features according to the size of these chi-square values ​​and obtain a ranked feature sequence (such as Figure 2 Rank_Cross section).

[0050] For this ranked feature sequence, the last 10% of features in this video segment are selected and optimized (such as seg j The original unprocessed feature sequence is After our ranking, we detected that the features to be optimized are and In our task, we hope that the optimized features still have the action information of the original action clip, so we use a crossover method similar to genetic algorithm. Specifically, we find the last 10% of the features in the video segment ( and ) in their respective positions in the original feature sequence, and then detect 2 frames before and after, and find the feature at the top of the sequence after the ranking in the 4 detected frames. Then, this feature is cross-operated with the feature to be optimized by the genetic algorithm, and the new feature is used to replace the original feature to obtain the reprocessed feature of the video segment. According to the above method, we only process 10% of the features, in this case, and part, then our processed feature sequence is This is the reprocessing feature of the video segment.

[0051] Finally, the reprocessed features of all video segments are combined to obtain the reprocessed feature sequence of the video:

[0052]

[0053] Step 3: Sequence the video's reprocessed features SEG cross The input is into the two-branch neural network, and the network parameters output by the two-branch stage are combined in the form of voting.

[0054] The reprocessed feature sequence SEG of the video cross , respectively input into the boundary branch and trunk branch of the two-branch neural network.

[0055] like Figure 2 As shown in the figure, the boundary branch of the dual-branch neural network consists of four modules with the same structure. A single module first matches the dimension through a 1x1 convolution, then processes the feature sequence through four fixed convolution kernels (we set it to 5), and finally matches the output dimension through a (N×K) convolution to obtain the output of a single module of the boundary branch.

[0056] The trunk branch uses the original structure of ASFormer, which also includes 4 modules, corresponding to the boundary branches.

[0057] The input of the subsequent stage modules of the boundary branch and the trunk branch is judged by the voting module (Vnote_Compare). The voting module (Vnote_Compare) uses a voting-like form to combine the outputs of the modules of each stage of the two branches. Specifically, let the output of the current stage module of the boundary branch be x1, and the output of the current stage module of the trunk branch be x2. The final output of the current stage of the dual-branch neural network is determined by the following formula:

[0058]

[0059] in Indicates that the label in the input x1 is k c The number of Represents the total number of inputs of x1. Indicates that the label in the input x2 is k c The number of Represents the total number of inputs to x2.

[0060] where k c is the set of salient labels (k c There are multiple).

[0061] The final output of the current stage of the two-branch neural network determined by the voting module (Vnote_Compare) is used as the input of the next stage module of the boundary branch and the trunk branch. The final output of the last stage of the boundary branch and the trunk branch is the final prediction matrix. After the prediction matrix is ​​normalized by SoftMax, the final prediction sequence can be obtained (for the input training set data, the training set segmentation result is obtained for calculating the loss).

[0062] Step 4: Back propagate to update parameters.

[0063] We not only use the loss function of the last layer to supervise the two-branch neural network, but also supervise the two-branch neural network by combining the standard cross entropy loss function of multiple stages.

[0064]

[0065] Where K represents the number of actions in the video, S is the total number of two-branch neural network stages used, and p(l|x i ) is the probability of predicting class l, y i represents the true label of the input sample, x i represents the input sample, q(l|x i ) represents the sample x i Is it the lth category (if yes, take 1, otherwise take 0).

[0066] Step 5: Test the trained network model and evaluate the trained model through indicators such as Acc accuracy, F1, and Edit.

[0067] The test set feature (T×N) dimension divided in step 1 is input into the final trained model in step 4, and finally the prediction matrix (T×K) dimension of the test set is obtained. After SoftMax normalization, the final prediction sequence Predict (T×1 dimension) of the test set is obtained. The predicted sequence is compared with the true sequence, and the Acc, Edit, and F1 indicators are calculated and used as evaluation indicators.

[0068] Figure 3 The figure shows the comparison of the effects of this method on the 50salads and Gtea datasets, where BaseLine represents the network ASFormer we compared, Ours represents the result of our method, and GT represents the true label of the video segment.

[0069] Indicator explanation:

[0070] Acc accuracy refers to the proportion of correctly predicted labels for all frames. The calculation method is:

[0071]

[0072] where p i and g i They represent the predicted category and label category of frame i respectively, and N is the total number of frames in the video.

[0073] F1: F1@{10, 25, 50} represents the comparison threshold. If the IOU between the predicted segmented segment and the labeled segment is greater than the threshold, the segmentation is considered correct.

[0074]

[0075] Edit: By solving the left-Wen distance of two action sequences, let the true value segmentation sequence be G = G1, G2, G3, ..., G M , the predicted segmentation sequence is P = P1, P2, P3, ..., P M , we calculate the split edit distance as:

[0076]

[0077] Our experimental results are shown in Table 1. For the 50salads dataset, our Acc is improved by 1.6%, the F1 score is improved by 1.9%, 1.4%, and 1.0%, and Edit is improved by 1.0.

[0078] For the Gtea dataset, our Acc is improved by 1.5%, the F1 scores are improved by 1.2%, 1.4%, 1.6%, and Edit is improved by 1.1%.

[0079]

[0080] Table 1

[0081] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0082] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A dual-branch network video segmentation method based on chi-square filtering, characterized in that: The steps include: Step 1: Use the existing 3D feature extraction network to extract features from the dataset; Step 2: Reprocess the features extracted from the dataset video; Step 3: Input the reprocessed features into the two-branch neural network. In order to combine the characteristics of the two-branch network, the boundary branch focuses on identifying the boundary, and the trunk branch focuses on identifying the action subject. The network parameters output by the two-branch stage are combined in the form of voting; Step 4: Back propagation updates parameters; Step 5: Test the trained network model and evaluate the trained model.

2. A dual-branch network video segmentation method based on chi-square filtering according to claim 1, characterized in that: Step 1: The existing 3D feature extraction network is used to extract features from the dataset data. The size of the extracted feature matrix is ​​T×N, where T is the number of frames in a single video and N is the extracted dimension. The dataset is divided into a training set and a test set. Then, representative actions are determined according to the video segmentation task, and the labels of representative actions are divided into a salient label set k. c middle.

3. A dual-branch network video segmentation method based on chi-square filtering according to claim 2, characterized in that: In step 1, according to the characteristics of the data set, the action with the largest number of labels in the data set is determined as the representative action, and the labels of the representative actions are divided into the significant label set k c among.

4. A dual-branch network video segmentation method based on chi-square filtering according to claim 2 or 3, characterized in that: Step 2: For a video D with K actions train =(x i ,y i ), where x i Represents the features extracted from the i-th frame image in the video, y i Represents the corresponding label. For the entire video, the extracted features are divided into K video segments SEG (seg1, seg2, ..., seg j ,…,seg K ), and then the chi-square value is calculated for the features in each video segment. The specific calculation formula is as follows: in represents the value of the feature of the zth frame of the jth video segment, represents the average value of the features of all frames in the jth video segment, Represents the video segment seg j The chi-square value of the z-th frame feature, z = 1...Z, where Z represents the number of frames in the video segment; According to the above operation, the video segment seg is traversed. j The chi-square value of all frames within The extracted features are ranked according to the size of these chi-square values ​​to obtain a ranked feature sequence; For the ranked feature sequence, the last 10% of the features in this video segment are screened out, and these features are optimized using a crossover method similar to a genetic algorithm; specifically, the positions of the last 10% of the features in the video segment in the original feature sequence are found, and then 2 frames are detected before and after, and the feature at the front of the sequence after the ranking is found in the 4 detected frames. This feature is then cross-operated with the feature to be optimized using a genetic algorithm, and the new feature is used to replace the original feature to obtain the reprocessed feature of the video segment. Finally, the reprocessed features of all video segments are combined to obtain the reprocessed feature sequence of the video:

5. A dual-branch network video segmentation method based on chi-square filtering according to claim 4, characterized in that: Step 3: The video reprocessing feature sequence SEC cross , respectively input into the boundary branch and trunk branch of the two-branch neural network; The boundary branch of the two-branch neural network consists of four modules with the same structure. A single module first matches the dimension through a 1x1 convolution, then processes the feature sequence through four fixed convolution kernels, and finally matches the output dimension through an N×K convolution to obtain the output of a single module of the boundary branch. The trunk branch uses the original structure of ASFormer, which also contains 4 modules, corresponding to the boundary branch; The input of the subsequent stage modules of the boundary branch and the trunk branch is judged by the voting module Vnote_Compare. The voting module uses a voting-like form to combine the outputs of the modules of each stage of the two branches. Specifically, let the output of the current stage module of the boundary branch be x1, and the output of the current stage module of the trunk branch be x2. The final output of the current stage of the two-branch neural network is determined by the following formula: in Indicates that the label in the input x1 is k c The number of Represents the total number of inputs of x1; Indicates that the label in the input x2 is k c The number of Represents the total number of inputs of x2; where k c It is the set of salient labels pre-defined in step 1; The final output of the current stage of the two-branch neural network determined by the voting module is used as the input of the next stage module of the boundary branch and the trunk branch; the final output of the last stage of the boundary branch and the trunk branch is the final prediction matrix. After the prediction matrix is ​​normalized by SoftMax, the final prediction sequence can be obtained.

6. A dual-branch network video segmentation method based on chi-square filtering according to claim 5, characterized in that: Step 4: The two-branch neural network is supervised by combining the standard cross-entropy loss function of multiple stages; Where K represents the number of actions in the video, S is the total number of two-branch neural network stages used, is the probability of predicting class l, and y i represents the true label of the input sample, x i represents the input sample, q(l|x i ) represents the sample x i Is it the first category?

Images