A chi-square filter-based dual-branch network video segmentation method

Abstract

The application discloses a kind of double-branch network video segmentation method based on chi-square filter.First, the existing 3D feature extraction network is used to extract features from the data set, and then the extracted features are reprocessed;The reprocessed features are input into the double-branch neural network, in order to combine the characteristics of the two branch networks, the boundary branch focuses on identifying the boundary, the main branch focuses on identifying the action subject, and the network parameters output by the double-branch stage are combined in the form of voting;Finally, the parameters are updated by back propagation.The application is aimed at the overfitting problem caused by training the neural network using features containing noise, and proposes a feature preprocessing method, which uses chi-square filter and genetic algorithm to optimize the unprocessed features.The application combines the preprocessing part with the double-branch neural network, improves the prediction accuracy, and achieves excellent results on 50Salads and Gtea data sets.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and mainly relates to feature preprocessing and video action segmentation, and in particular to a dual-branch network video segmentation method based on chi-square filtering. BACKGROUND

[0002] Existing research on action video segmentation is usually done in many ways, including using diffusion models and using TCN networks, and the most typical TCN is to complete this task by using two stages: first, using a feature extraction network such as a two-dimensional CNN, a double-flow neural network, or a 3D CNN to extract spatial or temporal features, and then using an RNN or a time convolution network to classify the extracted features. These models have better results than earlier methods on datasets with a small number of action classes and video clips. However, using a feature extraction network to extract spatial and temporal features as the input of the neural network often relies too much on the feature extraction network, and the features extracted by the existing feature extraction network often contain too much noise, which is not conducive to the training of the neural network and will greatly affect the accuracy of the final prediction. Moreover, if a single network is used to classify the extracted features for video action, these single networks are often too biased and will confuse the main part of the action with the boundary part, especially when identifying the same action in different segments, which we generally call over-segmentation error. If it is on a large dataset with different action categories, this will lead to more significant over-segmentation errors. SUMMARY

[0003] The present application is aimed at the shortcomings of the existing method, and provides a dual-branch network video segmentation method based on chi-square filtering. The present application solves the problem of low prediction accuracy caused by the existing time convolution network (TCN) when the training data contains many disturbances.

[0004] The present application first obtains the time sequence pictures and corresponding labels in the original dataset; then uses existing 3D convolution network (I3D or C3D) and other feature extraction networks to extract features for the purpose of neural network training and to obtain more action information and action boundary information. The chi-square filtering preprocessing module is used to further optimize the features extracted by the feature extraction network.

[0005] Then the optimized features are input into our dual-branch network for prediction to obtain the final prediction result. In our dual-branch network, by sharing the network parameters of each branch at each stage, we reduce the over-segmentation error caused by the excessive bias of a single network and improve the accuracy of the video action segmentation task.

[0006] The technical solution adopted by the present application to solve its technical problems is as follows:

[0007] Step 1: Feature extraction of the dataset by using existing 3D feature extraction network (such as I3D, C3D, etc.).

[0008] Step 2: Re-processing of the features extracted from the dataset video.

[0009] Step 3: Input the re-processed features into a dual-branch neural network. In order to combine the characteristics of the two branch networks, the boundary branch focuses on identifying boundaries, and the main branch focuses on identifying the main body of the action. The network parameters output by the two branch stages are combined by voting.

[0010] Step 4: Update the parameters by back propagation.

[0011] Step 5: Test the trained network model and evaluate the trained model.

[0012] Further, the specific method of step 1 is as follows:

[0013] Feature extraction of the dataset data by using existing 3D feature extraction network, the size of the extracted feature matrix is T x N, T is the frame number of a single video, N is the extracted dimension, and the dataset is divided into training set and test set. Then determine the representative action according to the video segmentation task, and divide the label of the representative action into the significant label set k c .

[0014] In an embodiment, in step 1, according to the characteristics of the dataset, the action with the most labels in the dataset is determined as the representative action, and the label of the representative action is divided into the significant label set k c .

[0015] Further, the specific method of step 2 is as follows:

[0016] For a video D train = (x i , y i ), where x i represents the extracted features of the i-th frame image in the video, y i represents the corresponding label, for the entire video, the extracted features are divided into K video segments SEG (seg1, seg2, …, seg j , …, seg K ) according to the initial label of the K actions, then the chi-square value calculation is performed for the features in each video segment, the specific calculation formula is as follows:

[0017]

[0018] wherein a value of a feature representing a z-th frame of a j-th video segment, an average value of a feature representing all frames of a j-th video segment, a chi-square value of a feature representing a z-th frame of a video segment seg j , z = 1…Z, Z represents the number of frames of the video segment.

[0019] According to the above operation, the chi-square values of all frames in the video segment seg j are obtained. According to the size of these chi-square values, the extracted features are ranked to obtain a ranked feature sequence.

[0020] For the ranked feature sequence, the last 10% of the features in this video segment are screened out, and these features are optimized using a crossover method similar to a genetic algorithm. Specifically, the positions of the last 10% of the features in the original feature sequence are found, and then 2 frames before and after are detected. In the 4 frames detected, the feature with the highest sequence after ranking is found, and then the genetic algorithm crossover operation is performed between this feature and the feature to be optimized to replace the original feature as a new feature, obtaining the reprocessed feature of the video segment

[0021] Finally, the reprocessed features of all video segments are combined to obtain the reprocessed feature sequence of the video:

[0022]

[0023] Further, the specific method of step 3 is as follows:

[0024] The reprocessed feature sequence SEG cross of the video is input into the boundary branch and the trunk branch of the double-branch neural network, respectively.

[0025] The boundary branch of the double-branch neural network is composed of 4 modules with the same structure. A single module in it: first, match the dimensions through a 1x1 convolution, then process the feature sequence through 4 fixed convolution kernels, and finally match the output dimensions through an NxK convolution to obtain the output of the single module of the boundary branch.

[0026] The trunk branch uses the original structure of ASFormer, which also contains 4 modules corresponding to the boundary branch.

[0027] The input of the boundary branch and the trunk branch subsequent stage module is judged by a voting module Vnote_Compare, the voting module adopts a voting-like form to combine the outputs of the two branch stage modules, and the specific process is as follows: the output of the boundary branch current stage module is x1, the output of the trunk branch current stage module is x2, and the final output of the double-branch neural network current stage is determined through the following formula:

[0028]

[0029] Wherein represents the number of labels k c in the input x1, represents the total number of inputs of x1. represents the number of labels k c in the input x2, represents the total number of inputs of x2.

[0030] Wherein k c is a set of significant labels predefined in step 1.

[0031] The final output of the double-branch neural network current stage determined by the voting module is used as the input of the next stage module of the boundary branch and the trunk branch. The final output of the last stage of the boundary branch and the trunk branch is the final prediction matrix, and after the prediction matrix is normalized by SoftMax, the final prediction sequence can be obtained.

[0032] Further, the specific method of step 4 is as follows:

[0033] The double-branch neural network is supervised by combining the standard cross-entropy loss functions of multiple stages.

[0034]

[0035] Wherein K represents the number of actions of the video, S is the total number of stages of the double-branch neural network adopted, is the probability of predicting class l, y i represents the real label of the input sample, x i represents the input sample, q(l|x i ) represents whether the sample x i is the l-th class (1 if yes, otherwise 0).

[0036] The present application has the following advantages:

[0037] To address the overfitting problem caused by using noisy features for training neural networks, we proposed a feature preprocessing method that uses chi-square filtering and genetic algorithms to optimize unprocessed features. This method is suitable for tasks that use extracted features as neural network input when processing video segmentation tasks. To address the over-segmentation and boundary prediction problems caused by a single network, we designed a neural network based on boundary prediction and backbone prediction. The boundary branch network is mainly used to identify the boundaries between different actions in the video, and the backbone branch mainly predicts the category to which each video action belongs. A voting method based on ensemble learning is used between the two branches to combine the advantages and disadvantages of the two neural networks. Based on this, we designed a two-branch neural network. By combining the preprocessing part with the two-branch neural network, we improved the prediction accuracy and achieved excellent results on the 50Salads and Gtea datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flowchart of the method execution of an embodiment of the present invention.

[0039] Figure 2 A schematic diagram of the overall network structure adopted by an embodiment of the present invention.

[0040] Figure 3 This figure compares the effects of the present invention on the 50salads and Gtea datasets. DETAILED DESCRIPTION

[0041] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0042] like Figure 1 As shown, a dual-branch network video segmentation method based on chi-square filtering includes the following steps:

[0043] Step 1: Use the existing 3D feature extraction network (I3D) to extract features from the dataset (50Salads, Gtea). The size of the extracted feature matrix is ​​(T×N) (T is the number of frames in a single video, N is the extracted dimension), and the dataset is divided into a training set and a test set with a specific division ratio of 4:1. Then, representative actions are determined according to the video segmentation task, and the labels of representative actions are divided into a salient label set k c The purpose of this is to determine the representative actions of the dataset based on the initial label ratio, and use this as a basis for optimizing the neural network. (For the 50Salads dataset, the labels containing the add action account for the highest proportion of labels in the dataset, so the labels of actions such as (add_oil, add_vinegar) are used as the representative action labels, and the label containing the stir action is used as the representative action label for Gtea.)

[0044] In one embodiment, the action with the most labels in the dataset is determined as the representative action according to the characteristics of the dataset in step 1, and the labels of the representative action are divided into the significant label set k c When.

[0045] Step 2: reprocessing the features extracted in the dataset video (corresponding to the chi-square filtering part in Figure 2 When, the reprocessed feature sequence of the video is obtained.

[0046] For a video D with K actions train = (x i , y i ), where x i represents the features extracted from the i-th image in the video, y i represents the corresponding label, and for the entire video, the extracted features are divided into K video segments SEG (seg1, seg2, …, seg j , …, seg K ) according to the K actions of the initial label, and then the chi-square value is calculated for the features in each video segment. The specific calculation formula is as follows:

[0047]

[0048] wherein represents the value of the z-th frame feature of the j-th video segment, represents the average value of the features of all frames of the j-th video segment, represents the chi-square value of the z-th frame feature of the video segment seg j , z = 1 … Z, Z represents the number of frames of the video segment.

[0049] According to the above operation, the chi-square values of all frames in the video segment seg j are obtained We rank the extracted features according to the size of these chi-square values to obtain a ranked feature sequence (such as the Rank_Cross part in Figure 2 When).

[0050] For this ranked feature sequence, the last 10% of the features in this video segment are selected, and these features are optimized (such as seg j The original feature sequence without processing is After our ranking, the features to be optimized are detected as and Part). In our task, we hope that the optimized features still have the action information of the original action segment, so we use a genetic algorithm-like crossover method. Specifically, we find the last 10% of the features in the video segment (the last 10% of the features in the video segment are the features that are closest to the end of the video segment) With ) the position of each in the original feature sequence, and then detect 2 frames before and after, find the feature closest to the top of the sequence after ranking in the 4 detected frames, then perform a genetic algorithm crossover operation between this feature and the feature to be optimized, and replace the original feature with the new feature as the reprocessed feature of the video segment According to the above method, we only process 10% of the features, in this case, we process the last 10% of the features With Part, then our processed feature sequence is This is the reprocessed feature of the video segment.

[0051] Finally, combine all the reprocessed features of the video segments to get the final reprocessed feature sequence of the video:

[0052]

[0053] Step 3: input the reprocessed feature sequence SEG cross of the video into the dual-branch neural network, and combine the network parameters output by the dual-branch stage using a voting method.

[0054] Input the reprocessed feature sequence SEG cross of the video into the boundary branch and the backbone branch of the dual-branch neural network, respectively.

[0055] As shown in Figure 2 , the boundary branch of the dual-branch neural network consists of 4 modules with the same structure. A single module: first match the dimensions through a 1x1 convolution, then process the feature sequence through 4 fixed convolution kernels (we set it to 5), and finally match the output dimensions through an (N x K) convolution to obtain the output of the boundary branch single module.

[0056] The backbone branch uses the original structure of ASFormer, which also contains 4 modules corresponding to the boundary branch.

[0057] The input of the subsequent stage modules of the boundary branch and the trunk branch is determined by a voting module (Vnote_Compare). The voting module (Vnote_Compare) combines the outputs of the stage modules of the two branches in a voting manner, and the specific process is as follows: let the output of the current stage module of the boundary branch be x1, and the output of the current stage module of the trunk branch be x2. The final output of the current stage of the double-branch neural network is determined by the following formula:

[0058]

[0059] wherein represents the number of labels k in the input x1, c represents the total number of inputs of x1. represents the number of labels k in the input x2, c represents the total number of inputs of x2.

[0060] wherein k c is a set of significant labels (k c has multiple) predefined in step 1.

[0061] The final output of the current stage of the double-branch neural network determined by the voting module (Vnote_Compare) is used as the input of the next stage module of the boundary branch and the trunk branch. The final output of the last stage of the boundary branch and the trunk branch is the final prediction matrix. After the prediction matrix is normalized by SoftMax, the final prediction sequence can be obtained (for the training set data, the segmentation result of the training set is obtained to calculate the loss).

[0062] Step 4: Update parameters by back propagation.

[0063] Instead of only using the loss function of the last layer to supervise the double-branch neural network, the double-branch neural network is supervised by combining the standard cross-entropy loss functions of multiple stages.

[0064]

[0065] wherein K represents the number of actions of the video, S is the total number of stages of the double-branch neural network adopted, p(l|x i ) is the probability of predicting class l, y i represents the true label of the input sample, x i represents the input sample, and q(l|x i ) represents whether the sample x i is of the lth class (1 if yes, and 0 if no).

[0066] ​​Step 5: Test the trained network model and evaluate it by Acc accuracy, F1, Edit, etc.

[0067] The test set features (T x N) dimension divided in step 1 are input into the final trained model in step 4, and the final prediction matrix (T x K) dimension of the test set is obtained. After SoftMax normalization processing, the final prediction sequence of the test set (T x 1 dimension) is obtained. The prediction sequence is compared with the real sequence to calculate Acc, Edit, and F1 indicators, which are used as evaluation indicators.

[0068] Figure 3 The results of the method on the 50salads and Gtea data sets are shown in the table, where BaseLine represents the network ASFormer we compared, Ours represents the results obtained by our method, and GT represents the real label of the video segment.

[0069] Indicator explanation:

[0070] Acc accuracy refers to the proportion of correct prediction labels for all frames. The calculation method is:

[0071]

[0072] where p i and g i represent the predicted class and label class of frame i, respectively, and N is the total number of frames in the video.

[0073] F1: F1@{10, 25, 50} indicates a comparison threshold. If the IOU between the predicted segment and the labeled segment is greater than the threshold, the segmentation is considered correct.

[0074]

[0075] Edit: By solving the left Levenshtein distance of two action sequences, let the true segmentation sequence be G = G1, G2, G3,..., G M , and the predicted segmentation sequence be P = P1, P2, P3,..., P M , we calculate the segmentation edit distance as:

[0076]

[0077] Our experimental results are shown in Table 1. For the 50salads data set, our Acc increased by 1.6%, F1 scores increased by 1.9%, 1.4%, 1.0%, and Edit increased by 1.0%.

[0078] For the Gtea dataset, our Acc improves by 1.5%, F1 scores by 1.2%, 1.4%, 1.6%, and Edits by 1.1%.

[0079]

[0080] Table 1

[0081] The above description is further detailed in conjunction with specific / preferred embodiments of the present application, and cannot be deemed to limit the specific implementation of the present application to these descriptions. For those of ordinary skill in the art to which the present application belongs, without departing from the concept of the present application, they can make several substitutions or modifications to these described embodiments, and these substitutions or modifications shall be deemed to belong to the protection scope of the present application.

[0082] The part of the present application not described in detail belongs to the technology known to those skilled in the art.

Claims

1. A dual-branch network video segmentation method based on chi-square filtering, characterized in that: The steps are as follows: Step 1: Use the 3D feature extraction network to extract features from the dataset; Step 2: Reprocess the features extracted from the dataset video; the specific method is as follows: For a Action video ,in Represents the video The features extracted from the frame image, Represents the corresponding label. For this entire video, according to the initial label The extracted features are divided into K video segments , and then calculate the chi-square value for the features in each video segment. The specific calculation formula is as follows: in Indicates the The first video segment The value of the frame's features, Representative The average value of the features of all frames in a video segment, Representative video segment The chi-square value of the z-th frame feature, z=1 Z, Z represents the number of frames in the video segment; Traverse to get video segments Chi-square values ​​for all frames within , ,… ,… ; Rank the extracted features according to the size of these chi-square values ​​to obtain a ranked feature sequence; For the ranked feature sequence, the last 10% of the features in this video segment are screened out, and these features are optimized using the crossover method of the genetic algorithm; the respective positions of the last 10% of the features in the video segment in the original feature sequence are found, and then 2 frames are detected before and after, and the feature at the top of the sequence after the ranking is found in these 4 detected frames. This feature is then cross-operated with the feature to be optimized using the genetic algorithm, and the new feature is used to replace the original feature to obtain the reprocessed feature of the video segment. ; Finally, the reprocessing features of all video segments are combined to obtain the reprocessing feature sequence of the video: Step 3: Input the reprocessed features into the two-branch neural network and use voting to combine the network parameters output from the two-branch stage. The specific method is as follows: The reprocessed feature sequence of the video , respectively input into the boundary branch and trunk branch of the two-branch neural network; The boundary branch of the two-branch neural network consists of four modules with the same structure. A single module first matches the dimension through a 1x1 convolution, then processes the feature sequence through four fixed convolution kernels, and finally matches the output dimension through an NxK convolution to obtain the output of a single module of the boundary branch. The trunk branch uses the original structure of ASFormer, which also contains 4 modules corresponding to the boundary branches; The input of the subsequent stage modules of the boundary branch and the main branch is judged by the voting module Vnote_Compare. The voting module uses voting to combine the outputs of the modules of each stage of the two branches. The details are as follows: Suppose the output of the current stage module of the boundary branch is , the output of the current stage module of the trunk branch is , the final output of the current stage of the two-branch neural network is determined by the following formula: in Indicates that the input The label is the number of express The total number of inputs; Indicates that the input The label is the number of express The total number of inputs; in is the set of salient labels pre-defined in step 1; The final output of the current stage of the two-branch neural network determined by the voting module is used as the input of the next stage module of the boundary branch and the trunk branch. The final output of the last stage of the boundary branch and the trunk branch is the final prediction matrix. After the prediction matrix is ​​normalized by SoftMax, the final prediction sequence can be obtained. Step 4: Back propagation to update parameters; Step 5: Test the trained network model and evaluate the trained model.

2. A dual-branch network video segmentation method based on chi-square filtering according to claim 1, characterized in that: Step 1: The 3D feature extraction network is used to extract features from the dataset data. The size of the extracted feature matrix is ​​T×N, where T is the number of frames in a single video and N is the extracted dimension. The dataset is divided into training and test sets. Representative actions are then determined based on the video segmentation task, and the labels of representative actions are divided into salient label sets. middle.

3. A dual-branch network video segmentation method based on chi-square filtering according to claim 2, characterized in that: In step 1, based on the characteristics of the dataset, the action with the largest number of labels in the dataset is determined as the representative action, and the labels of the representative actions are divided into the salient label set middle.

4. A dual-branch network video segmentation method based on chi-square filtering according to claim 3, characterized in that: Step 4: Supervise the two-branch neural network by combining the standard cross-entropy loss function of multiple stages; Where K represents the number of actions in the video, S is the total number of two-branch neural network stages used, and The probability of represents the true label of the input sample, represents the input sample, Representative samples Is it the first kind.

Images