An unsupervised video behavior segmentation method based on bidirectional time sequence coding and double information fusion

Abstract

The application provides a kind of unsupervised video behavior segmentation method based on bidirectional time series coding and double information fusion, specifically comprising: S1. constructing feature pre-extraction module, for extracting the trajectory and motion characteristics of object in video;S2. constructing bidirectional time series context feature coding network, containing two multilayer perception feature mapping layers, forward and backward autoregressive encoder, and feature fusion layer;S3. constructing two-stage video behavior segmentation module, containing initial segmentation module based on prediction-similar fusion information, and optimization module;S4. based on the pre-extracted video frame feature, training set is constructed, and the feature coding network is trained;S5. unsupervised behavior segmentation is carried out on input video using feature pre-extraction module, trained feature coding network and two-stage video behavior segmentation module.Compared with existing methods, the application makes full use of time series context information and prediction-similar fusion information, and improves the segmentation quality and performance.

Description

Technical Field

[0001] This invention belongs to the fields of deep learning, computer vision and video behavior understanding, and specifically relates to an unsupervised video behavior segmentation method based on bidirectional temporal coding and dual information fusion. Background Technology

[0002] Video behavior segmentation is a core task in the field of computer vision. Its main goal is to segment video behavior data into different time segments, each segment corresponding to a specific behavior or action, providing key application value for video understanding fields such as behavior recognition, detection, and security monitoring.

[0003] Existing video action segmentation methods typically rely on supervised learning, which requires significant time and resources for frame-level action labeling, severely limiting their feasibility in practical applications. Furthermore, supervised video action segmentation methods often struggle to generalize to different scenarios because they are usually trained on specific datasets and cannot cope with varying environments and data types.

[0004] To overcome the limitations of supervised video action segmentation methods, unsupervised video action segmentation methods have gained attention in recent years. These methods aim to automatically segment different action segments in videos without requiring pre-provided action labels, thus significantly reducing annotation costs and increasing the feasibility of practical applications. Currently, unsupervised video action segmentation methods are mainly divided into two categories: prediction-based methods and clustering-based methods. Prediction-based methods typically use temporal models (such as recurrent neural networks) to predict future frames, using local extrema of the prediction loss sequence as segmentation boundaries. However, these methods are susceptible to interference from an inappropriate number of extrema, leading to fragmented segmentation results and an inability to associate segments of the same action category, thus facing oversegmentation problems. Simultaneously, recurrent neural networks are limited in long-term temporal context modeling, also affecting segmentation performance. Clustering-based methods cluster video frame features to obtain action groups, but discrete frame-level clustering cannot fully utilize temporal context information, easily producing fragmented segmentation results and oversegmentation problems, thus affecting action segmentation performance.

[0005] In summary, current unsupervised video action segmentation methods still face several challenges, including insufficient temporal context modeling, fragmented segmentation results, and oversegmentation. Therefore, it is necessary to propose innovative unsupervised video action segmentation methods to improve segmentation quality and adaptability, thereby better meeting the needs of different domains and applications. Summary of the Invention

[0006] To address the aforementioned problems and challenges faced by existing unsupervised video behavior segmentation methods, this invention provides an unsupervised video behavior segmentation method based on bidirectional temporal coding and dual information fusion.

[0007] An unsupervised video behavior segmentation method based on bidirectional temporal coding and dual information fusion is proposed, with video frame sequences as input and segmented behavior segments as output.

[0008] This method consists of three parts: a feature pre-extraction module, a bidirectional temporal context feature encoding network, and a two-stage video behavior segmentation module. Specifically, it includes the following steps: S1. Construct a feature pre-extraction module to extract the trajectory and motion features of objects in the video; S2. Construct a bidirectional temporal context feature encoding network, which includes two multilayer perceptron feature mapping layers, a forward autoregressive encoder, a backward autoregressive encoder, and a feature fusion layer; S3. Construct a two-stage video behavior segmentation module, which includes a fine-grained initial segmentation module based on prediction loss and similarity fusion information, and an optimization module that moves from fine-grained to coarse-grained segmentation. S4. Construct a training set based on pre-extracted video frame features and train a bidirectional temporal context feature encoding network; S5. Use the feature pre-extraction module, the trained bidirectional temporal context feature encoding network, and the two-stage video behavior segmentation module to perform unsupervised behavior segmentation on the input video.

[0009] In step S1, the feature pre-extraction module is used to extract the trajectory and motion features of objects in the video. Its input is a video frame sequence, and its output is the improved dense trajectory (IDT) features extracted from the video.

[0010] In step S2, the bidirectional temporal context feature coding network is used to re-encode the pre-extracted video frame feature sequence to fully utilize the temporal context information. Specific steps include: S21. Construct a feature mapping layer based on a multilayer perceptron to map the input pre-extracted video frame feature sequence to a new feature space. Its input includes two parts: the feature sequence consisting of the first frame to the second-to-last frame, also called the forward copy; and the reverse sequence consisting of the second frame to the last frame, also called the reverse copy. S22. Construct a forward autoregressive encoder based on an autoregressive transformer. This encoder consists of 12 basic layers, each layer including: a multi-head attention module, a feedforward neural network, two residual connections, and layer normalization. Its input is a forward-order copy of the output of step S21 mapped to the new feature space, and its output is the frame features predicted in an autoregressive manner for one step forward, i.e., the predicted features from the second frame to the penultimate frame, thereby realizing the forward encoding of the video frame sequence; S23. Construct a backward autoregressive encoder that is completely isomorphic to the forward autoregressive encoder. The difference is that its input is a reversed copy of the output of step S21 mapped to a new feature space, and its output is the feature predicted from the penultimate frame to the first frame in an autoregressive manner, thereby realizing the reverse encoding of the video frame sequence. S24. Construct a feature mapping layer based on a multilayer perceptron, map the predicted features output in steps S22 and S23 back to the initial feature space, and reverse the reverse order of the reversed results so that they correspond to the prediction results from the first frame to the second to last frame. S25. Construct a feature fusion layer, including: fusing the forward and reverse prediction features of the corresponding frames by adding and averaging. In particular, for the first and last frames, the reverse and forward prediction features are directly used as fused features, and the fused features are the encoded features output by the bidirectional temporal context feature encoding network.

[0011] In step S3, the two-stage video behavior segmentation module consists of an initial segmentation module and an optimization module. Its inputs are the pre-extracted features of the video and the encoded features output by the bidirectional temporal context feature encoding network, and its output is the video behavior segmentation result. The specific steps include: S31. Construct an initial segmentation module. Its inputs are the pre-extracted features of the video and the features encoded by the bidirectional temporal context feature encoding network. The output is the initial segmented segment. The specific process includes: calculating the loss sequence between frame-level encoded features and pre-extracted features, i.e., calculating the Euclidean distance between the encoded features and pre-extracted features of each frame to obtain the predicted loss sequence; calculating the inter-frame feature similarity sequence, i.e., for each frame, concatenating the encoded features and pre-extracted features as its features, and then sequentially calculating the cosine distance between adjacent frames to obtain the inter-frame similarity sequence; fusing the predicted loss sequence and the inter-frame similarity sequence using a weighted summation method, and then using mean filtering to denoise them; finally, detecting the local maxima of the fused sequence as the segmentation boundary to obtain the initial segmentation result. S32. Construct an optimization module. This module uses K-means clustering to merge related initial segmentation segments to optimize the initial segmentation results and overcome oversegmentation. The input to this module is the initial segmentation segments output from step S31, along with the pre-extracted features and encoded features corresponding to the video. The output is the optimized segmentation result. The specific process includes: treating the concatenation of encoded features and pre-extracted features as the features of each frame; using the mean of the features of all frames contained in each segment as its segment feature; using the K-means method to merge related segments to optimize the result; and using cosine distance to guide the clustering and merging of related segments during the optimization process.

[0012] Preferably, in step S31, when fusing the frame-level prediction loss sequence and the inter-frame feature similarity sequence, the weighted summation weights of the two are 0.5.

[0013] Step S4 specifically includes: S41. Construct the training dataset, specifically including: for each video, use a sliding window strategy to divide its pre-extracted frame feature sequence into segments of equal length, where the sliding window size is 500 and the step size is 100, and the divided feature segments form the training dataset. S42. Based on the constructed training dataset, train a bidirectional temporal context feature encoding network.

[0014] Furthermore, the single-iteration process of training the bidirectional temporal context feature encoder in step S42 is as follows: A1. Create two copies of the input feature sequence: a forward-order copy consisting of the first frame to the second-to-last frame, and a reverse-order copy consisting of the second frame to the last frame, and reverse the reverse-order copy. A2. Forward copy encoding specifically includes: First, inputting the forward copy into the feature mapping layer constructed in step S21, and inputting its output into the forward autoregressive encoder constructed in step S22, performing one-step forward prediction in an autoregressive manner to obtain the predicted features for the second frame to the penultimate frame; then, inputting these predicted features into the feature mapping layer constructed in step S24, mapping them back to the initial feature space to obtain the forward predicted features for the input video sequence; A3. Reverse copy encoding specifically includes: First, inputting the reverse copy into the feature mapping layer constructed in step S21, and inputting its output into the backward autoregressive encoder constructed in step S23, performing one-step forward prediction in an autoregressive manner to obtain the predicted features from the penultimate frame to the first frame; then, inputting these predicted features into the feature mapping layer constructed in step S24, mapping them back to the initial feature space, and reversing them to obtain the reverse predicted features for the input video sequence; A4. Using the feature fusion layer constructed in step S25, the predicted features output by A2 and A3 are fused to obtain the encoding result of the pre-extracted features of the input video; A5. The Euclidean distance between the encoded features output from step A4 and the pre-extracted features is used as the loss function to guide gradient backpropagation and parameter updates. The calculation process of the loss function is as follows: Where n represents the number of frames in the input video, d represents the feature dimension, and F and These represent pre-extracted features and predicted features, respectively.

[0015] In step S5, the specific steps for unsupervised behavior segmentation of the input video include: S51. Use the feature pre-extraction module constructed in step S1 to extract features from the input video; S52. Input the pre-extracted features into the trained bidirectional temporal context feature encoding network to obtain the corresponding encoded features; S53. Input the encoded features and pre-extracted features into the two-stage video behavior segmentation module constructed in step S3 to obtain the unsupervised video behavior segmentation result.

[0016] The beneficial effects of this invention include:

[0017] 1. This invention addresses the problem of unsupervised video behavior segmentation by proposing a method based on bidirectional temporal coding and dual information fusion. This method effectively encodes the temporal context, fully utilizes prediction and similarity fusion information, effectively overcomes the oversegmentation problem, and improves the accuracy of unsupervised video behavior segmentation.

[0018] 2. In order to improve the modeling and encoding capabilities of temporal context in unsupervised behavior segmentation tasks, this invention proposes a bidirectional temporal context feature encoding network. Based on isomorphic forward and backward autoregressive encoders, it performs one-step forward and backward prediction of future frames, effectively encoding forward and backward temporal information, thereby improving the temporal context modeling capability.

[0019] 3. This invention addresses the widespread oversegmentation problem in unsupervised behavior segmentation tasks by proposing a two-stage video behavior segmentation module. This module fully utilizes the temporal context information extracted by a bidirectional temporal context feature encoding network, combined with prediction and similarity fusion information, to effectively improve the initial segmentation accuracy of the model. Furthermore, it employs a clustering-based optimization module to merge related initial segmentation segments, effectively avoiding the oversegmentation problem. Attached Figure Description

[0020] Figure 1 This is the overall flowchart of the present invention.

[0021] Figure 2 This is a schematic diagram of the bidirectional temporal context coding network structure and loss function.

[0022] Figure 3 This is a schematic diagram of a two-stage video segmentation module.

[0023] Figure 4 This is the reasoning process for the present invention. Detailed Implementation

[0024] The method of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0025] Example 1

[0026] like Figure 1 As shown, the unsupervised video behavior segmentation method based on bidirectional temporal coding and dual information fusion provided by this invention mainly includes the following five steps: S1. Construct a feature pre-extraction module to extract the trajectory and motion features of objects in the video; S2. Construct a bidirectional temporal context feature encoding network, which includes two multilayer perceptron feature mapping layers, a forward autoregressive encoder, a backward autoregressive encoder, and a feature fusion layer; S3. Construct a two-stage video behavior segmentation module, which includes a fine-grained initial segmentation module based on prediction loss and similarity fusion information, and an optimization module that moves from fine-grained to coarse-grained segmentation. S4. Construct a training set based on pre-extracted video frame features and train a bidirectional temporal context feature encoding network; S5. Use the feature pre-extraction module, the trained bidirectional temporal context feature encoding network, and the two-stage video behavior segmentation module to perform unsupervised behavior segmentation on the input video.

[0027] Step S1 specifically includes: extracting improved dense trajectory (IDT) features for each behavioral video segment to describe the trajectory and motion features of objects in the video.

[0028] like Figure 2 As shown, step S2 specifically includes: S21. Construct a feature mapping layer based on a multilayer perceptron to map the input pre-extracted video frame feature sequence to a new feature space. Its input includes two parts: the feature sequence consisting of the first frame to the second-to-last frame, also called the forward copy; and the reverse sequence consisting of the second frame to the last frame, also called the reverse copy. S22. Construct a forward autoregressive encoder based on an autoregressive transformer. This encoder consists of 12 basic layers, each layer including: a multi-head attention module, a feedforward neural network module, two residual connections, and layer normalization. Its input is a forward-order copy of the output of step S21 mapped to the new feature space, and it predicts the frame features of the next forward step in an autoregressive manner, that is: predicting the features from the second frame to the penultimate frame, thereby realizing the forward encoding of the video frame sequence; S23. Construct a backward autoregressive encoder that is completely isomorphic to the forward autoregressive encoder. The difference is that its input is the reversed copy of the output of step S21 mapped to the new feature space, and it predicts the features from the penultimate frame to the first frame in an autoregressive manner, thereby realizing the reverse encoding of the video frame sequence. S24. Construct a feature mapping layer based on a multilayer perceptron, map the predicted features output in steps S22 and S23 back to the initial feature space, and reverse the reverse order of the reversed results so that they correspond to the prediction results from the first frame to the second to last frame. S25. Construct a feature fusion layer, including: fusing the forward and reverse prediction features of the corresponding frames by adding and averaging. In particular, for the first and last frames, the reverse and forward prediction features are directly used as fused features, and the fused features are the encoded features output by the bidirectional temporal context feature encoding network.

[0029] like Figure 3 As shown, step S3 specifically includes: S31. Construct an initial segmentation module. Its inputs are the pre-extracted features of the video and the features encoded by the bidirectional temporal context feature encoding network. The output is the initial segmented segment. The specific process includes: calculating the loss sequence between frame-level encoded features and pre-extracted features, i.e., calculating the Euclidean distance between the encoded features and pre-extracted features of each frame to obtain the predicted loss sequence; calculating the inter-frame feature similarity sequence, i.e., for each frame, concatenating the encoded features and pre-extracted features as its features, and then sequentially calculating the cosine distance between adjacent frames to obtain the inter-frame similarity sequence; fusing the predicted loss sequence and the inter-frame similarity sequence using a weighted summation method, and then using mean filtering to denoise them; finally, detecting the local maxima of the fused sequence as the segmentation boundary to obtain the initial segmentation result. S32. Construct an optimization module. This module uses K-means clustering to merge related initial segmentation segments to optimize the initial segmentation results and overcome oversegmentation. The input to this module is the initial segmentation segments output from step S31, along with the pre-extracted features and encoded features corresponding to the video. The output is the optimized segmentation result. The specific process includes: treating the concatenation of encoded features and pre-extracted features as the features of each frame; using the mean of the features of all frames contained in each segment as its segment feature; using the K-means method to merge related segments to optimize the result; and using cosine distance to guide the clustering and merging of related segments during the optimization process.

[0030] Preferably, in step S31, when fusing the frame-level prediction loss sequence and the inter-frame feature similarity sequence, the weighted summation weights of the two are 0.5.

[0031] Step S4 specifically includes: S41. Constructing the training dataset: For each video, use a sliding window strategy to divide its corresponding pre-extracted frame feature sequence into segments of equal length. The sliding window size is 500 frames and the step size is 100 frames. The divided video feature segments form the training dataset. S42. Based on the constructed training dataset, a bidirectional temporal context feature encoder is trained using self-supervised prediction loss.

[0032] Furthermore, such as Figure 2 As shown, the single iteration process of training the bidirectional temporal context feature encoding network in step S42 is as follows: A1. Create two copies of the input feature sequence: a forward-order copy consisting of the first frame to the second-to-last frame, and a reverse-order copy consisting of the second frame to the last frame, and reverse the reverse-order copy. A2. Forward copy encoding specifically includes: First, inputting the forward copy into the initial feature mapping layer, and inputting its output into the forward autoregressive encoder to obtain the predicted features for the second frame to the penultimate frame; then, inputting these predicted features into the second feature mapping layer, mapping them back to the initial feature space to obtain the forward predicted features for the input video sequence; A3. Reverse copy encoding specifically includes: First, inputting the reverse copy into the initial feature mapping layer, and inputting its output into the backward autoregressive encoder to obtain the predicted features from the penultimate frame to the first frame; then, inputting these predicted features into the second feature mapping layer, mapping them back to the initial feature space, and reversing them to obtain the reverse predicted features for the input video sequence. A4. Use a feature fusion layer to fuse the predicted features output by A2 and A3 to obtain the encoding of the pre-extracted features of the input video; A5. The Euclidean distance between the encoded features output from step A4 and the pre-extracted features is used as the loss function to guide gradient backpropagation and parameter updates. The calculation process of the loss function is as follows: Where n represents the number of frames in the input video, d represents the feature dimension, and F and These represent pre-extracted features and predicted features, respectively.

[0033] like Figure 4 As shown, step S5 specifically includes: S51. Use the feature pre-extraction module to extract frame-level features for the input video; S52. Input the pre-extracted feature sequence into a bidirectional temporal context feature encoding network to obtain the encoded features; S53. Input the encoded features and pre-extracted features into the two-stage video behavior segmentation module constructed in step S3 to obtain the unsupervised video behavior segmentation result.

Claims

1. An unsupervised video behavior segmentation method based on bidirectional temporal coding and dual information fusion, characterized in that, Includes the following steps: S1. Construct a feature pre-extraction module to extract the trajectory and motion features of objects in the video; S2. Construct a bidirectional temporal context feature encoding network, which includes two multilayer perceptron feature mapping layers, a forward autoregressive encoder, a backward autoregressive encoder, and a feature fusion layer; The bidirectional temporal context feature coding network is used to re-encode the pre-extracted video frame feature sequence to fully utilize temporal context information. Specific steps include: S21. Construct a feature mapping layer based on a multilayer perceptron to map the input pre-extracted video frame feature sequence to a new feature space. Its input includes two parts: a feature sequence consisting of the first frame to the second to last frame, also called a forward copy. The reverse order of the feature sequence consisting of the second frame to the penultimate frame is also called the reversed copy; S22. Construct a forward autoregressive encoder based on an autoregressive transformer. The encoder consists of 12 basic layers. Each layer includes: a multi-head attention module, a feedforward neural network, two residual connections, and layer normalization. Its input is a positive-order copy of the output of step S21 mapped to the new feature space. Its output is the frame features predicted in an autoregressive manner for one step forward, namely: the predicted features from the second frame to the penultimate frame, thereby realizing the forward encoding of the video frame sequence. S23. Construct a backward autoregressive encoder that is completely isomorphic to the forward autoregressive encoder. The difference is that its input is a reversed copy of the output of step S21 mapped to a new feature space, and its output is the feature predicted from the penultimate frame to the first frame in an autoregressive manner, thereby realizing the reverse encoding of the video frame sequence. S24. Construct a feature mapping layer based on a multilayer perceptron, map the predicted features output in steps S22 and S23 back to the initial feature space, and reverse the reversed results so that they correspond to the prediction results from the first frame to the second to last frame. S25. Construct a feature fusion layer, including: fusing the forward and reverse prediction features of the corresponding frames by adding and averaging. For the first and last frames, the reverse and forward prediction features are directly used as the fused features, and the fused features are the encoded features output by the bidirectional temporal context feature coding network. S3. Construct a two-stage video behavior segmentation module, which includes a fine-grained initial segmentation module based on prediction loss and similarity fusion information, and an optimization module that moves from fine-grained to coarse-grained segmentation. Specifically, it includes: S31. Construct an initial segmentation module. Its inputs are the pre-extracted features of the video and the features encoded by the bidirectional temporal context feature encoding network. The output is the initial segmentation segment. The specific process includes: calculating the loss sequence between frame-level encoded features and pre-extracted features, i.e., calculating the Euclidean distance between the encoded features and pre-extracted features of each frame to obtain the predicted loss sequence; calculating the inter-frame feature similarity sequence, i.e., for each frame, concatenating the encoded features and pre-extracted features as its features, and then calculating the cosine distance between adjacent frames to obtain the inter-frame similarity sequence; fusing the predicted loss sequence and the inter-frame similarity sequence in a weighted summation manner, and then using mean filtering to denoise them; finally, detecting the local maxima points of the fused sequence as the segmentation boundary to obtain the initial segmentation result. S32. Construct an optimization module. This module uses the K-means clustering method to merge related initial segmentation segments to optimize the initial segmentation results and overcome the oversegmentation problem. The input of this module is the initial segmentation segment output in step S31, as well as the pre-extracted features and encoded features corresponding to the video. The output is the optimized segmentation result. The specific process includes: treating the connection between the encoded features and the pre-extracted features as the features of each frame, taking the mean of the features of all frames contained in each segment as its segment features, using the K-means method to merge related segments to optimize the results, and using cosine distance to guide the clustering and merging of related segments during the optimization process. S4. Construct a training set based on pre-extracted video frame features and train a bidirectional temporal context feature encoding network; S5. Use the feature pre-extraction module, the trained bidirectional temporal context feature encoding network, and the two-stage video behavior segmentation module to perform unsupervised behavior segmentation on the input video.

2. The method according to claim 1, characterized in that, Step S1 specifically includes: extracting improved dense trajectory features for the input behavioral video, which are used to describe the trajectory and motion features of objects in the video.

3. The method according to claim 1, characterized in that, Step S4 specifically includes: S41. Construct the training dataset, specifically including: for each video, use a sliding window strategy to divide its pre-extracted frame feature sequence into segments of equal length, where the sliding window size is 500 and the step size is 100, and the divided feature segments form the training dataset; S42. Based on the constructed training dataset, train a bidirectional temporal context feature encoding network.

4. The method according to claim 3, characterized in that, The single iteration process of training the bidirectional temporal context feature encoder in step S42 is as follows: A1. Create two copies of the input feature sequence: a forward copy consisting of the first frame to the second-to-last frame, and a reverse copy consisting of the second frame to the last frame, and reverse the reverse copy. A2. Forward copy encoding specifically includes: First, inputting the forward copy into the feature mapping layer constructed in step S21, and inputting its output into the forward autoregressive encoder constructed in step S22, performing one-step forward prediction in an autoregressive manner to obtain the predicted features for the second frame to the penultimate frame; then, inputting these predicted features into the feature mapping layer constructed in step S24, mapping them back to the initial feature space to obtain the forward predicted features for the input video sequence; A3. Reverse copy encoding, specifically including: First, inputting the reverse copy into the feature mapping layer constructed in step S21, and inputting its output into the backward autoregressive encoder constructed in step S23, performing one-step forward prediction in an autoregressive manner to obtain the predicted features from the penultimate frame to the first frame; then, inputting these predicted features into the feature mapping layer constructed in step S24, mapping them back to the initial feature space, and reversing them to obtain the reverse predicted features for the input video sequence; A4. Using the feature fusion layer constructed in step S25, the predicted features output by A2 and A3 are fused to obtain the encoding result of the pre-extracted features of the input video; A5. The Euclidean distance between the encoded features output from step A4 and the pre-extracted features is used as the loss function to guide gradient backpropagation and parameter updates. The calculation process of the loss function is as follows: ; in, This indicates the number of frames contained in the input video. Representing feature dimension, and These represent pre-extracted features and predicted features, respectively.

5. The method according to claim 1, characterized in that, In step S5, the specific steps for unsupervised behavior segmentation of the input video include: S51. The feature pre-extraction module constructed in step S1 is used to extract features from the input video; S52. Input the pre-extracted features into the trained bidirectional temporal context feature encoding network to obtain the corresponding encoded features; S53. Input the encoded features and pre-extracted features into the two-stage video behavior segmentation module constructed in step S3 to obtain the unsupervised video behavior segmentation result.

Images