Video temporal segmentation method for manifold structures

Abstract

The application discloses a video time sequence segmentation method of a manifold structure. The method comprises the following steps: generating a local manifold space feature according to a video frame of a video segment; generating a maximum neighbor time sequence feature according to the video frame of the video segment; and performing self-supervised learning of video representation according to the manifold space feature and the maximum neighbor time sequence feature. Thus, the video time sequence segmentation method is realized, the manifold information and the time sequence information in a long video are fully mined in the learning process, and the performance of the video time sequence segmentation task is improved.

Description

Technical Field

[0001] This invention relates to the field of video learning, and more particularly to a video temporal segmentation method for manifold structures. Background Technology

[0002] The purpose of video temporal segmentation is to divide an unlabeled long video into independent video segments by exploring the manifold structure and temporal relationships present in it, in order to extract content information from them.

[0003] The manifold structure-based video temporal segmentation method is based on self-supervised learning techniques. However, current video temporal segmentation algorithms cannot handle complex scenarios with long video recordings and have failed to effectively improve video temporal segmentation performance. Summary of the Invention

[0004] Research has revealed that current long-video temporal segmentation algorithms typically focus on improving segmentation performance by considering visual information in the image and the temporal relationships within the video, often neglecting the intrinsic connection between the manifold structure and temporal information within video frames. This results in insufficient attention and consideration being given to the temporal information in the video. However, in long-video scenarios, semantic information and temporal information are highly correlated.

[0005] This invention proposes a self-supervised learning method for long video temporal segmentation, which fully utilizes the ubiquitous and crucial manifold structure information during the learning process, thereby increasing the accuracy of temporal segmentation.

[0006] Some embodiments of this invention propose a self-supervised learning method based on manifold structures. Temporal segmentation of a first-view long video is used here as a specific example to explain this invention, but it is not intended to limit the invention. The method includes the following steps:

[0007] (1) Sample the original video at a certain frame rate to obtain a set of video frames with temporal relationship;

[0008] (2) Extract the initial visual features of the video frame set by using a ResNet50 neural network pre-trained on the ImageNet dataset;

[0009] (3) Construct manifold structure features based on the target frame according to the content of the video; in some embodiments, this includes: dividing the video into multiple groups using a sliding window method and calculating the manifold structure features of the frames within the group;

[0010] (4) Constructing a dynamic time-series network based on an LSTM network includes: constructing a nearest neighbor domain and constructing the LSTM time-series network; among which:

[0011] (4-1) In some embodiments, the construction of the nearest neighbor region includes: determining the maximum similarity nearest neighbor matrix by measuring the nearest neighbor relationship between the target frame and the frames in the neighbor region, and selecting stable adjacent video frames as the input of the dynamic temporal network;

[0012] (4-2) In some embodiments, the construction of the LSTM temporal network is as follows: a subset of the nearest neighbors with temporal relationships of a certain length is selected based on the maximum similarity nearest neighbor matrix and input into the LSTM network. The end of the LSTM network uses max pooling and shortcut to obtain the temporal features of the current target frame.

[0013] (5) The temporal features of the target frame are input into the decoder to reconstruct the original features;

[0014] (6) Calculate the reconstruction loss between the original features and the reconstructed features;

[0015] (7) Calculate the manifold structure loss between temporal features and manifold structure features;

[0016] (8) Iterate through the original feature set of the entire video frame in sequence, repeating the following process until the model converges:

[0017] (8-1) Calculate the reconstruction loss and manifold structure loss through steps (4) to (7);

[0018] (8-2) Minimize the loss in (8-1) and optimize the parameters in the dynamic temporal network;

[0019] (9) Input the original feature set of the entire video frame into the dynamic temporal network to obtain the low-dimensional high-quality feature set of the frame;

[0020] (10) Use clustering algorithm to obtain the temporal division of the video frame.

[0021] After the model converges, the Dynamic Temporal Network models the temporal relationships containing manifold structure features from the current video segment. For a given video segment, the Dynamic Temporal Network can output low-dimensional effective features for each frame, and use a clustering algorithm to obtain the unsupervised temporal partitioning results of the video frames.

[0022] As can be seen from the above training process, this invention uses a self-supervised learning mechanism to achieve unsupervised temporal segmentation in the field of video understanding, and the network can be trained using a gradient-based approach.

[0023] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0024] 1. Compared with existing mainstream self-supervised learning schemes, incorporating manifold structure information can comprehensively mine temporal information in long videos, thereby getting rid of data dependence and realizing unsupervised temporal segmentation of video data.

[0025] 2. In order to obtain local manifold structure information, this invention adopts a domain-based partitioning approach to model the manifold structure of local regions around video frames, thereby reducing semantic interference caused by distant video frames.

[0026] 3. To obtain effective temporal features, this invention constructs a dynamic temporal network based on an LSTM network structure. This network measures and obtains nearest neighbor information in the temporal domain by constructing a maximum similarity nearest neighbor matrix based on the target frame, thus preserving deeper semantic features. Attached Figure Description

[0027] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings, which will enable a clearer understanding of the present invention.

[0028] Figure 1 This is a schematic diagram of the process of the present invention.

[0029] Figure 2 This is a model diagram of the present invention.

[0030] Figure 3 For comparison, see the experimental diagram.

[0031] Figure 4 The temporal features output by this invention are visualized for different training stages.

[0032] Figure 5 This is an actual rendering.

[0033] Specific implementation process

[0034] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0035] This invention proposes a self-supervised learning method for long video temporal segmentation, which fully utilizes the ubiquitous and crucial manifold structure information during the learning process, thereby increasing the accuracy of temporal segmentation.

[0036] Figure 1 The flowchart illustrates a self-supervised video temporal segmentation method based on a manifold structure, according to some embodiments of the present invention.

[0037] like Figure 1 As shown, the method of this embodiment includes steps 1-10.

[0038] In step 1, the original video is sampled at a certain frame rate to obtain a set of video frames with temporal relationships;

[0039] Current video processing typically uses average frame sampling to process the video into single-frame images, and then processes the set of video frames. Using an appropriate frame rate for video sampling to meet different task requirements is beneficial to learning efficiency.

[0040] In step 2, the initial visual features of the video frame set are extracted using a ResNet50 neural network pre-trained on the ImageNet dataset.

[0041] For a video segment with N frames, the preprocessed ResNet50 model can obtain the initial semantic features X = {x1, x2, ..., x...} N After video frames are preprocessed using the ResNet50 model, visual features with basic semantic features can be obtained. Without large-scale data pre-training, this can greatly improve the accuracy of unsupervised tasks. Compared with random features, visual features obtained by using the preprocessed model are more suitable as the initial semantic features.

[0042] In step 3, based on the content of the video, a manifold structure feature based on the target frame is constructed; in some embodiments, this includes: dividing the video into multiple groups using a sliding window approach and calculating the manifold structure feature of the frames within each group;

[0043] Given initial semantic features X, construct a sliding window of length L; by moving the sliding window, segment the feature set into... There are three feature sets W of length L; where W = W1, W2, ..., W3. NL For each small segment W i We employ a manifold embedding method to generate a low-dimensional representation M of a uniform manifold. i This method is just one example; other manifold methods can also be used for embedding.

[0044] By normalizing x i x is represented by the distances to its k nearest neighbors. i The geographical distance between it and its nearest neighbors; then for each x i Individual custom distances are constructed and merged into a consistent global structure, which is then represented as a fuzzy set of global manifolds using spatial transformation. Finally, a closely matched low-dimensional vector is constructed by finding the topological structure of the original video frame feature set, thereby obtaining the manifold features M of the segment. i ;

[0045] In step 4, constructing a dynamic time-series network based on an LSTM network includes: constructing a nearest neighbor domain and constructing the LSTM time-series network; wherein:

[0046] (4-1) In some embodiments, the construction of the nearest neighbor region includes: determining the maximum similarity nearest neighbor matrix by measuring the nearest neighbor relationship between the target frame and the neighboring frames, and selecting stable adjacent video frames as the input of the dynamic temporal network;

[0047] Before inputting the data into subsequent networks, a maximum similarity nearest neighbor matrix needs to be constructed to increase the effectiveness of the input frames. Given the nearest neighbor length L and limiting the number of nearest neighbors ner, the nearest neighbor set of the video frame feature set X can be represented as follows: Let x1 be the distance between x1 and its other nearest neighbors within the current nearest neighbor range; the distance is calculated using the following formula:

[0048]

[0049] Where X ik For X i The eigenvalues ​​of the k-th dimension of X, where d is the eigenvalue of X. i The dimension is determined by arranging the nearest neighbor set pairwide_dist in ascending order, extracting the manifold feature vectors corresponding to the first ner+1 minimum distances, and finally forming the video frame set for network input.

[0050] (4-2) In some embodiments, the construction of the LSTM temporal network is as follows: a subset of the nearest neighbors with temporal relationships of a certain length is selected based on the maximum similarity nearest neighbor matrix and input into the LSTM network. The end of the LSTM network uses max pooling and shortcut operations to obtain the temporal features of the current target frame.

[0051] Given an input time-domain length T, construct an LSTM network and input the video frame feature set obtained in (4-1) into the network to obtain the hidden layer outputs {h1,...,h} of the LSTM network. t Then, after max pooling and shortcut operations, the final output g is obtained. t It can be obtained from the following formula:

[0052]

[0053] In step 5, the temporal features of the target frame are input into the decoder to reconstruct the original features;

[0054] To preserve as much information as possible about the original input features, this invention uses an autoencoder network structure to encode the temporal features g output by the dynamic temporal network. t Reconstructed into the original feature z t ;

[0055] z t =Decoder(g t )=σ(ω t (gt (10) + b)

[0056] The decoding process is represented by Decoder(*), where σ is the Sigmoid function;

[0057] In step 6, the reconstruction loss between the original features and the reconstructed features is calculated;

[0058] Since the reconstructed features learned cannot be completely identical to the original features, this invention uses the similarity between the two to measure the quality of the reconstruction task. This invention uses the cross-entropy function as a metric, but is not limited to this example.

[0059]

[0060] Therefore, the reconstruction loss is:

[0061]

[0062] In step 7, the manifold structure loss between the temporal features and the manifold structure features is calculated;

[0063] After obtaining the local manifold features and temporal features of the video, the low-dimensional temporal features can be optimized through self-supervised learning, using the manifold features as a supervision signal; therefore, a manifold structure feature loss can be constructed:

[0064]

[0065] In step (8), the original feature set of the entire video frame is traversed sequentially, and the following process is repeated until the model converges:

[0066] (8-1) Calculate the reconstruction loss and manifold structure loss through steps (4) to (7);

[0067] (8-2) Minimize the loss in (8-1) and optimize the parameters in the dynamic temporal network;

[0068] The Adam optimizer is used to optimize the model parameters by retaining the gradients from each run and using backpropagation to continuously optimize the model parameters.

[0069] In step (9), the original feature set of the entire video frame is input into the dynamic temporal network to obtain a low-dimensional, high-quality feature set of the frame;

[0070] By locking the model parameters, the original features X of the video frames are input into the dynamic temporal network and transformed into low-dimensional, high-quality features G.

[0071] In step (10), a clustering algorithm is used to obtain the temporal division corresponding to the video frame;

[0072] This invention uses the K-Means clustering algorithm as an example, but it is not intended to limit the invention; the distance formula used in clustering is:

[0073]

[0074] Where x is any data point, C is a cluster center in the clustering, and D is the feature dimension of x.

[0075] Figure 1 , Figure 2 These are a flowchart and a model diagram of the present invention.

[0076] After the model converges, the Dynamic Temporal Network models the temporal relationships containing manifold structure features from the current video segment. For a given video segment, the Dynamic Temporal Network can output low-dimensional effective features for each frame, and use a clustering algorithm to obtain the unsupervised temporal partitioning results of the video frames.

[0077] Figure 3 The experimental figures are for comparison. ACC, NMI, and ARI are commonly used evaluation metrics in cluster analysis, derived from... Figure 3 As can be seen, the SSMC algorithm of this invention has achieved good performance in multiple first-person perspective datasets.

[0078] Figure 4 This invention provides a visual representation of the temporal features output at different training stages. Figure 4 It can be seen that through continuous iteration, data points belonging to the same category are made as close as possible, while data points belonging to different categories are made as far apart as possible. At the same time, the NMI index is also gradually increasing.

[0079] Figure 5 This is an actual rendering. (By...) Figure 5 As can be seen, in practical applications, this invention achieves truly effective experimental results compared to other algorithms. To a certain extent, it can be used for unsupervised video temporal segmentation tasks in real-world scenarios.

[0080] Those skilled in the art will readily understand the embodiments of the present invention. The above descriptions are merely examples of basic implementations of the present invention and are not intended to limit the invention.

Claims

1.A method for video temporal segmentation of manifold structure, characterized in that: Step (1) sampling the original video according to a preset frame rate to obtain a video frame set with a time sequence relationship, Including: using the average frame sampling method, the video is processed into a video frame set with a time sequence relationship according to a certain frame rate; Step (2) extracts visual features of the video frame set using a ResNet50 neural network pre-trained on the ImageNet dataset to obtain a semantic feature set X = {x1, x2,..., xN}, where N is the total number of video frames. N} Step (3) dividing the semantic feature set X based on the sliding window method, calculating the manifold structure feature of each group of sub-semantic feature set, specifically including: Step (3-1) constructs a sliding window with length L, divides the semantic feature set X into W = {W1, W2, …, W n / L}; Step (3-2) for each sub-feature set W i ={x (i-1)*L+1 x (i-1)*L+2 ,...,x (i-1)*L+L }, generating a sub-feature set W through manifold embedding. i manifold characteristics M i ={m (i-1)*L+1 m (i-1)*L+2 , ..., m (i-1)*L+L }, including: normalized sub-feature set W i The distances between each x and its k nearest neighbors are represented as a fuzzy set of simplexes of the global manifold using spatial transformation; finally, the sub-feature set W is found. i The topological structure is used to construct closely matched low-dimensional vectors, resulting in a sub-feature set W. i manifold characteristics M i ; Step (4) constructing a dynamic time sequence network based on the LSTM network, including: Step (4-1) construction of the near neighbor domain, including: Setting the length of the neighborhood L N and the number of neighbors n, the semantic feature set X is divided into n sets of length L N of neighborhood distances where is the semantic feature x t of the video frame at time t and the set of distances of x N to the semantic features of other video frames within its neighborhood of length L Select the distance sets in ascending order. Contains x t The semantic features of the nearest neighbor frames corresponding to the first n+1 minimum distances constitute the semantic feature set of the nearest neighbor domain subset. Step (4-2) construction of the LSTM time sequence network, including: The nearly neighbor subset semantic feature set is sequentially input into an LSTM network with a time domain length of T+1 to obtain hidden layer outputs {h t-T , h t-T+1 ,..., h t}; Max-pooling and shortcut operation are performed on the output of the hidden layer to obtain the time sequence feature g t = pooling{h t-T , h t-T+1 ,..., h t} + h t (1) Step (5) applies a temporal characteristic g to the reconstructed feature z t input into the decoder to obtain a reconstructed feature z t ; z t = Decoder(g t ) = σ(ω t (g t )+ b) (2) Where the decoding process is represented by Decoder (*), and σ is the Sigmoid function; Step (6) computes the semantic feature x at the t-th time using the cross-entropy function as the metric t and the reconstruction loss between the reconstructed feature z t and the reconstruction loss: Sim(x t , z t ) = H(x t , z t ) + H(l - x t , l - z t ) (3) Therefore, the reconstruction loss is: Step (7) calculating the manifold structure loss between the time sequence feature and the manifold feature, including At time t, the manifold feature M t As a supervision signal, the timing feature g t Optimization, comprising: constructing a manifold structure feature loss: Step (8) traversing the semantic feature set X of the entire video frame in order, repeating the following processes 8-1 and 8-2 until the model converges: Step (8-1) calculates the reconstruction loss through steps (4) to (7) and the manifold structure loss Step (8-2) minimizing the loss in step (8-1) to optimize the parameters in the dynamic time sequence network; Using the Adam optimizer to optimize the parameters of the model, retaining the gradient of each run, in a backpropagation manner, constantly optimizing the model parameters; Step (9) locking the model parameters, inputting the semantic feature set X of the video frame into the dynamic time sequence network to generate the time sequence feature set G; Step (10) using the K-Means clustering algorithm to cluster and divide the time sequence feature set G, and the distance formula used in the clustering is: Where x is the time sequence feature at any time in the time sequence feature set G, C is a cluster center in the clustering, and D is the feature dimension of x.

Images