Self-supervised video representation learning method and device based on temporal restoration task

Abstract

The application relates to a self-supervised video representation learning method and device based on a time sequence restoration task, which comprises the following steps: acquiring a training video and dividing the training video into multiple segments, selecting multiple segments for training, shuffling the order and automatically labeling, and constructing a time sequence restoration task; inputting the segments for training into a backbone network, acquiring primary features corresponding to each segment, inputting the primary features into an encoder based on multi-head attention, and obtaining order perception features of each segment; aggregating the order perception features and inputting the order perception features into a decoder based on a hard attention pointer network, obtaining a restored segment sequence by comparing the time sequence logical relationship between the segments, realizing time sequence restoration, calculating a loss function value according to the restored segment sequence, and updating parameters of the backbone network; repeating the above steps, selecting different segments to train the backbone network, and acquiring a pre-trained backbone network.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a self-supervised video representation learning method and device based on a temporal reconstruction task. Background Technology

[0002] The main purpose of representation learning is to provide general pre-trained models for effective transfer learning in downstream tasks. Generalization is a crucial characteristic pursued by representation learning; stronger generalization indicates that the model has captured more shared information during pre-training, and also suggests that the model may perform well in more downstream tasks. Video representation learning plays a vital role in video understanding and reasoning tasks because videos contain rich spatiotemporal information, and the backbone networks used to extract video features typically have a large number of parameters. Skipping representation learning and training the backbone network from scratch in downstream tasks is relatively difficult. To alleviate the problem of requiring a large amount of manual annotation during pre-training, self-supervised representation learning is used to automatically generate annotations, thereby making more efficient use of large amounts of unlabeled data. Currently, self-supervised video representation learning can be mainly divided into two categories: one is based on contrastive learning, which learns the feature space by defining positive and negative samples; the other is based on surrogate tasks, which endows the backbone network with the ability to understand spatiotemporal information by solving tasks. Contrastive learning-based methods usually require dividing positive and negative samples according to an assumed distinguishing boundary, and then designing corresponding data augmentation methods to preserve the invariant properties learned while diversifying the variable properties. In contrast, proxy task-based methods do not artificially define positive and negative samples or design data augmentation. Instead, they directly utilize the natural characteristics of the data, such as the natural playback speed of a video or the natural temporal order of segments. These natural characteristics are transformed and then used for prediction as the pre-training task. Furthermore, to improve pre-training efficiency, the concept of course learning is adopted to divide pre-training into multiple stages of varying difficulty. Different levels of difficulty are set at different stages by gradually increasing the amount of data or the difficulty of the samples. After pre-training is completed, the effectiveness of the pre-training is typically validated on downstream tasks such as action recognition and nearest neighbor retrieval.

[0003] While contrastive learning-based methods have demonstrated superior performance, they still require manually defining positive and negative samples and designing corresponding data augmentation methods based on the characteristics of the dataset. In contrast, surrogate task-based methods can directly utilize the natural characteristics of the data without further assumptions about its distribution. Although surrogate task-based methods further reduce the influence of subjective factors on pre-training, their insufficient utilization of unlabeled data and inability to directly learn high-discretionary features hinder further performance improvements. Currently, surrogate tasks are often formulated as single-stage classification problems, lacking a reasonable definition of the distance between different classes. The modules used for further information processing and task completion typically consist of simple concatenation operations and linear transformations, making it difficult to efficiently establish relationships between segments. Therefore, improving the efficiency of the spatiotemporal logic of task modeling and effectively organizing pre-training are key to enhancing the performance of surrogate task-based methods. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art by providing a self-supervised video representation learning method and device based on temporal reconstruction tasks, so as to achieve efficient utilization of temporal characteristics.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] One aspect of the present invention provides a self-supervised video representation learning method based on a temporal reconstruction task, comprising the following steps:

[0007] Step S1: Obtain the training video and divide it into multiple segments. Select multiple segments for training, shuffle their order, and automatically label them to construct a time series restoration task.

[0008] Step S2: Input the segments used for training into the backbone network to obtain the primary features corresponding to each segment, and input the primary features into the multi-head attention-based encoder to obtain the sequential perception features of each segment.

[0009] Step S3: Aggregate the sequence-aware features and input them into the decoder based on the hard attention pointer network. By comparing the temporal logical relationships between segments, the restored segment sequence is obtained to achieve temporal restoration. The loss function value is calculated based on the restored segment sequence and the parameters of the backbone network are updated.

[0010] Step S4: Repeat steps S1-S3, select different segments to train the backbone network, and obtain the pre-trained backbone network.

[0011] As a preferred technical solution, in step S3, according to a preset pre-training strategy, the backbone network is trained in stages with different training difficulties by changing the number of segments to be divided and / or changing the number of segments used for training and / or adding a non-learning mask vector to the decoder to exclude the restored segments.

[0012] As a preferred technical solution, in step S3, the process of the decoder parsing the aggregated sequential sensing features is specifically as follows:

[0013] Based on the aggregated sequential awareness features, the first restored segment is predicted, and the decoder hidden state is updated.

[0014] The primary features corresponding to the previous segment are input into the hard attention pointer network to update the decoder hidden state. The next restored segment is predicted based on the similarity between the sequential awareness features and the hidden state. This step is repeated to obtain the restored segment sequence.

[0015] As a preferred technical solution, in step S3, after predicting each restored segment, the cross-entropy function is used to calculate the sub-loss, and finally the average of all sub-losses is taken as the loss function value.

[0016] As a preferred technical solution, the backbone network is an R3D network, a C3D network, or an R(2+1)D network.

[0017] As a preferred technical solution, step S1 specifically includes:

[0018] The training video is divided into non-overlapping segments along the timeline. Multiple segments are selected for training, their order is shuffled, and they are automatically labeled.

[0019] As a preferred technical solution, the self-supervised video representation learning method further includes:

[0020] Step S5: Transfer the pre-trained backbone network to downstream tasks, wherein the downstream tasks include at least one of action recognition, nearest neighbor retrieval, and video description.

[0021] As a preferred technical solution, for action recognition tasks, transfer is achieved by using a pre-trained backbone network as a feature extractor and fine-tuning its parameters; for nearest neighbor retrieval, transfer is achieved by freezing the parameters of the pre-trained backbone network and directly using it as a feature extractor for retrieval based on similarity; for video description, supplementary features are pre-extracted using a pre-trained backbone network, and then the supplementary features are combined with existing video description methods in a feature-level lightweight fusion manner to achieve transfer.

[0022] In another aspect, an electronic device is provided, comprising: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the above-described self-supervised video representation learning method based on a temporal restoration task.

[0023] In another aspect, the present invention provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing the above-described self-supervised video representation learning method based on a temporal restoration task.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] (1) High efficiency in utilizing temporal characteristics: In response to the problem that pre-trained models cannot fully capture temporal characteristics in action recognition, nearest neighbor retrieval and video description, unlike previous proxy tasks that utilize video temporal characteristics for pre-training, this invention aggregates sequential awareness features and inputs them into a decoder based on a hard attention pointer network during pre-training to obtain the restored segment sequence, thereby achieving temporal restoration. The provided temporal restoration task utilizes video temporal characteristics in multiple steps, gradually accumulating spatiotemporal clues to restore the disordered segment sequence. During the restoration process, the temporal logical relationship between segments is repeatedly compared, thereby achieving efficient utilization of temporal characteristics, which is also proven by the better results shown in downstream tasks.

[0026] (2) The pre-training strategy has strong scalability: The temporal restoration task and the task network used to solve the task provided by this invention have good flexibility. The encoder based on multi-head attention can handle different numbers of segments and establish temporal relationships between segments, while the decoder based on hard attention pointer network can restore different numbers of segments by adjusting only a few parameter layers. It can also adjust the difficulty of the pre-training task through the masking mechanism. Therefore, this invention can be used to explore pre-training strategies extensively and find effective pre-training modes.

[0027] (3) Strong generalization of the pre-trained model: The temporal restoration task provided by this invention utilizes the natural temporal characteristics of video data, enabling the backbone network to capture effective spatiotemporal logic during the pre-training stage. It is widely applicable to various video datasets and tasks, and has shown competitive results in downstream tasks such as action recognition and nearest neighbor retrieval, which are commonly used to verify the pre-training effect. In addition, the pre-training method proposed in this invention performs well in cross-modal video description tasks. By embedding existing video description methods through early feature fusion, it supplements spatiotemporal logic information and achieves general performance improvement on different datasets and metrics.

[0028] (4) No need for pre-labeling: When selecting multiple segments for training, the video is first evenly divided into non-overlapping segments along the time axis. Then, a certain number of segments are sampled and their order is shuffled. The position of the segment to be restored in each step is obtained according to the original order. This is used as the real label for the task, thus achieving automatic labeling without the need for manual labeling of the training segments in advance. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the timing restoration task and task network in the embodiment;

[0030] Figure 2 This is a schematic diagram of the three pre-training strategies provided in the embodiment. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] Example 1

[0033] This embodiment provides a self-supervised video representation learning method based on a temporal reconstruction task. Benefiting from the natural temporal order of different segments in a video, this method uses the reconstruction of disordered segment sequences as a self-supervised proxy task. During the pre-training phase, it efficiently models the relationships and ordering cues between segments, thereby endowing the backbone network with the ability to understand the spatiotemporal logic of the video and transferring this ability to various downstream tasks. Furthermore, since the temporal reconstruction task and its corresponding task network have good scalability, various curriculum-based pre-training strategies are classified and used accordingly. The self-supervised pre-training is further strengthened by varying the difficulty of the task. Specifically, the primary features extracted by the backbone network are first encoded with sequence information by a sequence awareness module. Then, the decoder reconstructs the correctly arranged segment sequence step by step, segment by segment. Finally, the spatiotemporal logic of the video is mined through the temporal reconstruction task. Furthermore, the dataset size, the number of segment samples, and the task network structure can be flexibly changed to use different pre-training strategies.

[0034] Self-supervised video representation learning mainly includes the following steps:

[0035] (1) Pre-training task formulation: Automatically generate the required annotations from the data, and pre-train the backbone network by solving the task. The basic assumption of the solution is that only with a full understanding of the spatiotemporal logic of the video can we further reason and solve the proxy task. Under the premise of zero manual annotation, formulate a task that can automatically generate annotations and perform pre-training. The pre-training task has the ability to capture the spatiotemporal logic of the video.

[0036] (2) Primary feature processing: The primary features extracted from the backbone network are processed to complete the pre-training task. The processing flow includes feature correlation modeling and global compact feature extraction. The specific processing details need to be adapted to the pre-training task. This invention uses a specially designed task network to process the primary features.

[0037] (3) Pre-training strategy design: Following the concept of solving tasks from easy to difficult, we designed a phased pre-training approach to further explore the potential of pre-training tasks and primary feature processing.

[0038] (4) Performance evaluation: The pre-trained backbone network is transferred to the downstream task for evaluation. There are three ways to transfer it: first, freeze the pre-trained backbone network and use it directly as the only feature extractor; second, continue to fine-tune the backbone network in the downstream task; and third, use the pre-trained backbone network to extract auxiliary information and embed it into the existing model framework in the form of feature-level fusion.

[0039] Specifically, the self-supervised video representation learning method based on the temporal reconstruction task includes the following steps:

[0040] Step S1, Pre-training the proxy task: Automatically generate the annotations required for the task from the data, and pre-train the backbone network by solving the task. The basic assumption of the solution is that only with a full understanding of the spatiotemporal logic of the video can we further reason and solve the proxy task.

[0041] Specifically, the pre-trained agent task is as follows: First, the video is divided into non-overlapping segments along the timeline. Then, some segments are sampled and their order is shuffled. When restoring the sequence, a segment-by-segment and step-by-step restoration method is adopted. Based on the restored segments, only one segment is restored in each step, and then the segment is added to the restored segments to accumulate clues as the basis for subsequent restoration. This process is repeated until a complete restored sequence is obtained.

[0042] To restore a scrambled sequence of video clips, the video is first divided into non-overlapping segments along the timeline. Then, a certain number of segments are sampled and shuffled. The position of the segment to be restored in each step is obtained from the original order, and this position is used as the true label for the task. The restoration process is carried out in multiple steps. Each step restores the current segment based on the previously restored segments and uses this segment as the input for the next step to predict the next segment to be restored. This process is repeated to gradually restore the complete sequence.

[0043] Step S2, Task Network: A specially designed module for processing primary features. It first receives the primary features extracted by the backbone network as input, passes them through an encoder based on multi-head attention and one-dimensional convolution, and then inputs them into a decoder based on a hard attention pointer network to generatively and progressively restore the disordered fragment sequence.

[0044] Specifically, the encoder of the task network directly processes the primary features. First, it receives the primary features extracted by the backbone network as input, and then passes them through the sequential perception encoder based on the attention mechanism, allowing the information of the shuffled segments to flow between each other to obtain sequential perception features. Then, a one-dimensional convolution is applied to these features to obtain global sequential perception features, which serve as the initial input to the decoder.

[0045] The encoder transforms initial features into sequence-aware features using a deep self-attention network. Then, it further aggregates these sequence-aware features using one-dimensional convolutions to obtain compact global sequence-aware features, which serve as the encoder's initial input. The deep self-attention network does not include positional encoding because the input segments are out of order and require no prior information about the sequence. The one-dimensional convolutions move along the length axis of the sequence-aware features, weighted summing the elements at the same position for each feature to obtain the element at that position in the global sequence-aware feature. The deep self-attention network consists of multiple layers, each layer further comprising a self-attention sublayer, a layer normalization sublayer, and a feedforward sublayer.

[0046] Step S3, Pre-training strategy: Drawing on the concept of course learning, pre-training can be completed in multiple stages, with tasks of varying difficulty completed at different stages. This approach makes organizing pre-training more reasonable and can further optimize the learning ability of pre-training.

[0047] Specifically, the decoder of the task network works as follows: The decoder first accepts global sequential awareness features as initial input, predicts the first restored segment, and in subsequent steps, it iteratively inputs the primary features corresponding to the restored segment of the previous step and calculates the decoder hidden state. Then, it predicts the segment to be restored in this step by calculating the similarity between the sequential awareness features and the hidden state. The hidden state of the decoder is continuously updated using a long short-term memory network to obtain the restored state of each step.

[0048] The decoder employs a recurrent structure based on a Long Short-Term Memory (LSTM) network. The first step input is the global sequence-aware feature output from the encoder. After passing through the LSM unit, the hidden state is updated. The hidden state and sequence-aware feature are then input into the hard attention pointer network, pointing to the segment in the disordered sequence that should be restored in this step. In subsequent steps, the primary features corresponding to the restored segments from the previous step are extracted by the backbone network. Similarly, the hidden state is updated, and the segment to be restored in this step is predicted. Specifically, the hard attention pointer network first performs a linear transformation on the hidden state and sequence-aware feature. Then, it calculates the cosine similarity between the transformed hidden state and each transformed sequence-aware feature. After normalizing the similarity, the probability of each segment in the disordered sequence being restored is obtained, and the segment with the highest probability is restored.

[0049] Furthermore, each step of the restoration includes a sub-loss using the cross-entropy loss function, while the sequence-level total loss function is the sum of the losses over all steps and then the average, used to measure the degree of deviation between different restoration results and the true label.

[0050] Specifically, the pre-training strategy is divided into three categories based on the factors affecting the difficulty of pre-training: task-based strategy, model-based strategy, and data-based strategy. The task-based strategy changes the difficulty of pre-training by changing the number of sampling segments, the model-based strategy adds non-learning mask vectors to the model to reduce the difficulty of the task, and the data-based strategy gradually increases the difficulty by controlling the size of the pre-training dataset from small to large.

[0051] Based on the concept of curriculum-based learning, pre-training strategies divide pre-training into multiple stages with varying training difficulty. These strategies are categorized into three types according to factors influencing training difficulty: task-based strategies, model-based strategies, and data-based strategies. Task-based strategies involve varying numbers of segments to be restored in different pre-training stages; the more segments to be restored, the greater the difficulty. Model-based strategies use different task networks in different pre-training stages. One type of network directly predicts the restored segments using the probabilities output by a hard attention pointer network, while another adds a non-learned mask vector to exclude already restored segments, thus reducing pre-training difficulty. Data-based strategies first train on a relatively small dataset, then train on a larger dataset.

[0052] Step S4, Pre-trained model transfer and performance evaluation: Transfer the pre-trained backbone network to downstream tasks in different ways, and decide whether to fine-tune the backbone network and how to complete the corresponding downstream task through the primary features and what structure.

[0053] Specifically, for action recognition tasks, the pre-trained backbone network is used as a feature extractor and its parameters are further fine-tuned; for nearest neighbor retrieval tasks, the parameters of the pre-trained backbone network are frozen and directly used as a feature extractor for retrieval based on similarity; and for video description tasks, supplementary features are pre-extracted using the pre-trained backbone network, and then the supplementary features are combined with existing video description methods in a feature-level lightweight fusion manner.

[0054] In this embodiment, the performance of pre-training was evaluated using three different transfer models across three downstream tasks. In the action recognition task, the pre-trained backbone network was additionally connected to a fully connected layer for fine-tuning. In the nearest neighbor retrieval task, the parameters of the pre-trained backbone network were frozen and directly used to extract features, which were then used to calculate similarity as a retrieval basis. In the video description task, the parameters of the pre-trained backbone network were also frozen, and the extracted features were used as supplementary information, fused with the existing video description framework using a feature-level weighted summation.

[0055] In a typical implementation, see Figure 1 Self-supervised video representation learning methods include the following:

[0056] S1, the pre-trained agent task, firstly divides the video V evenly into multiple segments, where each segment consists of 16 consecutive frames, with an 8-frame interval between any two adjacent segments; after the division, n segments are sampled to complete the temporal restoration task, and the n segments are arranged in the original order in the video as C1, C2, ..., C n Disordering them yields a sequence of disordered fragments. Where i1, i2, ..., i n It is a permutation of 1, 2, ..., n; next, the sequence order needs to be identified by restoring the disordered fragment sequence to complete the pre-training. The restoration is divided into n steps. In each step, a fragment is selected from the disordered sequence as the fragment to be restored in that step. This process is repeated until the original order of the sequence is gradually restored. Specifically, in the t-th step (1≤t≤n), the task network predicts the position p of the restored fragment in the disordered fragment. t ,Right now This step restores the sequence; therefore, the final restored fragment sequence is... During pre-training, the supervision signal satisfies i pt =t.

[0057] The encoder of the task network (S2, 2) aims to further process the initial features extracted by the skeleton network, allowing information to flow between different segments and obtaining sequence-aware features for subsequent segment reconstruction. The calculation process is represented as follows:

[0058]

[0059] in The term "Engineer" represents the backbone network for extracting initial features. It can be one of three convolutional neural networks: R3D, C3D, or R(2+1)D. This invention investigated all three backbone networks. C3D is the most basic type of 3D convolutional neural network, mainly composed of multiple 3D convolutional layers, 3D pooling layers, activation functions, and fully connected layers connected sequentially. R3D adds residual connections to C3D. R(2+1)D further decomposes the 3D convolutional layers into 2D and 1D convolutional layers, and the 3D pooling layers into 2D and 1D pooling layers, thereby reducing the number of parameters and mitigating overfitting. ENC represents a sequential sensing encoder. This is a sequence-aware feature. The main structure of ENC is a Transformer. Furthermore, the Transformer is composed of stacked Transformer layers with the same structure. Each layer consists of a self-attention sub-layer, a feedforward sub-layer, and a layer normalization module. ENC does not contain positional encoding because the input segment sequence is out of order.

[0060] The sequence-aware features corresponding to each segment are aggregated to obtain the global sequence-aware features. The calculation process is as follows:

[0061]

[0062] Where conv represents a one-dimensional convolution with a kernel size of 1×n that moves along the feature length axis; x glb The calculated global sequence-aware features will then be used as the initial input to the encoder to initiate the restoration process.

[0063] S3, the decoder of the task network, aims to gradually reconstruct the fragmented sequence and accumulate spatiotemporal cues during the reconstruction process, giving the skeleton network the ability to understand spatiotemporal logic. The computation process is represented as follows:

[0064]

[0065] Here, DEC represents the sequence restoration decoder, and its main structure is LSTM. The initial features extracted by the backbone network, and p t (1≤t≤n) represents the p-th element in the disordered sequence at step t. t A segment (i.e.) It was restored.

[0066] Specifically, the LSTM-based sequence reconstruction decoder uses either the global sequential awareness features (t=1) or the initial features corresponding to the reconstructed segment from the previous step (2≤t≤n) as input at step t. The calculation process is as follows:

[0067]

[0068] Where h t-1 and h t Let i represent the hidden states of the LSTM at times t-1 and t, respectively. prev This represents the position of the restored fragment in the disordered sequence at time t-1, i.e. Calculated by the Hard Attention Pointer (HAP) module, the input is the hidden state of the LSTM at this time and the sequential perceptual features of all segments. This process is represented as:

[0069]

[0070] Specifically, HAP calculates the reconstruction confidence of each segment in the disordered sequence at each step, and then selects the segment with the highest confidence as the segment to be reconstructed in this step. In step t-1, this process is represented as follows:

[0071]

[0072]

[0073] Where u is a learnable vector, and W1 and W2 are both learnable matrices. The learnable parameters are shared in all steps, and J = {1,2,…,n}.

[0074] S4, the pre-training strategy, aims to fully utilize the designed agent task and task network, referencing the "difficult to easy" learning model in the course, to further explore possibilities for improving pre-training performance. First, based on the factors affecting pre-training difficulty, the strategies are classified into three categories: task-based strategies (π). T Model-based strategy π M and data-based strategies π D , respectively Figure 2 (a) Figure 2 (b) and Figure 2 As shown in (c). Among these, the task-based strategy adjusts the pre-training difficulty by varying the number of restored segments, the model-based strategy uses non-learning masks to assist the model in reconstructing sequences, and the data-based strategy achieves course learning by increasing the dataset size. The pattern of these strategies is to first solve a basic temporal sequence reconstruction task to obtain the backbone network parameters θ. p Then through strategy π p Guided by the course of (p∈{T,M,D}), further solve the task and obtain θ′. p This allows for the enhancement of the representation of video data.

[0075] S5, the fusion of pre-trained features with existing video description methods, aims to leverage temporal reconstruction tasks to model the spatiotemporal logic of video data, extract supplementary features from a general dataset, and enhance the feature representation of existing video description methods through early feature-level fusion. Specifically, its fusion method can be represented as follows:

[0076] f 2D :f 2D +λ1ReLU(Af ss )

[0077] f 3D :=f 3D +λ2ReLU(Af ss )

[0078] Among them, f 2D and f 3D f represents the 2D and 3D features input into the video description model. ss λ1 and λ2 represent the features extracted by the backbone network after self-supervised pre-training, respectively, ReLU represents the corrected linear unit, and A represents the projection matrix.

[0079] S6. To verify the performance of the pre-trained agent task described above, the following experiment was designed.

[0080] Similar to other related methods, this approach uses action recognition and nearest neighbor retrieval as downstream tasks to validate the performance of temporal reconstruction tasks, and additionally adds video description to evaluate the performance of cross-modal understanding tasks. The classic benchmark datasets UCF101 and HMDB51 were used for validation of both action recognition and nearest neighbor retrieval tasks, while the classic benchmark datasets MSVD and MSR-VTT were used for video description tasks. The metric for the action recognition task is recognition accuracy (hereinafter referred to as Acc), and the results are shown in Table 1. The metric for the nearest neighbor retrieval task is topk (k∈{1,5,10,20,50}), representing the hit rate of k candidate segments containing the target category, and the results are shown in Tables 2 and 3. For the video description task, four commonly used metrics were used: M (METEOR), B@4 (BLEU@4), C (CIDEr), and R (ROUGE-L), and the results are shown in Tables 4 and 5. The SRSC method of this invention was compared with the latest state-of-the-art self-supervised learning methods based on agent tasks. For action recognition and nearest neighbor retrieval tasks, the best-performing C3D, R3D and R(2+1)D backbone networks of each method were used for comparison and evaluation. For video description tasks, all three backbone networks were used to verify the enhancement effect of the features extracted by the method of this invention on existing video description models.

[0081] See Table 1, which shows the accuracy of action recognition on the UCF101 and HMDB51 datasets, the results of the SRSC method of this invention and the state-of-the-art self-supervised video representation learning based on agent tasks, where the * indicates the results on UCF101 split 1, and the rest without * indicate the average results on split 1, 2 and 3.

[0082] Table 1. Accuracy of action recognition on the UCF101 and HMDB51 datasets.

[0083]

[0084] See Table 2 for the hit rate of nearest neighbor retrieval on the UCF101 dataset, and the results of the SRSC method of this invention and the state-of-the-art agent-based self-supervised video representation learning.

[0085] Table 2 shows the hit rate of nearest neighbor retrieval on the UCF101 dataset.

[0086]

[0087] See Table 3 for the hit rate of nearest neighbor retrieval on the HMDB51 dataset, and the results of the SRSC method of this invention and state-of-the-art agent-based self-supervised video representation learning.

[0088] Table 3 shows the hit rate of nearest neighbor retrieval on the HMDB51 dataset.

[0089]

[0090]

[0091] See Table 4 for the results of fusing the pre-trained features extracted by the SRSC method of this invention with existing video description methods on the MSVD dataset.

[0092] Table 4. Results of fusing pre-trained features extracted by SRSC with existing video description methods.

[0093]

[0094] See Table 5 for the results of fusing the pre-trained features extracted by the SRSC method of this invention with existing video description methods on the MSR-VTT dataset.

[0095] Table 5. Results of fusing pre-trained features extracted by SRSC with existing video description methods.

[0096]

[0097]

[0098] (1) Results of downstream action recognition tasks: Table 1 shows the comparison results between our method and state-of-the-art self-supervised learning methods based on surrogate tasks on the UCF101 and HMDB51 datasets. Since existing surrogate tasks are insufficient in modeling relationships between segments and lack the ability to accumulate cues for spatiotemporal logical reasoning, this paper designs a temporal reconstruction surrogate task and a task network for solving it. The task network consists of an encoder that perceives the order and a decoder that progressively reconstructs the sequence. Based on the action recognition accuracy shown in Table 1, our method outperforms existing methods on this downstream action recognition task.

[0099] (2) Results of downstream tasks in nearest neighbor retrieval: The comparison results of this method with state-of-the-art self-supervised learning methods based on surrogate tasks on the UCF101 and HMDB51 datasets are shown in Tables 2 and 3, respectively. Since existing surrogate tasks are inefficient and insufficient in calculating the relationships between fragments, it is difficult to verify the performance of more than 3 fragments. This method can flexibly expand the number of fragments and effectively model the fragments using task networks. Through exploration, it was found that setting the number of fragments to 4 results in better performance on the nearest neighbor retrieval task. The results on UCF101 shown in Table 2 show a significant improvement compared to other existing methods. The results on HMDB51 shown in Table 3 also demonstrate the competitive performance of the method of this invention.

[0100] (3) Results of downstream tasks in video description: Based on the state-of-the-art video description methods, this method embeds self-supervised pre-trained features through early feature fusion. Since the temporal characteristics of videos are common to all video datasets, and this method has excellent modeling capabilities for video spatiotemporal logic, it also performs well in extracting supplementary features and enhancing video representation for cross-modal understanding tasks using pre-trained backbone networks. The performance of video description tasks is shown in Tables 4 and 5. It can be seen that for the MSVD dataset, the three pre-trained backbone networks have improved compared to the original video description methods. For the MSR-VTT dataset, the three pre-trained backbone networks have also improved compared to the original video description methods in most metrics, with almost no performance degradation.

[0101] S7. To verify the effectiveness of the above pre-training strategy, the following experiment was designed.

[0102] The three pre-training strategies proposed in this invention were evaluated on downstream tasks of action recognition and nearest neighbor retrieval. Both downstream tasks were performed on the UCF101 dataset. The performance gain after using the pre-training strategies was used as the metric, and the gain comparison was based on the benchmark pre-training, which solves a single temporal reconstruction task without involving any course learning. The results are shown in Table 6. In the task-based strategy π... TIn the process, the validation for action recognition uses 3 segments as the baseline pre-training and 6 segments as the relay task, while the validation for nearest neighbor retrieval uses 4 segments as the baseline pre-training and 5 segments as the relay task; in the model-based policy π M In the action recognition validation, the number of segments is set to 3 as the baseline pre-training, and the number of segments is set to 5 using a masking strategy as a relay task. In the nearest neighbor retrieval validation, the number of segments is set to 4, and a masking strategy is used as the baseline pre-training, while the number of segments is set to 5 as a relay task. In the data-based strategy π... D In both cases, the number of segments was set to 3 as the baseline for pre-training. For the relay task dataset, Kinetics400 was used, with the number of segments remaining at 3. According to the results shown in Table 6, effective settings for further improving pre-training performance can be found for all three pre-training strategies.

[0103] Table 6 validates the curriculum-based pre-training strategy on the UCF101 dataset.

[0104]

[0105] Compared with existing technologies, this invention redefines the segment sequence prediction task, fully considers the differences between different sequence orders, can make more efficient and reasonable use of the temporal characteristics of videos, and utilizes the flexibility of temporal reconstruction tasks and task networks, and designs a variety of pre-training strategies based on course learning, which has the advantages of strong scalability and high computational efficiency.

[0106] Example 2

[0107] This embodiment provides an action recognition method that uses the skeleton network obtained in Embodiment 1 to achieve the recognition task through transfer learning. The goal of action recognition is to identify the actions that appear in the video.

[0108] Specifically, this method first uses a pre-trained backbone network as a feature extractor, and then adds an additional fully connected layer as a classification head to output a classification score. All parameters are fine-tuned using cross-entropy loss as the loss function to build an action recognition model. After the fine-tuning stage is completed, the video to be recognized can be obtained and input into the action recognition model. The model outputs the probability of each type of action appearing in the video, and the category with the highest probability is selected. The label corresponding to this category is the prediction result, and finally, action recognition is achieved.

[0109] Example 3

[0110] This embodiment provides a nearest neighbor retrieval method, which uses the skeleton network obtained in Embodiment 1 to achieve the retrieval task through transfer learning. The goal of nearest neighbor retrieval is to retrieve segments that have the same actions as the query segment.

[0111] Specifically, this method freezes the parameters of the pre-trained backbone network and directly uses the last pooling layer as a feature extractor to construct a nearest neighbor retrieval model. First, features of the query segment and the queried segment are extracted. The similarity between the query segment and the queried segment is calculated using the obtained features and cosine similarity. Then, the queried segments are sorted from high to low based on similarity. Finally, the top k segments are selected as the retrieval results based on a pre-set number of retrievals, k, thus achieving nearest neighbor retrieval.

[0112] Example 4

[0113] This embodiment provides a method for enhancing a video description framework. Based on the backbone network obtained in Embodiment 1, features of video segments are extracted and the extracted features are fused into an existing video description framework to enhance the video representation capability of the original framework.

[0114] Specifically, features of the target video segment are pre-extracted using a pre-trained backbone network. These features are then used as supplementary features and combined with existing video description methods using a feature-level lightweight fusion approach. In the video description task, the parameters of the pre-trained backbone network are also frozen and used to extract supplementary features from the target video. Since the backbone network used to extract these features was used to solve the temporal reconstruction task during pre-training, these features contain rich spatiotemporal logical information. As supplementary features to the existing video description framework, they are weighted and added to the original features to update the original features, which are then re-input into the existing video description framework.

[0115] Example 2

[0116] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the self-supervised video representation learning method based on the temporal restoration task as described in Embodiment 1.

[0117] Example 3

[0118] This embodiment provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing a self-supervised video representation learning method based on a temporal restoration task as described in Embodiment 1.

[0119] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A self-supervised video representation learning method based on a temporal restoration task, characterized in that, The method comprises the following steps: Step S1, obtaining a training video and dividing it into multiple segments, selecting multiple segments for training, shuffling the order and performing automatic labeling to construct a temporal restoration task; Step S2, inputting the segments for training into a backbone network to obtain primary features corresponding to each segment, inputting the primary features into an encoder based on multi-head attention to obtain sequence perception features of each segment; Step S3, aggregating and inputting the sequence perception features into a decoder based on a hard attention pointer network, obtaining a restored segment sequence by comparing the temporal logical relationship between the segments, realizing temporal restoration, calculating a loss function value according to the restored segment sequence, and updating parameters of the backbone network; Step S4, repeating steps S1-S3 to select different segments to train the backbone network, and obtaining a pre-trained backbone network, In step S3, the process of the decoder analyzing the aggregated sequence perception features is specifically: Based on the aggregated sequence perception features, the first restored segment is predicted, and the decoder hidden state is updated; The primary feature corresponding to the previous segment is input into the hard attention pointer network, the decoder hidden state is updated, the next restored segment is predicted based on the similarity between the sequence perception features and the hidden state, and the step is repeated to obtain the restored segment sequence. 2.The method of claim 1, wherein, In step S3, according to a preset pre-training strategy, the number of divided segments and / or the number of segments for training are changed and / or a non-learning mask vector for excluding restored segments is added in the decoder to train the backbone network in different training difficulties in stages. 3.The method of claim 1, wherein, In step S3, a cross-entropy function is used to calculate a sub-loss after predicting each restored segment, and finally all sub-losses are averaged as the loss function value.

4. The self-supervised video representation learning method based on a timing recovery task according to claim 1, characterized in that, The backbone network is an R3D network, a C3D network or an R(2+1)D network.

5. The self-supervised video representation learning method based on timing recovery tasks according to claim 1, characterized in that, Step S1 is specifically: Divide the training video into non-overlapping segments along the time axis, select multiple segments for training, shuffle the order and perform automatic labeling.

6. The self-supervised video representation learning method based on timing recovery tasks according to claim 1, characterized in that, The self-supervised video representation learning method further comprises: Step S5, migrating the pre-trained backbone network to a downstream task, wherein the downstream task comprises at least one of action recognition, nearest neighbor retrieval and video description.

7. The self-supervised video representation learning method based on a timing recovery task according to claim 6, characterized in that, For the action recognition task, the pre-trained backbone network is used as a feature extractor and the parameters are fine-tuned to realize migration; for the nearest neighbor retrieval, the pre-trained backbone network parameters are frozen and directly used as a feature extractor and retrieved according to the similarity to realize migration; for the video description, the pre-trained backbone network is used to extract supplementary features in advance, and then the supplementary features are combined with existing video description methods in a feature-level lightweight fusion manner to realize migration.

8. An electronic device, comprising: It comprises: One or more processors and a memory, the memory storing one or more programs, the one or more programs comprising instructions for executing the self-supervised video representation learning method based on the temporal restoration task according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, One or more programs for execution by one or more processors of an electronic device, the one or more programs including instructions for performing a self-supervised video representation learning method based on a timing recovery task as claimed in any of claims 1-7.

Images