Temporal contrast graph learning system and method for self-supervised video representation learning

Abstract

The present invention provides a temporal contrast graph learning system and method for self-supervised video representation learning. The system introduces a spatiotemporal knowledge discovery module (STKD), which extracts motion-enhanced spatiotemporal representations from videos based on frequency domain analysis of discrete cosine transforms. In order to explicitly construct a multi-scale temporal correlation model for unlabeled videos, prior knowledge about the order of frames and segments is integrated into a graph structure, namely, an intra-segment / inter-segment temporal contrast graph (TCG). Then, a specific temporal contrast graph learning module is designed to maximize the consistency between nodes in different graph views. In order to generate monitoring signals for unlabeled videos, an adaptive sequence prediction module is introduced, which uses the correlation knowledge between video segments to learn global context representations and adaptively recalibrate channel features.

Description

Technical Field

[0001] The present invention relates to the technical field of video representation learning, and more specifically, to a time domain contrast graph learning system and method for self-supervised video representation learning. Background Art

[0002] For video representation learning, a large number of supervised learning methods have received increasing attention. These methods include traditional methods and deep learning methods. Although these supervised methods have achieved good performance in building temporal correlation models, they require a large number of labeled videos to train complex models, which is undoubtedly time-consuming and laborious.

[0003] Self-supervised learning provides a feasible way to generate supervisory signals by modeling various proxy tasks using a large amount of unlabeled data. The models learned from proxy tasks can be directly applied to downstream tasks for feature extraction or fine-tuning. It requires the model to have strong expressive power to exploit rich spatiotemporal knowledge and generate effective supervisory signals from a large amount of unlabeled videos. However, existing methods fail to increase the temporal diversity of unlabeled videos and ignore the fine construction of multi-scale temporal correlation models in an explicit manner.

[0004] Unlike image data, which can be processed by defining proxy tasks for self-supervised learning, video data additionally contains temporal information that can be used to learn supervisory signals. Recently, a variety of methods have been proposed. Shuffle & Learn randomly shuffles video frames and trains the network to distinguish whether the order of these video frames is correct. Odd-one-out network proposes to identify irrelevant or odd video clips. Sequence Prediction Network (OPN) trains a network to predict the correct order of shuffled frames. VCOP uses a 3D convolutional network to predict the order of shuffled video clips. Video-pace uses a network to identify the correct pace of different video clips. SpeedNet designs a network to detect whether a video is played at a normal rate or accelerated. In addition to focusing on temporal correlation, Mas also proposed a self-supervised learning method by regressing motion and appearance statistics along spatial and temporal dimensions. ST-puzzle uses space-time cubic puzzles to design proxy tasks. IIC introduces negative samples by breaking the temporal relationship in video clips and uses these samples to build an internal contrast framework. XDC proposes a self-supervised approach that leverages unsupervised clustering in the audio modality as a supervisory signal for the video modality.

[0005] like Figure 1 Figure 1 shows a graphical representation of multi-scale temporal correlations. The handshake contains long-term (inter-segment) temporal correlations of forward walking, handshake, and hug, as well as short-term (intra-segment) temporal correlations of periodic hand and foot movements.

[0006] However, these methods only consider the temporal correlation of a single scale (i.e., short-term or long-term) and ignore the multi-scale temporal correlation, that is, they extract segment-level or frame-level features through two-dimensional / three-dimensional convolutional neural networks (2D / 3DCNN) but ignore integrating these features to model the complex multi-scale temporal correlation.

[0007] like Figure 2 Figure 1 shows a diagram of video frames with different frequencies. The first row is the original frame; the second row is the low frequency (focusing on the scene representation); the last row is the high frequency (focusing on different motion edges).

[0008] Frequency analysis has always been a powerful tool in the field of signal processing. Recently, several frequency analysis methods have been proposed in the field of deep learning. Most frequency-based methods aim to reduce the computational cost and parameters through Fourier transform (FT), thereby improving network efficiency. Some works have introduced frequency analysis of Joint Photographic Experts Group (JPEG) coding in convolutional neural networks (CNN). Discrete cosine transform (DCT) was introduced in a paper to reduce communication bandwidth. In some works, dedicated autoencoder-based networks for compression and inference tasks have been proposed. FcaNet generalizes the preprocessing of channel attention mechanism in frequency domain. Wang et al. applied the analysis of the connection between the distribution of frequency components in the input dataset to interpret CNN. Since the convolution operation in the spatial domain has been shown to be equivalent to multiplication in the frequency domain, video knowledge distillation is performed in the frequency domain for action recognition. Frequency Filter Embedding (FFE) uses graph Fourier transform and frequency filtering as graph Fourier domain operators to extract graph features. The method is different from previous works in two aspects. First, discriminative spatiotemporal knowledge in the frequency domain is discovered for video self-supervised learning. Second, a rigorous theoretical analysis is performed to verify that the discriminative spatiotemporal representation intrinsically highlights motion information in the high-frequency domain.

[0009] In the prior art, there is a patent that discloses a single-shot time-domain contrast measurement device for nanosecond time window laser pulses with time-spectral encoding, including a beam splitting module, a reference light generation or sampling module, a reference light delay module, a narrow-band filtering device for the light to be measured, a sampling module for the light to be measured, a nonlinear action module such as sum frequency, a spectrum intensity attenuation module, and a spectrum intensity detection device. The patent uses chirped broadened laser as reference light, and then performs nonlinear actions such as sum frequency with the ultra-short light to be measured after spectral narrow-band filtering, so as to encode and transfer the contrast information of the time domain of the pulse to be measured to the frequency domain of signal light such as sum frequency, realize time-spectral encoding, and finally obtain the time-domain contrast information of the light to be measured by measuring the spectrum, providing important parameter support for the study and interpretation of the interaction between high-energy laser pulses and matter. However, the patent does not involve any introduction to discriminative spatiotemporal knowledge in the frequency domain for video self-supervised learning. Summary of the invention

[0010] The present invention provides a temporal contrast graph learning system for self-supervised video representation learning, which can highlight motion information and discover distinctive spatiotemporal representations from videos.

[0011] Another object of the present invention is to provide a time domain contrast graph learning method for self-supervised video representation learning.

[0012] In order to achieve the above technical effects, the technical solution of the present invention is as follows:

[0013] A temporal contrast graph learning system for self-supervised video representation learning, comprising:

[0014] Sampling and random shuffling module, for each video, multiple segments are uniformly sampled and shuffled, and for each segment, all its frames are sampled into several fixed-length frame sets;

[0015] The spatiotemporal knowledge discovery module discovers distinguishable spatiotemporal representations of video segments and framesets, and uses 3DCNN to extract spatiotemporal features of all segments and framesets;

[0016] The temporal contrast map learning module uses the spatiotemporal features of segments and frames to construct two temporal contrast map structures, namely intra-segment map and inter-segment map.

[0017] Adaptive Sequence Prediction Module,The video segment features learned from the temporal contrast map learning,module are adaptively propagated through the adaptive segment sequence,prediction module to output the probability distribution of possible,sequences.

[0018] Furthermore, the specific processing process of the sampling and random scrambling module is:

[0019] Segments are uniformly sampled from the video at intervals of p frames. After sampling, the segments are shuffled to form segment tuples S = 1 ,s 2 ,…,s n >, for each segment s i , where all frames are evenly divided into m sets of equal length, and then the fragments s are obtained i Frameset F i = <f 1 ,f 2 ,…,f m >, for segment tuples, they contain the dynamic information and strict temporal correlation of the video, which is essentially the global temporal structure of the video. For the set of frames in a segment, the frame-level temporal correlation between frames provides the local temporal structure of the video. By considering both global and local temporal structures, the discriminative temporal correlation of the video can be explored.

[0020] ​Furthermore, the specific processing process of the spatiotemporal knowledge discovery module is:

[0021] The input video segment is represented as Where C is the channel number, L represents the number of frames, H and W are the width and height of the video frame respectively, and its spectrum is calculated along the time domain to obtain the output features Where K is the number of frequency bands, since the discrete cosine transform is expressed as:

[0022]

[0023] Let L = 2 in equation (1), then:

[0024]

[0025]

[0026] From formula (2), we can see that represents low-frequency information, that is, the sum of video features, and Represents high-frequency information, that is, the difference between adjacent video features. Therefore, low-frequency representation can retain most of the scene information, while at high frequencies, the scene information will be offset and obvious motion edges will be highlighted. In order to further verify this phenomenon, L is expanded to 3 and the following representation is obtained:

[0027]

[0028] In order to capture the temporal dynamics and scene appearance at different frequencies, All frequency components except for 2 are summed to avoid the influence of low-frequency information when modeling the time domain relationship. The discriminative spatiotemporal representation is obtained when L = 2 and 3 respectively.

[0029]

[0030] Therefore, when L = n, we get:

[0031]

[0032] Since the left side of formula (6) is a constant and can be ignored, so equation (6) is equivalent to:

[0033]

[0034] When L = n + 1, equation (7) is expanded to its general form:

[0035]

[0036] From equation (8), we can conclude that the discriminative spatiotemporal representation of a video clip is Essentially the original video footage With its temporal average pooling Subtracting along the time axis, the spatiotemporal knowledge discovery module can extract discriminative temporal representations of video clips without complex computation and is flexible enough to be inserted into existing models in a plug-and-play manner.

[0037] Furthermore, the specific processing process of the time domain contrast image learning module is:

[0038] After obtaining the spatiotemporal features of the clips and frame sets, two temporal comparison graph structures are constructed: inter-clip and intra-clip temporal comparison graphs to increase the temporal diversity of the video:

[0039] Suppose a video is shuffled into 4 segments, each segment is further sampled into 4 frame sets, and the corresponding segment S is obtained 2 ,s 4 ,s 1 ,s 3 > and for each video segment S i The frame set F (i=1,…,4) i = <f 1 ,f 2 ,f 3 ,f 4 >, based on the segment feature S and frame set feature F i (i=1,…,4) and prior knowledge of time domain relations, construct time domain comparison graphs; in order to construct intra-segment and inter-segment time domain comparison graphs, the prior knowledge about time domain relations and corresponding feature vectors is used. In order to fix the symbols, the intra-segment and inter-segment graphs are represented as and G inter =f(X inter ,A inter ), where k = 1, ..., m, m is the number of frame sets in the video clip. The correct order of frames in the frame set and the correct order of segments in the segment tuple are known. The prior temporal relationship is used to determine the edges of the graph. In order to jointly discover the correlation between the graphs between segments and within segments and make the graph learning consistent, G intra and G inter Share the same structure but different weight parameters, G inter and They are all undirected graphs. For G inter , randomly remove edges and mask node features to generate two graph views and The node embeddings of the two generated views are represented as and ​Since different graph views provide different contexts for each node, the original graph is corrupted at the structure and attribute levels to achieve contrastive learning between node embeddings from different views. Therefore, two strategies for generating graph views are proposed: removing edges and masking nodes;

[0040] Using a random masking matrix Randomly remove edges in the original graph; when A ij =1, The input belongs to Bernoulli distribution otherwise Here r represents the probability of each edge being removed, then the resulting adjacency matrix can be calculated as follows, where is the Hadamard product:

[0041]

[0042] In addition, using random vectors Mask some node features with zero; each dimension of which comes from a Bernoulli distribution Then, the generated shielding feature The calculation is as follows:

[0043]

[0044] where [·;·] is a concatenation operator that jointly exploits these two strategies to generate a graph view;

[0045] Different from the noise contrast estimation loss that only considers negative samples between views, it is necessary to consider negative samples between views and within views at the same time, distinguish the embeddings of the same node in two different views from other node embeddings. Given a positive pair, the negative samples come from all other nodes in the two views, that is, between views or within views. In order to calculate the relationship between embeddings u and v from two views, the relationship function φ(u,v)=f(g(u),g(v)) is defined, where f is the L2 normalized dot product similarity, g is a nonlinear projection with two layers of multi-layer perception, and the positive pair (u i ,v i ) is defined as:

[0046]

[0047] II [k≠i] ∈{0,1} is an indicator function, which is equal to 1 if k≠i. τ is a temperature parameter, which is set to 0.5 based on experience. The first term in the denominator represents positive pairs, the second term represents negative pairs between views, and the third term represents negative pairs within a view. Since the two views are symmetric, the loss of the other view is 1 for l(v i ,u i) is similar to the definition of G inter The overall contrast loss is defined as follows:

[0048]

[0049] Clip image The contrast loss can be compared with G inter Similarly calculated:

[0050]

[0051] Then, the overall temporal contrast map loss is defined as follows, where α and β are the weights of the intra-clip map and the inter-clip map, respectively:

[0052]

[0053] Furthermore, the specific processing process of the adaptive sequence prediction module is:

[0054] The adaptive sequence prediction module receives features from different video clips and learns a global context embedding, which is then used to recalibrate the input features from different clips, using the video clip features learned from the temporal contrast map as input and the probability distribution of the sequence as output, formulating the sequence prediction task as a classification task;

[0055] In order to fix the symbol, suppose a video is shuffled into n segments, and the segment feature of the node learned from the temporal comparison graph between segments is {f 1 ,…,f n},in To exploit the correlation between these fragments, these feature vectors are concatenated and passed through a fully connected layer to obtain a joint representation:

[0056] Z u =W s [f 1 ,…,f n ]+b s (15)

[0057] Where [·,·] represents a join operation, represents the joint representation, W s and b s are the weights and biases of the fully connected layer; choose To limit the model capacity and increase its generalization ability, in order to take advantage of the joint representation Z u The global context information aggregated in , predicts its activation signal through a fully connected layer:

[0058] E=W e Z u +b e (16)

[0059] Where W e and b e is the weight and bias of the fully connected layer, and Finally, it is used to adaptively recalibrate the input features f through a simple gating mechanism k :

[0060]

[0061] where ⊙ is the channel product operation for each element in the channel dimension, and δ(·) is the ReLU function, which allows the features of one segment to recalibrate the features of another segment while maintaining the correlation between different segments; finally, these refined feature vectors is concatenated and fed into a two-layer perceptron with a normalized exponential function to output segment order predictions, and the cross entropy loss is used to measure the correctness of the predictions:

[0062]

[0063] where y i and p i Represent the probability that a sample belongs to order class i in the real situation and prediction, respectively, and C represents the number of all possible orders;

[0064] The overall self-supervised learning loss of the system is obtained by combining equations (14) and (18), where λ g and λ o Separate control and The contributions are as follows:

[0065]

[0066] A temporal contrast graph learning method for self-supervised video representation learning comprises the following steps:

[0067] S1: For each video, multiple segments are uniformly sampled and shuffled by the sampling and random shuffling module. For each segment, all its frames are sampled into several fixed-length frame sets;

[0068] S2: The spatiotemporal knowledge discovery module discovers discriminative spatiotemporal representations of video segments and framesets, and uses 3DCNN to extract spatiotemporal features of all segments and framesets;

[0069] S3: The temporal contrast map learning module uses the spatiotemporal features of segments and frames to construct two temporal contrast map structures, namely intra-segment map and inter-segment map.

[0070] S4: The video segment features learned from the temporal contrast map learning module are adaptively propagated through the adaptive segment order prediction module to output a probability distribution of possible orders.

[0071] Furthermore, the specific process of step S1 is:

[0072] Segments are uniformly sampled from the video at intervals of p frames. After sampling, the segments are shuffled to form segment tuples S = 1 ,s 2 ,…,s n >, for each segment s i , where all frames are evenly divided into m sets of equal length, and then the fragments s are obtained i Frameset F i = <f 1 ,f 2 ,…,f m >, for segment tuples, they contain the dynamic information and strict temporal correlation of the video, which is essentially the global temporal structure of the video. For the set of frames in a segment, the frame-level temporal correlation between frames provides the local temporal structure of the video. By considering both global and local temporal structures, the discriminative temporal correlation of the video can be explored.

[0073] Furthermore, the specific process of step S2 is:

[0074] The input video segment is represented as Where C is the channel number, L represents the number of frames, H and W are the width and height of the video frame respectively, and its spectrum is calculated along the time domain to obtain the output features Where K is the number of frequency bands, since the discrete cosine transform is expressed as:

[0075]

[0076] Let L = 2 in equation (1), then:

[0077]

[0078]

[0079] From formula (2), we can see that represents low-frequency information, that is, the sum of video features, and Represents high-frequency information, that is, the difference between adjacent video features. Therefore, low-frequency representation can retain most of the scene information, while at high frequencies, the scene information will be offset and obvious motion edges will be highlighted. In order to further verify this phenomenon, L is expanded to 3 and the following representation is obtained:

[0080] ​

[0081] In order to capture the temporal dynamics and scene appearance at different frequencies, All frequency components except for 2 are summed to avoid the influence of low-frequency information when modeling the time domain relationship. The discriminative spatiotemporal representation is obtained when L = 2 and 3 respectively.

[0082]

[0083] Therefore, when L = n, we get:

[0084]

[0085] Since the left side of formula (6) is a constant and can be ignored, so equation (6) is equivalent to:

[0086]

[0087] When L = n + 1, equation (7) is expanded to its general form:

[0088]

[0089] From equation (8), we can conclude that the discriminative spatiotemporal representation of a video clip is Essentially the original video footage With its temporal average pooling Subtracting along the time axis, the spatiotemporal knowledge discovery module can extract discriminative temporal representations of video clips without complex computation and is flexible enough to be inserted into existing models in a plug-and-play manner.

[0090] Furthermore, the specific process of step S3 is:

[0091] After obtaining the spatiotemporal features of the clips and frame sets, two temporal comparison graph structures are constructed: inter-clip and intra-clip temporal comparison graphs to increase the temporal diversity of the video:

[0092] Suppose a video is shuffled into 4 segments, each segment is further sampled into 4 frame sets, and the corresponding segment S is obtained 2 ,s 4 ,s 1 ,s 3 > and for each video segment S i The frame set F (i=1,…,4) i = <f 1 ,f 2 ,f 3 ,f 4 >, based on the segment feature S and frame set feature F​i (i=1,…,4) and prior knowledge of time domain relations, construct time domain comparison graphs; in order to construct intra-segment and inter-segment time domain comparison graphs, the prior knowledge about time domain relations and corresponding feature vectors is used. In order to fix the symbols, the intra-segment and inter-segment graphs are represented as and G inter =f(X inter ,A inter ), where k = 1, ..., m, m is the number of frame sets in the video clip. The correct order of frames in the frame set and the correct order of segments in the segment tuple are known. The prior temporal relationship is used to determine the edges of the graph. In order to jointly discover the correlation between the graphs between segments and within segments and make the graph learning consistent, G intra and G inter Share the same structure but different weight parameters, G inter and They are all undirected graphs. For G inter , randomly remove edges and mask node features to generate two graph views and The node embeddings of the two generated views are represented as and Since different graph views provide different contexts for each node, the original graph is corrupted at the structure and attribute levels to achieve contrastive learning between node embeddings from different views. Therefore, two strategies for generating graph views are proposed: removing edges and masking nodes;

[0093] Using a random masking matrix Randomly remove edges in the original graph; when A ij =1, The input belongs to Bernoulli distribution otherwise Here r represents the probability of each edge being removed, then the resulting adjacency matrix can be calculated as follows, where is the Hadamard product:

[0094]

[0095] In addition, using random vectors Mask some node features with zero; each dimension of which comes from a Bernoulli distribution Then, the generated shielding feature The calculation is as follows:

[0096]

[0097] where [·;·] is a concatenation operator that jointly exploits these two strategies to generate a graph view;

[0098] Different from the noise contrast estimation loss that only considers negative samples between views, it is necessary to consider negative samples between views and within views at the same time, distinguish the embeddings of the same node in two different views from other node embeddings. Given a positive pair, the negative samples come from all other nodes in the two views, that is, between views or within views. In order to calculate the relationship between embeddings u and v from two views, the relationship function φ(u,v)=f(g(u),g(v)) is defined, where f is the L2 normalized dot product similarity, g is a nonlinear projection with two layers of multi-layer perception, and the positive pair (u i ,v i ) is defined as:

[0099]

[0100] II [k≠i] ∈{0,1} is an indicator function, which is equal to 1 if k≠i. τ is a temperature parameter, which is set to 0.5 based on experience. The first term in the denominator represents positive pairs, the second term represents negative pairs between views, and the third term represents negative pairs within a view. Since the two views are symmetric, the loss of the other view is 1 for l(v i ,u i ) is similar to the definition of G inter The overall contrast loss is defined as follows:

[0101]

[0102] Clip image The contrast loss can be compared with G inter Similarly calculated:

[0103]

[0104] Then, the overall temporal contrast map loss is defined as follows, where α and β are the weights of the intra-clip map and the inter-clip map, respectively:

[0105]

[0106] Furthermore, the specific process of step S4 is:

[0107] The adaptive sequence prediction module receives features from different video clips and learns a global context embedding, which is then used to recalibrate the input features from different clips, using the video clip features learned from the temporal contrast map as input and the probability distribution of the sequence as output, formulating the sequence prediction task as a classification task;

[0108] In order to fix the symbol, suppose a video is shuffled into n segments, and the segment feature of the node learned from the temporal comparison graph between segments is {f 1 ,…,f n},in (k=1,…,n), in order to exploit the correlation between these fragments, these feature vectors are concatenated and passed through a fully connected layer to obtain a joint representation:

[0109] Z u =W s [f 1 ,…,f n ]+b s (15)

[0110] Where [·,·] represents a join operation, represents the joint representation, W s and b s are the weights and biases of the fully connected layer; choose To limit the model capacity and increase its generalization ability, in order to take advantage of the joint representation Z u The global context information aggregated in , predicts its activation signal through a fully connected layer:

[0111] E=W e Z u +b e (16)

[0112] Where W e and b e is the weight and bias of the fully connected layer, and obtains the excitation signal E∈ c Finally, it is used to adaptively recalibrate the input features f through a simple gating mechanism k :

[0113]

[0114] where ⊙ is the channel product operation for each element in the channel dimension, and δ(·) is the ReLU function, which allows the features of one segment to recalibrate the features of another segment while maintaining the correlation between different segments; finally, these refined feature vectors is concatenated and fed into a two-layer perceptron with a normalized exponential function to output segment order predictions, and the cross entropy loss is used to measure the correctness of the predictions:

[0115]

[0116] where y i and p i Represent the probability that a sample belongs to order class i in the real situation and prediction, respectively, and C represents the number of all possible orders;

[0117] The overall self-supervised learning loss of the system is obtained by combining equations (14) and (18), where λ g and λ o Separate control and The contributions are as follows:

[0118]

[0119] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0120] The present invention firstly introduces a spatiotemporal knowledge discovery module (STKD), which extracts motion-enhanced spatiotemporal representations from videos based on frequency domain analysis of discrete cosine transform. In order to explicitly construct a multi-scale temporal correlation model for unlabeled videos, the prior knowledge about the order of frames and clips is integrated into a graph structure, namely, the intra-clip / inter-clip temporal contrast graph (TCG). Then, a specific temporal contrast graph learning module is designed to maximize the consistency between nodes in different graph views. In order to generate monitoring signals for unlabeled videos, an adaptive sequence prediction module is introduced, which utilizes the correlation knowledge between video clips to learn the global context representation and adaptively recalibrate the channel features. BRIEF DESCRIPTION OF THE DRAWINGS

[0121] Figure 1 It is a schematic diagram of multi-scale time domain correlation in the prior art;

[0122] Figure 2 is a schematic diagram of video frames with different frequencies in the prior art;

[0123] Figure 3 It is a framework diagram of the system of the present invention;

[0124] Figure 4 is a schematic diagram of the spatiotemporal knowledge discovery (STKD) module;

[0125] Figure 5 It is a time domain comparison diagram within the clip;

[0126] Figure 6 It is a time domain comparison diagram between clips;

[0127] Figure 7 Schematic diagram of the adaptive fragment order prediction module. DETAILED DESCRIPTION

[0128] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0129] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0130] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0131] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0132] Example 1

[0133] like Figure 3 As shown, a temporal contrast graph learning system for self-supervised video representation learning includes:

[0134] Sampling and random shuffling module, for each video, multiple segments are uniformly sampled and shuffled, and for each segment, all its frames are sampled into several fixed-length frame sets;

[0135] The spatiotemporal knowledge discovery module discovers distinguishable spatiotemporal representations of video segments and framesets, and uses 3DCNN to extract spatiotemporal features of all segments and framesets;

[0136] The temporal contrast map learning module uses the spatiotemporal features of segments and frames to construct two temporal contrast map structures, namely intra-segment map and inter-segment map.

[0137] Adaptive Sequence Prediction Module,The video segment features learned from the temporal contrast map learning,module are adaptively propagated through the adaptive segment sequence,prediction module to output the probability distribution of possible,sequences.

[0138] The specific processing process of the sampling and random shuffling module is:

[0139] Segments are uniformly sampled from the video at intervals of p frames. After sampling, the segments are shuffled to form segment tuples S = 1 ,s 2 ,…,s n >, for each segment s i , where all frames are evenly divided into m sets of equal length, and then the fragments s are obtained i Frameset F i = <f 1 ,f 2 ,…,f m >, for segment tuples, they contain the dynamic information and strict temporal correlation of the video, which is essentially the global temporal structure of the video. For the set of frames in a segment, the frame-level temporal correlation between frames provides the local temporal structure of the video. By considering both global and local temporal structures, the discriminative temporal correlation of the video can be explored.

[0140] like Figure 4 As shown in Figure 2, the specific processing process of the spatiotemporal knowledge discovery module is:

[0141] The input video segment is represented as Where C is the channel number, L represents the number of frames, H and W are the width and height of the video frame respectively, and its spectrum is calculated along the time domain to obtain the output features ​ Where K is the number of frequency bands, since the discrete cosine transform is expressed as:

[0142]

[0143] Let L = 2 in equation (1), then:

[0144]

[0145]

[0146] From formula (2), we can see that represents low-frequency information, that is, the sum of video features, and Represents high-frequency information, that is, the difference between adjacent video features. Therefore, low-frequency representation can retain most of the scene information, while at high frequencies, the scene information will be offset and obvious motion edges will be highlighted. In order to further verify this phenomenon, L is expanded to 3 and the following representation is obtained:

[0147]

[0148] In order to capture the temporal dynamics and scene appearance at different frequencies, All frequency components except for 2 are summed to avoid the influence of low-frequency information when modeling the time domain relationship. The discriminative spatiotemporal representation is obtained when L = 2 and 3 respectively.

[0149]

[0150] Therefore, when L = n, we get:

[0151]

[0152] Since the left side of formula (6) is a constant and can be ignored, so equation (6) is equivalent to:

[0153]

[0154] When L = n + 1, equation (7) is expanded to its general form:

[0155]

[0156] From equation (8), we can conclude that the discriminative spatiotemporal representation of a video clip is Essentially the original video footage With its temporal average pooling Subtracting along the time axis, the spatiotemporal knowledge discovery module can extract discriminative temporal representations of video clips without complex computation and is flexible enough to be inserted into existing models in a plug-and-play manner.

[0157] like Figure 5-6 As shown in Figure 2, the specific processing process of the time domain contrast graph learning module is:

[0158] After obtaining the spatiotemporal features of the clips and frame sets, two temporal comparison graph structures are constructed: inter-clip and intra-clip temporal comparison graphs to increase the temporal diversity of the video:

[0159] Suppose a video is shuffled into 4 segments, each segment is further sampled into 4 frame sets, and the corresponding segment S is obtained 2 ,s 4 ,s 1 ,s 3 > and for each video segment S i The frame set F (i=1,…,4) i = <f 1 ,f 2 ,f 3 ,f 4 >, based on the segment feature S and frame set feature F i (i=1,…,4) and prior knowledge of time domain relations, construct time domain comparison graphs; in order to construct intra-segment and inter-segment time domain comparison graphs, the prior knowledge about time domain relations and corresponding feature vectors is used. In order to fix the symbols, the intra-segment and inter-segment graphs are represented as and G inter =f(X inter ,A inter ), where k = 1, ..., m, m is the number of frame sets in the video clip. The correct order of frames in the frame set and the correct order of segments in the segment tuple are known. The prior temporal relationship is used to determine the edges of the graph. In order to jointly discover the correlation between the graphs between segments and within segments and make the graph learning consistent, G intra and G inter Share the same structure but different weight parameters, G inter and They are all undirected graphs. For G inter , randomly remove edges and mask node features to generate two graph views and The node embeddings of the two generated views are represented as and Since different graph views provide different contexts for each node, the original graph is corrupted at the structure and attribute levels to achieve contrastive learning between node embeddings from different views. Therefore, two strategies for generating graph views are proposed: removing edges and masking nodes;

[0160] ​Using a random masking matrix Randomly remove edges in the original graph; when A ij =1, The input belongs to Bernoulli distribution otherwise Here r represents the probability of each edge being removed, then the resulting adjacency matrix can be calculated as follows, where is the Hadamard product:

[0161]

[0162] In addition, using random vectors Mask some node features with zero; each dimension of which comes from a Bernoulli distribution Then, the generated shielding feature The calculation is as follows:

[0163]

[0164] where [·;·] is a concatenation operator that jointly exploits these two strategies to generate a graph view;

[0165] Different from the noise contrast estimation loss that only considers negative samples between views, it is necessary to consider negative samples between views and within views at the same time, distinguish the embeddings of the same node in two different views from other node embeddings. Given a positive pair, the negative samples come from all other nodes in the two views, that is, between views or within views. In order to calculate the relationship between embeddings u and v from two views, the relationship function φ(u,v)=f(g(u),g(v)) is defined, where f is the L2 normalized dot product similarity, g is a nonlinear projection with two layers of multi-layer perception, and the positive pair (u i ,v i ) is defined as:

[0166]

[0167] II [k≠i] ∈{0,1} is an indicator function, which is equal to 1 if k≠i. τ is a temperature parameter, which is set to 0.5 based on experience. The first term in the denominator represents positive pairs, the second term represents negative pairs between views, and the third term represents negative pairs within a view. Since the two views are symmetric, the loss of the other view is 1 for l(v i ,u i ) is similar to the definition of G inter The overall contrast loss is defined as follows:

[0168]

[0169] Clip image The contrast loss can be compared with G inter Similarly calculated:

[0170]

[0171] Then, the overall temporal contrast map loss is defined as follows, where α and β are the weights of the intra-clip map and the inter-clip map, respectively:

[0172]

[0173] like Figure 7 As shown in Figure 1, the specific processing process of the adaptive sequence prediction module is:

[0174] The adaptive sequence prediction module receives features from different video clips and learns a global context embedding, which is then used to recalibrate the input features from different clips, using the video clip features learned from the temporal contrast map as input and the probability distribution of the sequence as output, formulating the sequence prediction task as a classification task;

[0175] In order to fix the symbol, suppose a video is shuffled into n segments, and the segment feature of the node learned from the temporal comparison graph between segments is {f 1 ,…,f n},in To exploit the correlation between these fragments, these feature vectors are concatenated and passed through a fully connected layer to obtain a joint representation:

[0176] Z u =W s [f 1 ,…,f n ]+b s (15)

[0177] Where [·,·] represents a join operation, represents the joint representation, W s and b s are the weights and biases of the fully connected layer; choose To limit the model capacity and increase its generalization ability, in order to take advantage of the joint representation Z u The global context information aggregated in , predicts its activation signal through a fully connected layer:

[0178] E=W e Z u +b e (16)

[0179] Where W e and b e is the weight and bias of the fully connected layer, and Finally, it is used to adaptively recalibrate the input features f through a simple gating mechanism k :

[0180]

[0181] where ⊙ is the channel product operation for each element in the channel dimension, and δ(·) is the ReLU function, which allows the features of one segment to recalibrate the features of another segment while maintaining the correlation between different segments; finally, these refined feature vectors is concatenated and fed into a two-layer perceptron with a normalized exponential function to output segment order predictions, and the cross entropy loss is used to measure the correctness of the predictions:

[0182]

[0183] where y i and p i Represent the probability that a sample belongs to order class i in the real situation and prediction, respectively, and C represents the number of all possible orders;

[0184] The overall self-supervised learning loss of the system is obtained by combining equations (14) and (18), where λ g and λ o Separate control and The contributions are as follows:

[0185]

[0186] Example 2

[0187] A temporal contrast graph learning method for self-supervised video representation learning comprises the following steps:

[0188] S1: For each video, multiple segments are uniformly sampled and shuffled by the sampling and random shuffling module. For each segment, all its frames are sampled into several fixed-length frame sets;

[0189] S2: The spatiotemporal knowledge discovery module discovers discriminative spatiotemporal representations of video segments and framesets, and uses 3DCNN to extract spatiotemporal features of all segments and framesets;

[0190] S3: The temporal contrast map learning module uses the spatiotemporal features of segments and frames to construct two temporal contrast map structures, namely intra-segment map and inter-segment map.

[0191] S4: The video segment features learned from the temporal contrast map learning module are adaptively propagated through the adaptive segment order prediction module to output a probability distribution of possible orders.

[0192] Furthermore, the specific process of step S1 is:

[0193] Segments are uniformly sampled from the video at intervals of p frames. After sampling, the segments are shuffled to form segment tuples S = 1 ,s 2 ,…,s n >, for each segment s i , where all frames are evenly divided into m sets of equal length, and then the fragments s are obtained i Frameset F i = <f 1 ,f 2 ,…,f m >, for segment tuples, they contain the dynamic information and strict temporal correlation of the video, which is essentially the global temporal structure of the video. For the set of frames in a segment, the frame-level temporal correlation between frames provides the local temporal structure of the video. By considering both global and local temporal structures, the discriminative temporal correlation of the video can be explored.

[0194] Furthermore, the specific process of step S2 is:

[0195] The input video segment is represented as Where C is the channel number, L represents the number of frames, H and W are the width and height of the video frame respectively, and its spectrum is calculated along the time domain to obtain the output features Where K is the number of frequency bands, since the discrete cosine transform is expressed as:

[0196]

[0197] Let L = 2 in equation (1), then:

[0198]

[0199]

[0200] From formula (2), we can see that represents low-frequency information, that is, the sum of video features, and Represents high-frequency information, that is, the difference between adjacent video features. Therefore, low-frequency representation can retain most of the scene information, while at high frequencies, the scene information will be offset and obvious motion edges will be highlighted. In order to further verify this phenomenon, L is expanded to 3 and the following representation is obtained:

[0201]

[0202] In order to capture the temporal dynamics and scene appearance at different frequencies, ​All frequency components except for 2 are summed to avoid the influence of low-frequency information when modeling the time domain relationship. The discriminative spatiotemporal representation is obtained when L = 2 and 3 respectively.

[0203]

[0204] Therefore, when L = n, we get:

[0205]

[0206] Since the left side of formula (6) is a constant and can be ignored, so equation (6) is equivalent to:

[0207]

[0208] When L = n + 1, equation (7) is expanded to its general form:

[0209]

[0210] From equation (8), we can conclude that the discriminative spatiotemporal representation of a video clip is Essentially the original video footage With its temporal average pooling Subtracting along the time axis, the spatiotemporal knowledge discovery module can extract discriminative temporal representations of video clips without complex computation and is flexible enough to be inserted into existing models in a plug-and-play manner.

[0211] Furthermore, the specific process of step S3 is:

[0212] After obtaining the spatiotemporal features of the clips and frame sets, two temporal comparison graph structures are constructed: inter-clip and intra-clip temporal comparison graphs to increase the temporal diversity of the video:

[0213] Suppose a video is shuffled into 4 segments, each segment is further sampled into 4 frame sets, and the corresponding segment S is obtained 2 ,s 4 ,s 1 ,s 3 > and for each video segment S i The frame set F (i=1,…,4) i = <f 1 ,f 2 ,f 3 ,f 4 >, based on the segment feature S and frame set feature F i ​(i=1,…,4) and prior knowledge of time domain relations, construct time domain comparison graphs; in order to construct intra-segment and inter-segment time domain comparison graphs, the prior knowledge about time domain relations and corresponding feature vectors is used. In order to fix the symbols, the intra-segment and inter-segment graphs are represented as and G inter =f(X inter ,A inter ), where k = 1, ..., m, m is the number of frame sets in the video clip. The correct order of frames in the frame set and the correct order of segments in the segment tuple are known. The prior temporal relationship is used to determine the edges of the graph. In order to jointly discover the correlation between the graphs between segments and within segments and make the graph learning consistent, G intra and G inter Share the same structure but different weight parameters, G inter and They are all undirected graphs. For G inter , randomly remove edges and mask node features to generate two graph views and The node embeddings of the two generated views are represented as and Since different graph views provide different contexts for each node, the original graph is corrupted at the structure and attribute levels to achieve contrastive learning between node embeddings from different views. Therefore, two strategies for generating graph views are proposed: removing edges and masking nodes;

[0214] Using a random masking matrix Randomly remove edges in the original graph; when A ij =1, The input belongs to Bernoulli distribution otherwise Here r represents the probability of each edge being removed, then the resulting adjacency matrix can be calculated as follows, where is the Hadamard product:

[0215]

[0216] In addition, using random vectors Mask some node features with zero; each dimension of which comes from a Bernoulli distribution Then, the generated shielding feature The calculation is as follows:

[0217]

[0218] where [·;·] is a concatenation operator that jointly exploits these two strategies to generate a graph view;

[0219] Different from the noise contrast estimation loss that only considers negative samples between views, it is necessary to consider negative samples between views and within views at the same time, distinguish the embeddings of the same node in two different views from other node embeddings. Given a positive pair, the negative samples come from all other nodes in the two views, that is, between views or within views. In order to calculate the relationship between embeddings u and v from two views, the relationship function φ(u,v)=f(g(u),g(v)) is defined, where f is the L2 normalized dot product similarity, g is a nonlinear projection with two layers of multi-layer perception, and the positive pair (u i ,v i ) is defined as:

[0220]

[0221] II [k≠i] ∈{0,1} is an indicator function, which is equal to 1 if k≠i. τ is a temperature parameter, which is set to 0.5 based on experience. The first term in the denominator represents positive pairs, the second term represents negative pairs between views, and the third term represents negative pairs within a view. Since the two views are symmetric, the loss of the other view is 1 for l(v i ,u i ) is similar to the definition of G inter The overall contrast loss is defined as follows:

[0222]

[0223] Clip image The contrast loss can be compared with G inter Similarly calculated:

[0224]

[0225] Then, the overall temporal contrast map loss is defined as follows, where α and β are the weights of the intra-clip map and the inter-clip map, respectively:

[0226]

[0227] Furthermore, the specific process of step S4 is:

[0228] The adaptive sequence prediction module receives features from different video clips and learns a global context embedding, which is then used to recalibrate the input features from different clips, using the video clip features learned from the temporal contrast map as input and the probability distribution of the sequence as output, formulating the sequence prediction task as a classification task;

[0229] In order to fix the symbol, suppose a video is shuffled into n segments, and the segment feature of the node learned from the temporal comparison graph between segments is {f 1 ,…,f n},in (k=1,…,n), in order to exploit the correlation between these fragments, these feature vectors are concatenated and passed through a fully connected layer to obtain a joint representation:

[0230] Z u =W s [f 1 ,…,f n ]+b s (15)

[0231] Where [·,·] represents a join operation, represents the joint representation, W s and b s are the weights and biases of the fully connected layer; choose To limit the model capacity and increase its generalization ability, in order to take advantage of the joint representation Z u The global context information aggregated in , predicts its activation signal through a fully connected layer:

[0232] E=W e Z u +b e (16)

[0233] Where W e and b e is the weight and bias of the fully connected layer, and obtains the excitation signal E∈ c Finally, it is used to adaptively recalibrate the input features f through a simple gating mechanism k :

[0234]

[0235] where ⊙ is the channel product operation for each element in the channel dimension, and δ(·) is the ReLU function, which allows the features of one segment to recalibrate the features of another segment while maintaining the correlation between different segments; finally, these refined feature vectors is concatenated and fed into a two-layer perceptron with a normalized exponential function to output segment order predictions, and the cross entropy loss is used to measure the correctness of the predictions:

[0236]

[0237] where y i and p i Represent the probability that a sample belongs to order class i in the real situation and prediction, respectively, and C represents the number of all possible orders;

[0238] The overall self-supervised learning loss of the system is obtained by combining equations (14) and (18), where λ g and λ o Separate control and The contributions are as follows:

[0239]

[0240] The same or similar reference numerals correspond to the same or similar components;

[0241] The positional relationships described in the drawings are only for illustrative purposes and should not be construed as limiting the present patent.

[0242] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A temporal contrast graph learning system for self-supervised video representation learning, characterized in that: include: Sampling and random shuffling module, for each video, multiple segments are uniformly sampled and shuffled, and for each segment, all its frames are sampled into several fixed-length frame sets; The spatiotemporal knowledge discovery module discovers distinguishable spatiotemporal representations of video segments and framesets, and uses 3DCNN to extract spatiotemporal features of all segments and framesets; The specific processing process of the spatiotemporal knowledge discovery module is: The input video segment is represented as ,in is the channel number, Indicates the frame number. and are the width and height of the video frame, respectively. Calculate its spectrum along the time domain and obtain the output features ,in is the number of frequency bands, since the discrete cosine transform is expressed as: （1） In the command (1), ,So: （2） （3） From formula (2), we can see that represents low-frequency information, that is, the sum of video features, and represents high-frequency information, that is, the difference between adjacent video features. Therefore, low-frequency representation can retain most of the scene information, while at high frequencies, the scene information will be offset and obvious motion edges will be highlighted. In order to further verify this phenomenon, Expanding to 3 gives the following representation: （4） In order to capture the temporal dynamics and scene appearance at different frequencies, All frequency components except for are summed to avoid the influence of low-frequency information when modeling time-domain relationships and obtain a discriminative spatiotemporal representation. The time is : （5） Therefore, when When , we get: （6） Since the left side of formula (6) is a constant and can be ignored, so equation (6) is equivalent to: （7） when When , equation (7) is expanded to its general form: （8） From equation (8), we can conclude that the discriminative spatiotemporal representation of a video clip is Essentially the original video footage With its temporal average pooling By subtracting along the time axis, the spatiotemporal knowledge discovery module can extract discriminative temporal representations of video clips without complex computations and is flexible enough to be inserted into existing models in a plug-and-play manner; The temporal contrast map learning module uses the temporal and spatial features of segments and frames to construct two temporal contrast map structures, namely intra-segment map and inter-segment map. The specific processing process of the time domain contrast graph learning module is: After obtaining the spatiotemporal features of the clips and frame sets, two temporal comparison graph structures are constructed: inter-clip and intra-clip temporal comparison graphs to increase the temporal diversity of the video: Suppose a video is shuffled into 4 segments, each segment is further sampled into 4 frame sets, and the corresponding segments are obtained And for each video segment Frameset , based on fragment features , frame set features and prior knowledge of time domain relations, construct time domain comparison graphs; in order to construct intra-fragment and inter-fragment time domain comparison graphs, the prior knowledge of time domain relations and corresponding feature vectors is used. In order to fix the symbols, the intra-fragment and inter-fragment graphs are represented as and , where k=1,…,m, m is the number of frame sets in the video clip. The correct order of frames in the frame set and the correct order of segments in the segment tuple are known. The prior temporal relationship is used to determine the edges of the graph. In order to jointly discover the correlation between inter-segment and intra-segment graphs and make the graph learning consistent, and share the same structure but different weight parameters, and are all undirected graphs. , randomly remove edges and mask node features to generate two graph views and , the node embeddings of the two generated views are expressed as and ,Since different graph views provide different contexts for each node, the original graph is ,destroyed at the structure and attribute levels to achieve contrastive ,learning between node embeddings from different views.,Therefore, two strategies for generating graph views are ,proposed: removing edges and masking nodes; Using a random masking matrix Randomly remove edges in the original graph; when hour, The input belongs to Bernoulli distribution ,otherwise ,here represents the probability of each edge being removed, then the resulting adjacency matrix can be calculated as follows, where is the Hadamard product: （9） In addition, using random vectors Mask some node features with zero; each dimension of which comes from a Bernoulli distribution , , then, the generated shielding feature The calculation is as follows: （10） where [·;·] is a concatenation operator that exploits these two strategies together to generate a graph view; Different from the noise contrast estimation loss that only considers negative samples between views, it is necessary to consider negative samples between views and within views at the same time, distinguishing the embeddings of the same node in two different views from other node embeddings. Given a positive pair, the negative samples come from all other nodes in the two views, that is, between views or within views. In order to calculate the relationship between the embeddings u and v from the two views, the relationship function is defined ,in is the L2 normalized dot product similarity, It is a nonlinear projection with two layers of multi-layer perception, facing The pairwise comparison target of is defined as: （11） in is an indicator function if =1, τ is a temperature parameter, which is set to 0.5 according to experience. The first term in the denominator represents positive pairs, the second term represents negative pairs between views, and the third term represents negative pairs within a view. Since the two views are symmetrical, the loss of the other view is The definition is similar to, The overall contrast loss is defined as follows: （12） Clip image The contrast loss of (k=1,…,m) can be compared with Similarly calculated: （13） Then, the overall temporal contrast map loss is defined as follows, where α and β are the weights of the intra-clip map and the inter-clip map, respectively: （14） Adaptive Sequence Prediction Module,The video segment features learned from the temporal contrast map learning,module are adaptively propagated through the adaptive segment sequence,prediction module to output the probability distribution of possible,sequences.

2. The temporal contrast graph learning system for self-supervised video representation learning according to claim 1, characterized in that: The specific processing process of the sampling and random scrambling module is: Segments are uniformly sampled from the video at intervals of p frames. After sampling, the segments are shuffled to form segment tuples. , for each fragment , where all frames are evenly divided into m sets of equal length, and then the fragments are obtained Frameset ,For the segment tuples, they contain the dynamic information and strict ,temporal correlation of the video, which is essentially the global temporal structure of the video. ,For the set of frames in a segment, the frame-level temporal correlation between frames provides the ,local temporal structure of the video. By considering both the global and local ,temporal structures, the discriminative temporal correlation of the video can ,be explored.

3. The temporal contrast graph learning system for self-supervised video representation learning according to claim 2, characterized in that: The specific processing process of the adaptive sequence prediction module is: The adaptive sequence prediction module receives features from different video clips and learns a global context embedding, which is then used to recalibrate the input features from different clips, using the video clip features learned from the temporal contrast map as input and the probability distribution of the sequence as output, formulating the sequence prediction task as a classification task; In order to fix the symbol, suppose a video is shuffled into n segments, and the segment feature of the node learned from the temporal comparison diagram between segments is ,in , in order to exploit the correlation between these fragments, these feature vectors are concatenated and passed through a fully connected layer to obtain a joint representation: （15） Where [·,·] represents a join operation, It means joint representation, and are the weights and biases of the fully connected layer; choose To limit the model capacity and increase its generalization ability, in order to take advantage of the joint representation The global context information aggregated in , predicts its activation signal through a fully connected layer: （16） in and is the weight and bias of the fully connected layer, and Finally, it is used to adaptively recalibrate the input features through a simple gating mechanism : （17） in is the channel product operation for each element in the channel dimension, is a linear rectification function (ReLU), which allows the features of one segment to recalibrate the features of another segment while maintaining the correlation between different segments; finally, these refined feature vectors is concatenated and fed into a two-layer perceptron with a normalized exponential function to output segment order predictions, and the cross entropy loss is used to measure the correctness of the predictions: （18） in and Represent the probability that a sample belongs to order class i in the real situation and prediction, respectively, and C represents the number of all possible orders; The overall self-supervised learning loss of the system is obtained by combining equations (14) and (18), where and Separate control and The contributions are as follows: （19）。 4. A learning method for a time domain contrast graph learning system for self-supervised video representation learning according to claim 3, characterized in that: The following steps are involved: S1: For each video, multiple segments are uniformly sampled and shuffled by the sampling and random shuffling module. For each segment, all its frames are sampled into several fixed-length frame sets; S2: The spatiotemporal knowledge discovery module discovers discriminative spatiotemporal representations of video segments and framesets, and extracts spatiotemporal features of all segments and framesets using 3DCNN; S3: The temporal contrast map learning module uses the spatiotemporal features of segments and frames to construct two temporal contrast map structures, namely intra-segment map and inter-segment map. S4: The video segment features learned from the temporal contrast map learning module are adaptively propagated through the adaptive segment order prediction module to output a probability distribution of possible orders.

5. The temporal contrast graph learning method for self-supervised video representation learning according to claim 4, characterized in that: The specific process of step S1 is: Segments are uniformly sampled from the video at intervals of p frames. After sampling, the segments are shuffled to form segment tuples. , for each fragment , where all frames are evenly divided into m sets of equal length, and then the fragments are obtained Frameset ,For the segment tuples, they contain the dynamic information and strict ,temporal correlation of the video, which is essentially the global temporal structure of the video. ,For the set of frames in a segment, the frame-level temporal correlation between frames provides the ,local temporal structure of the video. By considering both the global and local ,temporal structures, the discriminative temporal correlation of the video can ,be explored.

6. The temporal contrast graph learning method for self-supervised video representation learning according to claim 5, characterized in that: The specific process of step S2 is: The input video segment is represented as ,in is the channel number, Indicates the frame number. and are the width and height of the video frame, respectively. Calculate its spectrum along the time domain and obtain the output features ,in is the number of frequency bands, since the discrete cosine transform is expressed as: （1） In the command (1), ,So: （2） （3） From formula (2), we can see that represents low-frequency information, that is, the sum of video features, and represents high-frequency information, that is, the difference between adjacent video features. Therefore, low-frequency representation can retain most of the scene information, while at high frequencies, the scene information will be offset and obvious motion edges will be highlighted. In order to further verify this phenomenon, Expanding to 3 gives the following representation: （4） In order to capture the temporal dynamics and scene appearance at different frequencies, All frequency components except for are summed to avoid the influence of low-frequency information when modeling time-domain relationships and obtain a discriminative spatiotemporal representation. The time is : （5） Therefore, when When , we get: （6） Since the left side of formula (6) is a constant and can be ignored, so equation (6) is equivalent to: （7） when When , equation (7) is expanded to its general form: （8） From equation (8), we can conclude that the discriminative spatiotemporal representation of a video clip is Essentially the original video footage With its temporal average pooling Subtracting along the time axis, the spatiotemporal knowledge discovery module can extract discriminative temporal representations of video clips without complex computation and is flexible enough to be inserted into existing models in a plug-and-play manner.

7. The temporal contrast graph learning method for self-supervised video representation learning according to claim 6, characterized in that: The specific process of step S3 is: After obtaining the spatiotemporal features of the clips and frame sets, two temporal comparison graph structures are constructed: inter-clip and intra-clip temporal comparison graphs to increase the temporal diversity of the video: Suppose a video is shuffled into 4 segments, each segment is further sampled into 4 frame sets, and the corresponding segments are obtained And for each video segment Frameset , based on fragment features , frame set features and prior knowledge of time domain relations, construct time domain comparison graphs; in order to construct intra-fragment and inter-fragment time domain comparison graphs, the prior knowledge of time domain relations and corresponding feature vectors is used. In order to fix the symbols, the intra-fragment and inter-fragment graphs are represented as and , where k=1,…,m, m is the number of frame sets in the video clip. The correct order of frames in the frame set and the correct order of segments in the segment tuple are known. The prior temporal relationship is used to determine the edges of the graph. In order to jointly discover the correlation between inter-segment and intra-segment graphs and make the graph learning consistent, and share the same structure but different weight parameters, and are all undirected graphs. , randomly remove edges and mask node features to generate two graph views and , the node embeddings of the two generated views are expressed as and ,Since different graph views provide different contexts for each node, the original graph is ,destroyed at the structure and attribute levels to achieve contrastive ,learning between node embeddings from different views.,Therefore, two strategies for generating graph views are ,proposed: removing edges and masking nodes; Using a random masking matrix Randomly remove edges in the original graph; when hour, The input belongs to Bernoulli distribution ,otherwise ,here represents the probability of each edge being removed, then the resulting adjacency matrix can be calculated as follows, where is the Hadamard product: （9） In addition, using random vectors Mask some node features with zero; each dimension of which comes from a Bernoulli distribution , , then, the generated shielding feature The calculation is as follows: （10） where [·;·] is a concatenation operator that exploits these two strategies together to generate a graph view; Different from the noise contrast estimation loss that only considers negative samples between views, it is necessary to consider negative samples between views and within views at the same time, distinguishing the embeddings of the same node in two different views from other node embeddings. Given a positive pair, the negative samples come from all other nodes in the two views, that is, between views or within views. In order to calculate the relationship between the embeddings u and v from the two views, the relationship function is defined ,in is the L2 normalized dot product similarity, It is a nonlinear projection with two layers of multi-layer perception, facing The pairwise comparison target of is defined as: （11） in is an indicator function if =1, τ is a temperature parameter, which is set to 0.5 according to experience. The first term in the denominator represents positive pairs, the second term represents negative pairs between views, and the third term represents negative pairs within a view. Since the two views are symmetrical, the loss of the other view is The definition is similar to, The overall contrast loss is defined as follows: （12） Clip image The contrast loss of (k=1,…,m) can be compared with Similarly calculated: （13） Then, the overall temporal contrast map loss is defined as follows, where α and β are the weights of the intra-clip map and the inter-clip map, respectively: （14）。 8. The temporal contrast graph learning method for self-supervised video representation learning according to claim 7, characterized in that: The specific process of step S4 is: The adaptive sequence prediction module receives features from different video clips and learns a global context embedding, which is then used to recalibrate the input features from different clips, using the video clip features learned from the temporal contrast map as input and the probability distribution of the sequence as output, formulating the sequence prediction task as a classification task; In order to fix the symbol, suppose a video is shuffled into n segments, and the segment feature of the node learned from the temporal comparison diagram between segments is ,in , in order to exploit the correlation between these fragments, these feature vectors are concatenated and passed through a fully connected layer to obtain a joint representation: （15） Where [·,·] represents a join operation, It means joint representation, and are the weights and biases of the fully connected layer; choose To limit the model capacity and increase its generalization ability, in order to take advantage of the joint representation The global context information aggregated in , predicts its activation signal through a fully connected layer: （16） in and is the weight and bias of the fully connected layer, and Finally, it is used to adaptively recalibrate the input features through a simple gating mechanism : （17） in is the channel product operation for each element in the channel dimension, is a linear rectification function (ReLU), which allows the features of one segment to recalibrate the features of another segment while maintaining the correlation between different segments; finally, these refined feature vectors is concatenated and fed into a two-layer perceptron with a normalized exponential function to output segment order predictions, and the cross entropy loss is used to measure the correctness of the predictions: （18） in and Represent the probability that a sample belongs to order class i in the real situation and prediction, respectively, and C represents the number of all possible orders; The overall self-supervised learning loss of the system is obtained by combining equations (14) and (18), where and Separate control and The contributions are as follows: （19）。

Images