A training method for a video similarity learning model

Abstract

The application provides a training method for a video similarity learning model, comprising: obtaining a training set, including a plurality of original samples and labels indicating corresponding sample categories, the original sample being a video and including a plurality of video frames; training the model multiple times using the training set to obtain a trained model, each time comprising: for each category, taking one original sample of the category as a query sample, and taking other original samples and the query sample as sample pairs to obtain a plurality of sample pairs; extracting feature vectors of samples in the sample pairs using the model, including block-level feature sequences of each video frame of the samples; constructing a plurality of pairs of video frames based on the sample pairs, determining frame-level similarities of the plurality of pairs of video frames according to the block-level feature sequences of the plurality of video frames of the samples, and determining video-level similarities between two samples in the sample pairs according to the frame-level similarities; and updating model parameters according to a total loss calculated by a preset loss function according to the video-level similarities of the plurality of sample pairs corresponding to the query sample.

Description

Technical Field

[0001] This invention relates to the field of computer science, more specifically to the field of video similarity learning, and more specifically to a training method for a video similarity learning model. Background Technology

[0002] The rapid growth of online video resources has brought serious challenges to efficient video search and analysis. To address this issue, researchers have begun to focus on content-based video retrieval, the core of which lies in video similarity learning. Specifically, the goal of video similarity learning is to assign higher similarity scores to related videos, thereby providing a valuable reference for various fine-grained downstream tasks in the field of video retrieval (such as near-duplicate video retrieval and video duplication detection).

[0003] Because the label distribution in video similarity learning tasks is severely imbalanced, with the number of positive samples being far less than the number of negative samples, the latest existing techniques use Average Precision (AP) to measure the overall performance of the model. Intuitively, AP assigns higher weights to the top-ranked positive samples, satisfying the downstream task's requirement to rank samples according to priority. However, the current mainstream method for video similarity learning uses an objective function based on positive and negative sample pairs for training, with the aim of maximizing the similarity score of positive samples and minimizing the similarity score of negative samples in a training batch. For example, references [1], [2], and [3] are all trained using the Triplet objective function, and references [4] and [5] are trained using the InfoNCE objective function. Although the model improves the Recall@K evaluation metric after training with these objective functions, it does not improve the AP evaluation metric, resulting in a bias between the training objective and the evaluation metric.

[0004] In addition, to address similar issues in the image domain, methods have emerged that directly use AP as the objective function. Furthermore, to address the non-differentiability of the original AP, numerous AP optimization methods have emerged, including methods that approximate AP using histograms (e.g., references [6], [7], [8], [9]), black-box optimization methods (e.g., reference

[10] ), and methods that approximate step functions using indicator functions (e.g., reference

[11] ). Although AP optimization methods have achieved great success in the image domain, two substantial challenges in video tasks prevent their direct application to the video domain. The first substantial challenge: Videos have a time dimension compared to images, leading to significant difficulties in feature representation and similarity measurement. The second substantial challenge: Existing AP optimization methods tend to approximate the original AP with the smallest possible approximation error, thus ignoring the gradient vanishing problem in low AP regions. In this case, even with a small approximation error, the optimization effect is not good, and the difficulty in video similarity measurement makes the gradient vanishing phenomenon even more pronounced.

[0005] Based on the above analysis, existing AP optimization methods suffer from poor optimization performance. Furthermore, since videos have an additional time dimension compared to images, there are significant difficulties in feature representation and similarity measurement. This leads to even worse AP optimization results when applying existing AP optimization methods to models in the video domain.

[0006] It should be noted that the background information provided is only for illustrating relevant information about the present invention to aid in understanding the technical solutions of the present invention, and does not imply that the relevant information is necessarily prior art. In the absence of evidence indicating that the relevant information was disclosed before the filing date of this invention, the relevant information should not be considered prior art.

[0007] The references are as follows:

[0008] [1]Giorgos Kordopatis-Zilos,Symeon Papadopoulos,Ioannis Patras,andYiannis Kompatsiaris.Near-duplicate video retrieval with deep metriclearning.In Proceedings of the IEEE international conference on computervision workshops,pages 347–356,2017.

[0009] [2]Giorgos Kordopatis-Zilos,Symeon Papadopoulos,Ioannis Patras,andIoannis Kompatsiaris.Visil:Fine-grained spatiotemporal video similaritylearning.In International Conference on Computer Vision,pages 6351–6360,2019.

[0010] [3]Giorgos Kordopatis-Zilos,Christos Tzelepis,Symeon Papadopoulos,Ioannis Kompatsiaris,and Ioannis Patras.Dns:Distill-and-select for efficientand accurate video indexing and retrieval.International Journal of ComputerVision,130(10):2385–2407,2022.

[0011] [4]Giorgos Kordopatis-Zilos,Giorgos Tolias,Christos Tzelepis,IoannisKompatsiaris,Ioannis Patras,and Symeon Papadopoulos.Self-supervised videosimilarity learning.In IEEE / CVF Conference on Computer Vision and PatternRecognition,pages 4755–4765,2023.

[0012] [5]Jie Shao,Xin Wen,Bingchen Zhao,and Xiangyang Xue.Temporal contextaggregation for video retrieval with contrastive learning.In Proceedings ofthe IEEE / CVF winter conference on applications of computer vision,pages 3268–3278,2021.

[0013] [6]Fatih Cakir,Kun He,Xide Xia,Brian Kulis,and Stan Sclaroff.Deepmetric learning to rank.In IEEE / CVF Conference on Computer Vision and PatternRecognition,pages 1861–1870,2019.

[0014] [7]Zhe Cao,Tao Qin,Tie-Yan Liu,Ming-Feng Tsai,and Hang Li.Learning torank:from pairwise approach to listwise approach.In International Conferenceon Machine Learning,pages 129–136,2007.

[0015] [8]Jerome Revaud,Jon Almaz′an,Rafael S Rezende,and Cesar Roberto deSouza.Learning with average precision:Training image retrieval with alistwise loss.In International Conference on Computer Vision,pages 5107–5116,2019.

[0016] [9] Evgeniya Ustinova and Victor Lempitsky. Learning deep embeddings with histogram loss. Advances in Neural Information Processing Systems, 29, 2016.

[0017]

[10] Michal Rol′Inek, V′It Musil, Anselm Paulus, Marin Vlastelica, Claudio Michaelis, and Georg Martius. Optimizing rank-based metrics with blackbox differentiation. In IEEE / CVF Conference on Computer Vision and PatternRecognition, pages 7620–7630, 2020.

[0018]

[11] Andrew Brown, Weidi Xie, Vicky Kalogeiton, and Andrew Zisserman. Smooth-ap: Smoothing the path towards large-scale image retrieval. InEuropean Conference on Computer Vision, pages 677–694. Springer, 2020. Summary of the Invention

[0019] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a training method for video similarity learning models.

[0020] The objective of this invention is achieved through the following technical solution:

[0021] According to a first aspect of the present invention, a training method for a video similarity learning model is provided, comprising: acquiring a training set, which includes multiple original samples and corresponding labels, wherein an original sample is a video segment, a video segment includes multiple video frames, and the labels indicate the category of the corresponding original sample; using the training set, iteratively training the video similarity learning model in the following manner to obtain a trained video similarity learning model: based on the labels, for each category, taking each original sample in the training set under that category as a query sample, taking other original samples under that category as positive samples, taking original samples from other categories as negative samples, and assigning each positive sample to each... Each negative sample is paired with a query sample to form a sample pair, resulting in multiple sample pairs. The model is used to extract the feature vector of each sample in each sample pair. The feature vector includes the block-level feature sequence of each video frame of the corresponding sample. Multiple pairs of video frames are constructed based on each sample pair. The frame-level similarity of each pair of video frames is determined according to the block-level feature sequence of each video frame of each sample in each sample pair. The video-level similarity between two samples in the sample pair is determined according to the frame-level similarity of the multiple pairs of video frames. The total loss is calculated using a preset loss function based on the video-level similarity of the multiple sample pairs corresponding to each query sample in all query samples. The parameters of the model are updated according to the total loss.

[0022] In some embodiments of the present invention, the preset loss function is as follows during a single training iteration:

[0023]

[0024] Where N represents the number of query samples, K represents the query sample ID, and |S k+ | represents the number of sample pairs formed by the k-th query sample and all positive samples, s ki ∈S k+ S represents the video-level similarity between the k-th query sample and the i-th positive sample. k+ This represents the set of video-level similarities between the k-th query sample and all positive samples. s kj ∈S k- S represents the video-level similarity between the k-th query sample and the j-th negative sample. k- This represents the set of video-level similarities between the k-th query sample and all negative samples. And x < -δ, where x represents s kj -s ki τ represents the gradient hyperparameter, and δ represents the discrimination boundary hyperparameter. And x≥-δ, s kj ∈S k+This represents the video-level similarity between the k-th query sample and the j-th positive sample.

[0025] In some embodiments of the present invention, the model includes an encoder and a feature mapping layer constructed using a deep neural network. The feature vector is obtained by: extracting features of multiple image blocks of each video frame of each sample through the encoder to obtain multiple original block-level features; performing feature mapping on the multiple original block-level features of each video frame through the feature mapping layer to obtain a block-level feature sequence, the block-level feature sequence including multiple block-level features; wherein, the block-level feature sequence of multiple video frames of each sample constitutes the feature vector of that sample.

[0026] In some embodiments of the present invention, the feature vector is obtained by: extracting features from E image blocks of each video frame of a sample with dimensions AxBxCxD using an encoder to obtain E original block-level features of dimension F, where A represents the number of video frames, B represents the height of the video frame, C represents the width of the video frame, and D and F both represent the number of channels; performing feature mapping on all the original block-level features of each video frame through a feature mapping layer to obtain a block-level feature sequence of dimension ExF, the block-level feature sequence including E block-level features of dimension F; wherein, the block-level feature sequence of A video frames of each sample constitutes a feature vector of dimension AxExF.

[0027] In some embodiments of the present invention, the encoder employs a ResNet50 backbone network or a VGG16 network, and the feature mapping layer includes a fully connected network and a multilayer perceptron.

[0028] In some embodiments of the present invention, constructing multiple pairs of video frames includes: for each sample pair, forming a pair of video frames by combining each video frame in all video frames of the query sample with each video frame in all video frames of another sample, thereby obtaining multiple pairs of video frames; wherein, the frame-level similarity of each pair of video frames in the frame-level similarity of the multiple pairs of video frames is determined in the following manner: for each pair of video frames, forming a pair of block-level features by combining each block-level feature in all block-level features of one video frame with each block-level feature in all block-level features of another video frame, thereby obtaining multiple pairs of block-level features; calculating the block-level similarity between the two block-level features in each pair of block-level features, thereby obtaining multiple block-level similarities corresponding to each block-level feature of the query sample video frame; selecting the maximum block-level similarity from the multiple block-level similarities corresponding to each block-level feature of the query sample video frame, and averaging the sum of the maximum block-level similarities corresponding to all block-level features of the query sample video frame, thereby obtaining the frame-level similarity of each pair of video frames.

[0029] In some embodiments of the present invention, the method of determining the video-level similarity of each sample pair includes: selecting the maximum frame-level similarity from multiple frame-level similarities of multiple pairs of video frames corresponding to each video frame of the query sample, and averaging the sum of the maximum frame-level similarities corresponding to all video frames of the query sample to obtain the video-level similarity of each sample pair.

[0030] According to a second aspect of the present invention, a video retrieval system is provided, comprising: a trained video similarity learning model obtained based on the method described in the first aspect of the present invention, used to extract feature vectors of multiple videos and feature vectors of an input query video; and a video retrieval module used to calculate the video-level similarity between the feature vectors of each video and the feature vector of the query video, and to select a predetermined number of videos with the highest video-level similarity as retrieval results based on the video-level similarity of the multiple videos.

[0031] According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to implement the steps of the method according to any one of the first aspects of the present invention by executing the executable instructions.

[0032] Compared with the prior art, the advantages of the present invention are as follows:

[0033] In the training method of this invention, firstly, multiple sample pairs related to the query sample are constructed. Based on the video similarity learning model, block-level feature sequences of each video frame of each sample in the sample pair are extracted, avoiding constraints on video duration. This flexible method allows the model to retain the original spatiotemporal overall structure of the video, thereby ensuring a comprehensive representation of video features. Secondly, a hierarchical similarity measurement method using frame-level and video-level similarity can accurately measure the video-level similarity between two videos of different durations. Finally, the preset loss function, which calculates the total loss based on the corresponding video-level similarity of each sample pair, can more effectively promote backpropagation and parameter updates during the model's learning process, improving the AP metric optimization effect. The model obtained by this training method has a significant improvement in the AP metric, providing a good comprehensive evaluation standard for ranking positive and negative samples in downstream tasks. Attached Figure Description

[0034] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0035] Figure 1 This is a flowchart illustrating a method for performing one iteration of training a video similarity learning model according to an embodiment of the present invention.

[0036] Figure 2This is a flowchart illustrating a method for calculating the frame-level similarity of each pair of video frames according to an embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0038] As mentioned in the background section, existing AP optimization methods suffer from poor optimization performance. Furthermore, since videos have an additional time dimension compared to images, there are significant difficulties in feature representation and similarity measurement. This leads to even worse AP optimization results when applying existing AP optimization methods to models in the video domain.

[0039] To address the aforementioned issues, the inventors propose a training method for a video similarity learning model, comprising: Step S1, obtaining a training set, which includes multiple original samples and corresponding labels, wherein an original sample is a video segment, a video segment includes multiple video frames, and the labels indicate the category of the corresponding original sample; Step S2, using the training set, performing multiple iterations of training on the video similarity learning model to obtain the trained video similarity learning model. In each iteration of training, the inventors first address the significant difficulties in video feature representation by using the labels as a basis. For each category, each original sample in the training set is used as a query sample, other original samples in that category are used as positive samples, and original samples from other categories are used as negative samples. Each positive sample and each negative sample forms a sample pair with the query sample, resulting in multiple sample pairs. Then, the model is used to extract the feature vector of each sample in each sample pair. The feature vector includes the block-level feature sequence of each video frame of the corresponding sample. This invention extracts the block-level feature sequence of each video frame of each sample based on this model, avoiding constraints on video duration. This flexible method allows the model to preserve the original spatiotemporal structure of the video, thereby ensuring a comprehensive representation of video features. At the same time, it simplifies the feature representation process for videos of different times and durations. Secondly, addressing the difficulty of measuring video similarity, the inventors construct multiple pairs of video frames for each sample pair. Based on the block-level feature sequence of each video frame in each sample pair, they determine the frame-level similarity of each pair of video frames. Then, based on the frame-level similarity of the multiple pairs of video frames, they determine the video-level similarity between the two samples in the sample pair. This invention, using a hierarchical similarity measurement method of frame-level and video-level similarity, can accurately measure the video-level similarity between two videos of different durations. Finally, addressing the problem of poor optimization performance in existing AP optimization methods, this invention uses a pre-defined loss function to calculate the total loss based on the video-level similarity of multiple sample pairs corresponding to each query sample in all query samples. The model parameters are updated based on the total loss, more effectively promoting the AP performance optimization during the learning process. Furthermore, the model trained using this method shows a significant improvement in AP, providing a good comprehensive benchmark for ranking positive and negative samples in downstream tasks.

[0040] To better understand this invention, the following detailed description, in conjunction with the accompanying drawings and specific embodiments, covers four aspects: training set, model structure, model training process, and application scenarios.

[0041] I. Training Set

[0042] According to one embodiment of the present invention, an existing video dataset or a self-made video dataset is first acquired. A subset is extracted from the video dataset as a training set. The subset includes multiple original samples and corresponding labels. The original sample is a video segment, which includes multiple video frames. The label indicates the category of the corresponding original sample. Illustratively, the existing video dataset can be the HMDB51 dataset, UCF101 dataset, THUMOS14 dataset, or EVVE dataset. Each dataset includes multiple original samples, and each original sample is a video segment, which includes multiple video frames. The label indicates the category of the corresponding original sample. Specifically, the HMDB51 dataset defines 51 categories, such as "running" and "jumping," the UCF101 dataset defines 101 categories, such as "playing basketball," "long jump," and "cycling," the THUMOS14 dataset defines 20 categories, such as "diving," "football penalty kick," and "golf swing," and the EVVE dataset defines 13 categories.

[0043] II. Model Structure

[0044] According to an embodiment of the present invention, a video similarity learning model includes an encoder and a feature mapping layer constructed using a deep neural network. The model extracts the feature vector of each sample by: extracting features from multiple image blocks of each video frame of each sample using the encoder to obtain multiple original block-level features; and performing feature mapping on the multiple original block-level features of each video frame using the feature mapping layer to obtain a block-level feature sequence, which includes multiple block-level features. The block-level feature sequence of multiple video frames of each sample constitutes the feature vector of that sample. This embodiment achieves at least the following beneficial technical effects: The present invention extracts the block-level feature sequence of each video frame of a video segment using an encoder, avoiding constraints on video duration. This flexible method allows the model to retain the original spatiotemporal structure of the video, thereby ensuring a comprehensive representation of video features. Furthermore, compared to traditional methods that forcibly cut videos within a training batch into equal lengths, leading to information loss, the feature extraction method of the present invention allows for flexible batch training that varies with video duration, improving the model's generalization ability.

[0045] According to one embodiment of the present invention, the encoder of the model adopts a ResNet50 backbone network or a VGG16 network, and the feature mapping layer includes a fully connected network and a multilayer perceptron. The multilayer perceptron adopts a single-layer hidden layer, but multilayer perceptrons with different numbers of layers can also be used. Specifically, features of multiple image blocks of each video frame for each sample are extracted through the ResNet50 backbone network or VGG16 network to obtain multiple raw block-level features. The multiple raw block-level features of each video frame are sequentially processed by a fully connected network and a multilayer perceptron to obtain a block-level feature sequence, which includes multiple block-level features. The block-level feature sequence of multiple video frames for each sample constitutes the feature vector of that sample.

[0046] According to one embodiment of the present invention, the method of extracting each sample by model includes: extracting features of E image blocks of each video frame of the sample with dimensions AxBxCxD by an encoder to obtain E original block-level features with dimension F, where A represents the number of video frames, B represents the height of the video frame, C represents the width of the video frame, and D and F both represent the number of channels; performing feature mapping on all the original block-level features of each video frame by a feature mapping layer to obtain a block-level feature sequence with dimension ExF, the block-level feature sequence including E block-level features with dimension F; wherein, the block-level feature sequence of A video frames of each sample constitutes a feature vector with dimension AxExF.

[0047] Indicatively, the encoder extracts features from nine image blocks of each video frame of a 30x224x224x3 sample, resulting in nine original block-level features of dimension 512. Here, 30 represents the number of video frames, the height and width of each video frame are both 224, 3 represents the number of channels in the sample, and 512 represents the number of channels in the original block-level features. A feature mapping layer is then used to map all the original block-level features of each video frame, resulting in a block-level feature sequence of dimension 9x512. This block-level feature sequence includes nine block-level features of dimension 512. The block-level feature sequence of the 30 video frames for each sample constitutes a feature vector of dimension 30x9x512.

[0048] According to another embodiment of the present invention, the video similarity learning model may also include only an encoder constructed using a deep neural network. The encoder adopts a ResNet50 backbone network or a VGG16 network. The encoder is used to extract features of multiple image blocks of each video frame of each sample to obtain a block-level feature sequence including multiple original block-level features. The block-level feature sequence of multiple video frames of each sample constitutes the feature vector of the sample.

[0049] III. Model Training Process

[0050] According to an embodiment of the present invention, a training method for a video similarity learning model is provided, the method comprising steps S1 and S2. To better understand the present invention, each step will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] In step S1, a training set is obtained, which includes multiple original samples and corresponding labels. The original sample is a video segment, which includes multiple video frames. The label indicates the category of the corresponding original sample.

[0052] In step S2, the video similarity learning model is trained iteratively multiple times using the training set in the following manner to obtain the trained video similarity learning model.

[0053] According to one embodiment of the present invention, in one iteration of training, see Figure 1 This is a flowchart illustrating the method for iteratively training a video similarity learning model. It includes steps S21, S22, S23, and S24.

[0054] In step S21, based on the label, for each category, each original sample in the training set of that category is used as a query sample, other original samples in that category are used as positive samples, and original samples in other categories are used as negative samples. Each positive sample and each negative sample are paired with the query sample to obtain multiple sample pairs.

[0055] According to one embodiment of the present invention, N videos are randomly sampled from a given video data training set to obtain N original samples. The N videos contain d categories, and each category contains M original samples, i.e., N = d × M. For each category, each original sample in that category of the training set is used as a query sample. For any query sample V... k M-1 original samples belonging to the same category are taken as positive samples, thus forming a positive sample set V. k+ The NM original samples from other categories are used as negative samples to form a negative sample set V. k- Each positive sample and each negative sample form a sample pair with the query sample, resulting in N-1 sample pairs.

[0056] In step S22, the model is used to extract the feature vector of each sample in each sample pair. The feature vector includes the block-level feature sequence of each video frame of the corresponding sample.

[0057] According to one embodiment of the present invention, given two videos V and V′ in a sample pair, where V is the query sample and V′ is the other sample in the sample pair, for the a-th video frame V in V... a The b-th video frame V in V′ bAfter the model extracts two video frames, block-level feature sequences g(V) are obtained respectively. a and g(V′) b , g(V) a and g(V′) b Both are R×D dimensional tensors, where R is the number of block-level features and D is the number of channels.

[0058] In step S23, multiple pairs of video frames are constructed based on each sample pair. The frame-level similarity of each pair of video frames is determined according to the block-level feature sequence of each video frame of each sample in each sample pair. The video-level similarity between two samples in the sample pair is determined according to the frame-level similarity of the multiple pairs of video frames.

[0059] According to one embodiment of the present invention, constructing multiple pairs of video frames includes: for each sample pair, forming a pair of video frames by combining each video frame in all video frames of the query sample with each video frame in all video frames of the other sample, thus obtaining multiple pairs of video frames. Illustratively, given two videos V and V′ in a sample pair, video V includes T video frames, video V′ includes T′ video frames, the first video frame in video V forms a pair of video frames by combining each video frame in the T′ video frames of video V′, resulting in T′ pairs of video frames. Thus, the two videos V and V′ together constitute T*T′ pairs of video frames.

[0060] According to one embodiment of the present invention, the frame-level similarity of each pair of video frames in the above plurality of pairs of video frames is determined in the following manner, see [reference]. Figure 2 The flowchart shows the method for calculating the frame-level similarity of each pair of video frames, including steps a1, a2, and a3:

[0061] In step a1, for each pair of video frames, each block-level feature in all block-level features of one video frame is combined with each block-level feature in all block-level features of another video frame to form a pair of block-level features, resulting in multiple pairs of block-level features.

[0062] According to one embodiment of the present invention, for any pair of video frames of two videos V and V′ in a sample pair, for example, the a-th video frame V of video V. a and the b-th video frame V of video V′ b A pair of video frames consisting of '', the a-th video frame V a The block-level feature sequence g(V) a Includes R block-level features, and the b-th video frame V b The block-level feature sequence g(V′) of V′ b It includes E block-level features, and the block-level feature sequence g(V) is... a The first block-level feature and the block-level feature sequence g(V′) bEach of the E block-level features constitutes a pair of block-level features, resulting in E pairs of block-level features. Thus, the video frame V... a and video frame V b A total of R*E pairs of block-level features are formed, and R = E.

[0063] In step a2, the block-level similarity between the two block-level features in each pair of block-level features is calculated to obtain multiple block-level similarities corresponding to each block-level feature of the video frame of the query sample.

[0064] According to one embodiment of the present invention, the block-level similarity between two block-level features in each pair of block-level features can be calculated by tensor dot product of a pair of video frames, thereby obtaining the block-level similarity of multiple pairs of block-level features. The block-level similarity of multiple pairs of block-level features constitutes a block-level similarity matrix, with video frame V as the pivot. a and video frame V b Taking the R*E pairs of block-level features as an example, the block-level similarity of the R*E pairs of block-level features constitutes the block-level similarity matrix S(g(V)). a , g(V′) b The following is represented:

[0065]

[0066] Wherein, S(g(V) a , g(V′) b ) 1，1 Represents g(V) a The first block-level feature is related to g(V′). b The block-level similarity of the first block-level feature, g(V). a Let g(V′) represent the block-level feature sequence of the a-th video frame of the query sample. b Let S(g(V)) represent the block-level feature sequence of the b-th video frame of another sample. a , g(V′) b ) 1，E Represents g(V) a The first block-level feature is related to g(V′). b The block-level similarity of the E-th block-level feature, where E represents the total number of block-level features in the block-level feature sequence of the b-th video frame, S(g(V)). a , g(V′) b ) R，1 Represents g(V) a The R-th block-level feature and g(V′) b The block-level similarity of the first block-level feature, R represents the total number of block-level features in the block-level feature sequence of the a-th video frame, and S(g(V)). a , g(V′) b ) R，ERepresents g(V) a The R-th block-level feature and g(V′) b The block-level similarity of the Eth block-level feature.

[0067] According to one embodiment of the present invention, V a The block-level feature sequence g(V) a Any block-level feature and V b The block-level feature sequence g(V′) of V′ b The block-level similarity between any two block-level features can be calculated using the cosine similarity method. An illustrative calculation method is shown below:

[0068]

[0069] Wherein, S(g(V) a , g(V′) b ) i , j For g(V) a The i-th block-level feature and g(V′) b The cosine similarity (i.e., block-level similarity) of the j-th block-level feature. Represents g(V) a The i-th block-level feature, T denotes transpose, g(V′) bj Represents g(V′) b The j-th block-level feature. The method of this invention intuitively reflects the degree of block-level similarity between two block-level features.

[0070] In step a3, the maximum block-level similarity is selected from multiple block-level similarities corresponding to each block-level feature of the video frame of the query sample. The sum of the maximum block-level similarities corresponding to all block-level features of the video frame of the query sample is averaged to obtain the frame-level similarity of each pair of video frames. The technical solution of this embodiment can achieve at least the following beneficial technical effects: The similarity calculation method of the present invention selects the maximum block-level similarity from multiple block-level similarities corresponding to each block-level feature of the video frame of the query sample. For any block-level feature in the video frame of the query sample, the block-level feature with the highest block-level similarity in another sample in the query sample pair is selected. Then, the query results of all block-level features in the video frame of the query sample are averaged, reflecting the local spatial similarity.

[0071] According to one embodiment of the present invention, the frame-level similarity of each pair of video frames is determined as follows:

[0072]

[0073] Among them, CS p (g(V) a , g(V′)b S(g(V)) represents the frame-level similarity between the a-th video frame of query sample V and the b-th video frame of another sample V′, r represents the number of the r-th block-level feature in the block-level feature sequence of the a-th video frame of V, and e represents the number of the e-th block-level feature in the block-level feature sequence of the b-th video frame of V′. a , g(V′) b ) r，e This represents the block-level similarity of a pair of block-level features consisting of the r-th block-level feature of the a-th video frame and the e-th block-level feature of the b-th video frame.

[0074] According to one embodiment of the present invention, since a video consists of multiple video frames, multiple pairs of video frames can be constructed based on each sample pair. For example, the number of video frames for a sample pair of two samples V and y′ are T and T′ respectively. The two videos V and V′ constitute a total of T*T′ pairs of video frames. After calculating the frame-level similarity of all pairs of video frames using steps a1, a2, and a3 of the above embodiment, the following frame-level similarity matrix is ​​obtained:

[0075]

[0076] Among them, CS p (g(V)1, g(V′)1) represents the frame-level similarity between the first video frame of video V and the first video frame of video V′, g(V)1 represents the block-level feature sequence of the first video frame of video V, and g(V′)1 represents the block-level feature sequence of the first video frame of video V′. CS p (g(V)1,g(V′) T Let g(V′) represent the frame-level similarity between the first video frame of video V and the T′-th video frame of video V′. T ' represents the block-level feature sequence of the T'th video frame of video V', CS p (g(V) T Let g(V′)1) represent the frame-level similarity between the T-th video frame of video V and the 1-th video frame of video V′. T CS represents the block-level feature sequence of the T-th video frame of video V. p (g(V) T , g(V′) T ,) represents the frame-level similarity between the T-th video frame of video V and the T′-th video frame of video V′.

[0077] According to an embodiment of the present invention, the method for determining the video-level similarity of each sample pair based on the frame-level similarity matrix formed by the frame-level similarity of all pairs of video frames corresponding to a sample pair in the above embodiment includes: selecting the maximum frame-level similarity from multiple frame-level similarities of multiple pairs of video frames corresponding to each video frame of the query sample, and averaging the sum of the maximum frame-level similarities corresponding to all video frames of the query sample to obtain the video-level similarity of each sample pair. The technical solution of this embodiment can at least achieve the following beneficial technical effects: the present invention calculates the average hit rate, that is, for a video frame g(V) in the query sample V... i The algorithm retrieves the video frame with the highest frame-level similarity from g(V′), and then averages the results of all video frames in g(V). Therefore, video-level similarity reflects not only the spatial similarity (i.e., image similarity) between two videos, but also their temporal similarity. Finally, the present invention, using the above embodiments and sequentially employing block-level similarity, frame-level similarity, and video-level similarity measurement methods, can flexibly vary with video duration, accurately measuring the video-level similarity between two videos of different durations.

[0078] According to one embodiment of the present invention, the video-level similarity of each sample pair is determined as follows:

[0079]

[0080] Among them, CS f (g(V), g(V′)) represents the video-level similarity of the sample pair, g(V) represents the feature vector of the query sample V in the sample pair, g(V′) represents the feature vector of the other sample V′ in the sample pair, T represents the total number of video frames of the query sample, i represents the video frame number of the query sample, j represents the video frame number of the other sample, T′ represents the number of video frames of the other sample, CS p (g(V)i,g(V′) j Let g(V) represent the frame-level similarity between the i-th video frame of the query sample and the j-th video frame of another sample. i Let g(V′) represent the block-level feature sequence of the i-th video frame of the query sample. j This represents the block-level feature sequence of the j-th video frame of another sample. The video-level similarity value ranges from [0, 1]. The technical solution of this embodiment can achieve at least the following beneficial technical effects: The present invention CS f (g(V), g(V′)) calculates the average hit rate, which is the hit rate for a video frame g(V) in g(V). i , Find the video frame in g(V′) with the highest similarity, and then average the query results of all video frames in g(V). Therefore, CS f(g(V), g(V′)) reflects temporal similarity. Ultimately, CS f (g(V), g(V′)) is the video-level similarity mapping function that the model needs to learn.

[0081] In step S24, a preset loss function is used to calculate the total loss based on the video-level similarity of multiple sample pairs corresponding to each query sample in all query samples. The model parameters are then updated based on the total loss. This invention uses the Adam optimizer to update the model parameters by backpropagating along the model's neural network based on the total loss. This reduces the total loss in the next training iteration until the total loss is reduced to within a predetermined threshold and remains relatively stable, thus completing the training and obtaining the trained model.

[0082] According to one embodiment of the present invention, in one iteration of training, the preset loss function is as follows:

[0083]

[0084] Where N represents the total number of query samples, k represents the query sample number, and |S k+ | represents the number of sample pairs formed by the k-th query sample and all positive samples, s ki ∈S k+ S represents the video-level similarity between the k-th query sample and the i-th positive sample. k+ This represents the set of video-level similarities between the k-th query sample and all positive samples. s kj ∈S k- S represents the video-level similarity between the k-th query sample and the j-th negative sample. k- This represents the set of video-level similarities between the k-th query sample and all negative samples. And x < -δ, where x represents s kj -s ki τ represents the gradient hyperparameter, and δ represents the discrimination boundary hyperparameter. And x≥-δ, s kj ∈S k+ This represents the video-level similarity between the k-th query sample and the j-th positive sample. This invention proposes an AP optimization method (also known as SigLinear-AP) based on a preset loss function to minimize the total loss. The technical solution of this embodiment can achieve at least the following beneficial technical effects: The preset loss function in this invention is differentiable. At the same time, the loss function assigns appropriate gradients and discrimination boundaries based on the video-level similarity of the sample pairs corresponding to positive samples and the video-level similarity of the sample pairs corresponding to negative samples. The total loss calculated by this loss function is used to back-update the model parameters, reducing the probability of incorrect ranking of each positive sample and each negative sample in the downstream task. This promotes more effective backpropagation and parameter updates during the learning process, thereby improving the AP index optimization effect of the model.

[0085] According to one embodiment of the present invention, the gradient τ takes values ​​in the range of (0, 1), and the discrimination boundary δ takes values ​​in the range of [0, 1). Experiments show that the present invention makes the optimal parameter values ​​of the model τ = 0.03 and δ = 0.05. It should be understood that this is only for illustration, and the values ​​can also be τ = 0.028, τ = 0.033, δ = 0.047, δ = 0.052, etc. In actual applications, the values ​​may be slightly fluctuated around these values ​​depending on the different datasets, and the present invention does not impose any restrictions on this.

[0086] According to an embodiment of the present invention, the design principle of the preset loss function is as follows:

[0087] First, we define the original AP score. In the video similarity learning task, for a query sample V... k , the positive sample set V k+ Any positive sample V in i With V k The video-level similarity obtained by calculating similarity is denoted as s. ki ∈S k+ The set S consists of multiple video-level similarities calculated from all positive samples and query samples. k+ The negative sample set V k- Any negative sample V in i With V k The video-level similarity obtained by calculating similarity is denoted as S. ki ∈S k- The set S consists of multiple video-level similarities calculated from all negative samples and query samples. k- Based on the video-level similarity of multiple sample pairs corresponding to the query sample, the original AP score is determined, and the original AP score is defined as follows:

[0088]

[0089] Among them, stop P k (f) represents the original AP score of the k-th query sample, |Sk+ | represents the number of video-level similarities calculated between all positive samples and the query sample. V represents the positive sample i With query sample V k Video-level similarity s ki Rank in Sk+, the set of video-level similarities corresponding to all positive samples. V represents the positive sample i With query sample V k Video-level similarity S ki The set of video-level similarities S across all samples k+ ∪S k- The ranking is based on video-level similarity, with positive and negative samples ranked from highest to lowest. Defined as a ranking function, its form is: The meaning is: for a value s, iterate through all other values ​​s′ in the set S, and whenever there is a s′ greater than or equal to s, rank them. It increases by 1. Clearly, the ranking function... The value of is between 1 and N, where N is the number of values ​​in set S. s) is a step function, i.e. Therefore, in this invention equals s kj -s ki .

[0090] Secondly, due to Since AP is non-differentiable, the original AP score is also non-differentiable and cannot be used as the objective function during training. Therefore, AP needs to be optimized. Thus, to maximize the model's AP score during training, and since maximizing the AP score is equivalent to minimizing the AP risk, the relationship between AP score and AP risk is as follows:

[0091]

[0092] Among them, molecules S represents the corresponding negative sample set in the statistical analysis. k- The S value is greater than or equal to the corresponding value in the positive sample set. k+ The i-th video-level similarity x ki The number of negative samples, denominator The video-level similarity s represents the negative sample. kj The set of video-level similarities across all samples S k+ ∪S k- The ranking in Let represent the AP risk of the k-th query sample. Intuitively, the goal of AP optimization is to make the numerator as close to 0 as possible, that is, to minimize the AP risk.

[0093] Then, the AP risk is further derived and transformed, and the transformation process is as follows:

[0094]

[0095] Among them, make You can see The middle molecule is s kj ∈S k- The relevant function, the denominator is s kj ∈S k+ The relevant functions will be designed separately for these two parts in the following sections. The substitution function, which will change the denominator Convert to In the molecules Convert to x represents s kj -s ki τ represents the gradient hyperparameter, and δ represents the discrimination boundary hyperparameter.

[0096] Finally, the AP risk of the k-th query sample is represented in the following differentiable form:

[0097]

[0098] For each iteration, the AP risk is calculated based on the video-level similarity of multiple sample pairs corresponding to each query sample in all query samples to obtain the total loss. The total loss is then calculated end-to-end using the following pre-defined loss function, and the model parameters are learned and updated based on the total loss:

[0099]

[0100] According to another embodiment of the present invention, in a single iteration of training, the preset loss function can also be represented as follows:

[0101]

[0102] in,

[0103] IV. Application Scenarios

[0104] According to an embodiment of the present invention, a video retrieval system is provided, comprising: a trained video similarity learning model obtained based on the method of the above embodiment, used to extract feature vectors of multiple videos and the feature vector of an input query video; and a video retrieval module used to calculate the video-level similarity between the feature vectors of each video and the feature vector of the query video, and to select a predetermined number of videos with the highest video-level similarity as the retrieval results based on the video-level similarity of the multiple videos. This invention can be applied to many downstream tasks of video similarity learning, and has wide applications, such as video retrieval, video recommendation, video duplicate segment detection, and near-duplicate video detection.

[0105] To verify the performance of the model of this invention, the inventors conducted the following comparative experiments:

[0106] Using the HMDB51, THUMOS14, UCF101, and EVVE datasets, subsets are extracted from each dataset to obtain corresponding training and test sets. The video similarity learning model of this invention is then trained using a pre-defined loss function. Subsequently, the test set is input into the trained video similarity learning model of this invention (denoted as VSAPO) and existing models to obtain the scores of the model of this invention and existing models on the mAP and Recall@K metrics. The existing technologies include TCA, ViSiL, DnS(Stu A), DnS(Stu B), and S2VS. TCA is a temporal aggregation model, ViSiL is a spatiotemporal similarity learning model, DnS(Stu A) is a distillation selection model (attention mechanism version), DnS(Stu B) is a distillation selection model (binarization version), S2VS is a self-supervised video similarity learning model, and VSAPO is the AP-optimized video similarity learning model of this invention. mAP represents average precision, Recall@K represents the recall rate of the top K queries (ranging from 0 to 100 percentages, with higher values ​​indicating better model performance), Recall@1 represents the recall rate of the top 1 query, and Recall@8 represents the recall rate of the top 8 queries. Finally, the performance comparison between the model of this invention and the latest existing models is shown in Tables 1 and 2 below. Table 1 below shows the performance comparison results between the model of the present invention and the existing model on the datasets HMDB51 and THUMOS14, and Table 2 below shows the performance comparison results between the model of the present invention and the existing model on the datasets UCF101 and EVVE.

[0107] Table 1: Performance comparison between the model of this invention and existing models on HMDB51 and THUMOS14

[0108]

[0109] According to the data in Table 1 above, the model of this invention has a significant improvement in mAP and recall compared to other existing models on the HMDB51 dataset. On the THUMOS14 dataset, the recall of the model of this invention is not much different from other existing models, and the mAP is also improved. This shows that the performance of the model of this invention is greatly improved compared to existing models on the HMDB51 and THUMOS14 datasets.

[0110] Table 2: Performance comparison of the proposed model with existing models on datasets UCF101 and EVVE.

[0111]

[0112] According to the data in Table 2 above, under the datasets UCF101 and EVVE, the model of this invention has a significant improvement in mAP compared to other existing models when the recall rate is not much different. This indicates that the model of this invention has a significant performance improvement over existing models under the datasets UCF101 and EVVE.

[0113] Using the HMDB51 and UCF101 datasets, subsets are extracted from each dataset to obtain corresponding training and test sets. The video similarity learning model of this invention is trained using the pre-defined loss function (denoted as SigLinear AP). Similarly, the same training set is used to train video similarity learning models using other loss functions. Subsequently, the test set is input into the video similarity learning model (denoted as VSAPO) trained using the pre-defined loss function of this invention, and the test set is also input into video similarity models trained using other existing loss functions to obtain the scores of different video similarity learning models trained under different loss functions on the mAP and Recall@K metrics. Other existing loss functions include CE, Focal, Contrastive, Triplet, InfoNCE, FastAP, BlackBoxAP, Listwise, Smooth AP, and SigLinear AP. Specifically, CE is the cross-entropy loss function, Focal is the focus loss function, Contrastive is the contrastive loss function, Triplet is the triplet loss function, InfoNCE is the information-noise contrastive estimation loss function, FastAP is the fast average precision loss function, BlackBoxAP is the black-box average precision loss function, Listwise is the list-based loss function, Smooth AP is the smooth average precision loss function, and SigLinear AP is the smooth linear average precision loss function. The final results are shown in Table 3 below, which compares the performance of models trained using the loss function of this invention with other existing loss functions.

[0114] Table 3: Performance Comparison of Models Trained Using the Loss Function of the Present Invention and Existing Loss Functions

[0115]

[0116]

[0117] According to the data in Table 3 above, under the datasets HMDB51 and UCF101, the model trained by the loss function of this invention is comparable to the model trained by other existing loss functions in terms of recall. However, the model trained by the loss function of this invention has a significant improvement in mAP, indicating that the loss function of this invention has a better optimization effect on the model's mAP and greatly improves the model performance.

[0118] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0119] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0120] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0121] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A training method for a video similarity learning model, characterized in that, include: Obtain the training set, which includes multiple original samples and corresponding labels. The original sample is a video segment, which includes multiple video frames. The labels indicate the category of the corresponding original sample. Using the training set, the video similarity learning model is trained iteratively multiple times in the following manner to obtain the trained video similarity learning model: Based on the labels, for each category, each original sample in the training set of that category is used as a query sample, other original samples in that category are used as positive samples, and original samples in other categories are used as negative samples. Each positive sample and each negative sample are paired with the query sample to obtain multiple sample pairs. The model is used to extract the feature vector of each sample in each sample pair. The feature vector includes the block-level feature sequence of each video frame of the corresponding sample. Multiple pairs of video frames are constructed based on each sample pair. The frame-level similarity of each pair of video frames is determined based on the block-level feature sequence of each video frame of each sample in each sample pair. The video-level similarity between two samples in the sample pair is determined based on the frame-level similarity of the multiple pairs of video frames. The total loss is calculated using a preset loss function based on the video-level similarity of multiple sample pairs corresponding to each query sample in all query samples. The model parameters are then updated based on the total loss. The preset loss function is as follows: ， in, Indicates the number of samples queried. Indicates the query for sample number. Indicates the first The number of sample pairs consisting of a query sample and all positive samples. Indicates the first The query sample and the first Video-level similarity of sample pairs consisting of positive samples. Indicates the first The set of video-level similarities between each query sample and all positive samples. , , Indicates the first The query sample and the first Video-level similarity of sample pairs consisting of negative samples. Indicates the first The set of video-level similarities between a query sample and all negative samples. , express , Represents the gradient hyperparameter. This represents the discrimination boundary hyperparameters. , Indicates the first The query sample and the first Video-level similarity of sample pairs consisting of positive samples. .

2. The method according to claim 1, characterized in that, The model includes an encoder and a feature mapping layer constructed using a deep neural network, and the feature vectors are obtained in the following ways: The encoder extracts features from multiple image blocks of each video frame for each sample, resulting in multiple raw block-level features; The feature mapping layer performs feature mapping on multiple original block-level features of each video frame to obtain a block-level feature sequence, which includes multiple block-level features. In this sample, the block-level feature sequence of multiple video frames constitutes the feature vector of that sample.

3. The method according to claim 2, characterized in that, The feature vectors are obtained in the following ways: The encoder extracts features from E image blocks of each video frame of the sample with dimensions AxBxCxD, resulting in E raw block-level features with dimension F, where A represents the number of video frames, B represents the height of the video frame, C represents the width of the video frame, and D and F both represent the number of channels. The feature mapping layer performs feature mapping on all the original block-level features of each video frame to obtain a block-level feature sequence of dimension ExF. The block-level feature sequence includes E block-level features of dimension F. In this sample, the block-level feature sequence of A video frames constitutes a feature vector with dimension AxExF.

4. The method according to claim 2, characterized in that, The encoder uses a ResNet50 backbone network or a VGG16 network, and the feature mapping layer includes a fully connected network and a multilayer perceptron.

5. The method according to claim 2, characterized in that, The construction of multiple pairs of video frames includes: for each sample pair, forming a pair of video frames by combining each video frame in all video frames of the query sample with each video frame in all video frames of another sample, thus obtaining multiple pairs of video frames; The frame-level similarity of each pair of video frames is determined in the following manner: For each pair of video frames, each block-level feature in all block-level features of one video frame is combined with each block-level feature in all block-level features of another video frame to form a pair of block-level features, resulting in multiple pairs of block-level features. Calculate the block-level similarity between the two block-level features in each pair of block-level features to obtain multiple block-level similarities corresponding to each block-level feature of the video frame of the query sample. Select the maximum block similarity from the multiple block similarities corresponding to each block-level feature of the video frame of the query sample, and average the sum of the maximum block similarities corresponding to all block-level features of the video frame of the query sample to obtain the frame-level similarity of each pair of video frames.

6. The method according to claim 5, characterized in that, The methods for determining the video-level similarity of each sample pair include: Select the maximum frame-level similarity from multiple pairs of video frames corresponding to each video frame of the query sample, and average the sum of the maximum frame-level similarities for all video frames of the query sample to obtain the video-level similarity of each sample pair.

7. A video retrieval system, characterized in that, include: A trained video similarity learning model obtained based on the method of any one of claims 1-6 is used to extract feature vectors of multiple videos and feature vectors of the input query video. The video retrieval module is used to calculate the video-level similarity between the feature vector of each video and the feature vector of the query video, and select a predetermined number of videos with the highest video-level similarity as the retrieval results.

8. A computer-readable storage medium, characterized in that, It contains a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method according to any one of claims 1 to 6 by executing the executable instructions.

Images