Video clip retrieval method and device based on comparative learning, equipment and medium

Abstract

The invention provides a video clip retrieval method and device based on comparative learning, equipment and a medium, and relates to the technical field of video retrieval. According to the method, by adopting the pre-trained video clip retrieval model, the first text feature corresponding to the video retrieval text and the first video feature in the target video data can be efficiently extracted from the text and the video content, and the feature similarity between the features is calculated by applying the comparative learning strategy; and thus, related video clips can be accurately matched. According to the method, the dependence on computing resources is reduced, the resource consumption in the retrieval process is reduced, the retrieval speed and accuracy are improved, and the practicability and efficiency of video retrieval are enhanced.

Description

Technical Field

[0001] The present application relates to the field of video retrieval technology, and in particular to a video clip retrieval method, device, equipment and medium based on contrastive learning. Background Art

[0002] Video clip retrieval is an important multimedia task with a wide range of applications. It involves retrieving video clips related to a text query from a large video repository. For example, text-based video clip retrieval can be applied in the customer service domain to provide more accurate and efficient customer support. Customers can query for solution videos by using text describing the problem, thereby resolving queries and problems.

[0003] However, video retrieval involves a lot of computation, especially in the feature extraction and matching process. The large-scale feature extraction and indexing performance requirements consume a lot of computing resources, affecting the real-time and efficiency of retrieval.

[0004] Therefore, how to improve the efficiency of video clip retrieval has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a video clip retrieval method, apparatus, device and medium based on contrastive learning, aiming to improve the accuracy of laughter detection.

[0006] In a first aspect, the present application provides a video clip retrieval method based on contrastive learning, the video clip retrieval method based on contrastive learning comprising the following steps:

[0007] Obtain the video search text and target video data input by the target user;

[0008] Based on a pre-trained video segment retrieval model, extracting a first text feature corresponding to the video retrieval text and a first video feature corresponding to at least one video segment in the target video data respectively;

[0009] Based on a contrastive learning strategy, the feature similarity between the first text feature and the first video feature corresponding to each of the video clips is calculated to determine a target video clip that matches the video retrieval text.

[0010] In a second aspect, the present application further provides a video segment retrieval device based on contrastive learning, the video segment retrieval device based on contrastive learning comprising:

[0011] A data acquisition module, used to acquire the video search text input by the target user and the target video data;

[0012] A feature extraction module, used for extracting a first text feature corresponding to the video retrieval text and a first video feature corresponding to at least one video segment in the target video data based on a pre-trained video segment retrieval model;

[0013] The segment retrieval module is used to calculate the feature similarity between the first text feature and the first video feature corresponding to each of the video segments based on a contrastive learning strategy, and determine a target video segment that matches the video retrieval text.

[0014] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the video clip retrieval method based on contrastive learning as described above are implemented.

[0015] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the video clip retrieval method based on contrastive learning as described above are implemented.

[0016] The present application provides a video clip retrieval method, device, computer equipment and storage medium based on contrastive learning. The method of the present application includes obtaining a video retrieval text and target video data input by a target user; based on a pre-trained video clip retrieval model, respectively extracting a first text feature corresponding to the video retrieval text and a first video feature corresponding to at least one video clip in the target video data; based on a contrastive learning strategy, calculating the feature similarity between the first text feature and the first video feature corresponding to each of the video clips, and determining a target video clip that matches the video retrieval text. In the above manner, the present application can efficiently extract the first text feature corresponding to the video retrieval text and the first video feature in the target video data from the text and video content by adopting a pre-trained video clip retrieval model, and use a contrastive learning strategy to calculate the feature similarity between the features, so as to accurately match the relevant video clips. It not only reduces the dependence on computing resources and reduces the resource consumption in the retrieval process, but also improves the retrieval speed and accuracy, and enhances the practicality and efficiency of video retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A schematic diagram of a flow chart of a first embodiment of a video clip retrieval method based on contrastive learning provided by the present application;

[0019] Figure 2 A schematic diagram of a video retrieval process of a video clip retrieval model provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of a flow chart of a second embodiment of a video clip retrieval method based on contrastive learning provided by the present application;

[0021] Figure 4 A schematic diagram of an extended flow chart of the second step of the second embodiment of the video clip retrieval method based on contrastive learning provided in an embodiment of the present application;

[0022] Figure 5 It is a structural schematic diagram of a first embodiment of a video clip retrieval device based on contrastive learning provided by the present application;

[0023] Figure 6 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0024] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0027] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0028] Please refer to Figure 1 , Figure 1 A flowchart of a first embodiment of a video clip retrieval method based on contrastive learning provided in the present application.

[0029] like Figure 1As shown, the video clip retrieval method based on contrastive learning includes steps S101 to S103.

[0030] S101, obtaining a video search text and target video data input by a target user;

[0031] In one embodiment, the text input of the video segment retrieval system generally refers to the video retrieval text submitted by the user, which describes the content that the user hopes to find in the target video data.

[0032] In one embodiment, in order for the retrieval system to accurately return relevant video clips, the video retrieval text needs to follow certain format and content specifications. Generally, the video retrieval text should be as concise as possible and directly express the user's retrieval intention; use specific words and phrases to avoid vague descriptions. Use keywords to describe the video content, such as events, people, places, actions, etc.

[0033] S102, based on a pre-trained video segment retrieval model, respectively extracting a first text feature corresponding to the video retrieval text and a first video feature corresponding to at least one video segment in the target video data;

[0034] In one embodiment, if Figure 2 As shown, first, the target video data and video retrieval text are subjected to data enhancement such as video compression and random front and back cropping, and then feature encoding is performed. The video clip features use the CLIP model as a teacher model and perform distillation, and the text features directly use the CLIP model. Finally, the video clip and text features are compared and learned, and retrieval enhancement is performed in the process of comparative learning.

[0035] In one embodiment, the CLIP (Contrastive Language-Image Pre-Training) model is a multimodal pre-trained neural network, which is an effective and scalable method for learning from natural language supervision. The CLIP model has two modalities, one is a textual modality and the other is a visual modality. The core idea of ​​the CLIP model is to use a large amount of paired data of images and texts for pre-training to learn the alignment relationship between images and texts.

[0036] In one embodiment, the video clip retrieval model extracts the first text feature corresponding to the video retrieval text and the first video feature in the target video data through a text feature extraction model structure and a video feature extraction model structure, so as to identify the video clip that matches the video retrieval text by comparing the first text feature and the first video feature.

[0037] S103: Based on a contrastive learning strategy, calculate the feature similarity between the first text feature and the first video feature corresponding to each of the video clips, and determine a target video clip that matches the video retrieval text.

[0038] In one embodiment, a similarity metric (such as cosine similarity) may be used to calculate the similarity between the first text feature and the first video feature corresponding to each video segment.

[0039] In one embodiment, a dynamic similarity threshold can be set to determine the degree of match between text features and video features. This threshold can be adjusted based on the performance of the model during the training process. Based on the similarity score, the video clip that exceeds the threshold is considered to be related to the text and is determined to be the video clip that best matches the text.

[0040] This embodiment provides a video clip retrieval method based on contrastive learning. The method of this application adopts a pre-trained video clip retrieval model to efficiently extract the first text feature corresponding to the video retrieval text and the first video feature in the target video data from the text and video content, and uses a contrastive learning strategy to calculate the feature similarity between the features, so as to accurately match the relevant video clips. It not only reduces the dependence on computing resources and reduces the resource consumption in the retrieval process, but also improves the retrieval speed and accuracy, and enhances the practicality and efficiency of video retrieval.

[0041] Please refer to Figure 3 , Figure 3 A flowchart of a second embodiment of a video clip retrieval method based on contrastive learning provided in the present application.

[0042] In this embodiment, Figure 3 As shown, based on the above Figure 1 In the illustrated embodiment, before S101, the following further includes:

[0043] S201, acquiring model training data, where the model training data includes first video data and first text data matching the first video data;

[0044] In one embodiment, video clip retrieval is usually based on the text description input by the user, and video clips similar to the text description are retrieved. That is, video clip retrieval is the process of matching the similarity between the text description features and the video clip features. Therefore, in the model training process, it is necessary to train the model to learn the feature similarity between the text description features and the video clip features.

[0045] In one embodiment, the model training data may select a video with a corresponding text description, and use the video as the first video data, and its corresponding text description as the first text data. For example, a scenic spot promotion video generally has an introduction to the scenic spot, and the video clip with the introduction to the scenic spot and its corresponding introduction text can be used as model training data with an associated relationship to constitute positive sample training data; other non-scenic spot video clips are used as model training data without text association to constitute negative sample training data.

[0046] Furthermore, based on a data enhancement algorithm, data processing is performed on the first video data, wherein the data enhancement algorithm includes video compression and front and back random cropping.

[0047] In one embodiment, after obtaining the model training data, the model training data can be preprocessed by data enhancement, thereby increasing the robustness and generalization ability of the model.

[0048] Exemplarily, data enhancement may include data processing operations such as video compression and random front and back cropping. For example, by video compression, the first video data is compressed to simulate different resolutions and bit rates. By random front and back cropping, video clips in the first video data are randomly selected to simulate queries of different time scales.

[0049] Generally, if time-related words exist in the first text data, the time sequence needs to be taken into consideration when performing corresponding video cropping.

[0050] Exemplarily, preprocessing of the model training data can be implemented by a video processing library (such as a multimedia video processing tool FFmpeg (Fast Forward Mpeg)).

[0051] In one embodiment, the data enhancement strategy uses a series of data enhancement techniques, including video compression and random front and back cropping, to simulate situations of different resolutions and time scales. This data enhancement strategy not only increases the robustness of the model, but also enriches the training data, which helps to improve the generalization ability and adaptability of the model.

[0052] S202: Based on the model training data, perform model training on the pre-trained model to obtain a first retrieval model;

[0053] In one embodiment, the pre-trained model is trained by a model training model, so that the pre-trained model can accurately identify positive samples and negative samples, that is, the corresponding video clip is retrieved according to the text description, and the retrieval accuracy of the video clip is greater than or equal to a preset accuracy threshold (such as 95%), and the pre-trained model can be used as the first retrieval model.

[0054] Furthermore, if Figure 4 As shown, step S202 specifically includes:

[0055] S2021. Based on a feature encoding algorithm, feature encode the model training data to obtain a second text feature corresponding to the first text data and a second video feature corresponding to the first video data;

[0056] In one embodiment, feature encoding is a key step in converting video clips and text queries into vector representations. In the embodiment of the present application, the CLIP model is used as an example for feature encoding.

[0057] Among them, the CLIP model includes a video encoder and a text encoder.

[0058] Video encoder: The input is a video V with N frames and the start and end frames of the video clip. The image encoder in the CLIP model (ViT) is used to extract the features of each frame of the video, but it ignores the temporal component of the video. Therefore, a temporal modeling module is added to bridge the gap between the image and the video. Then the features of the video clip frames are aggregated to obtain the encoding of the video clip X V ′∈R d .

[0059] Text encoder: Given a text Q containing multiple words, directly input it into the CLIP model (Sentence-BERT) pre-trained sentence encoder to obtain the sentence-level text representation X Q ′∈R d .

[0060] S2022. Perform feature distillation on the second video feature based on a knowledge distillation algorithm to obtain a third video feature;

[0061] In one embodiment, during the knowledge distillation process, in order to more effectively transfer knowledge from the teacher model to the student model, multiple technical steps can be taken to expand the feature distillation process. Specifically:

[0062] Optimize the distillation process through Logit standardization. By adjusting the distillation temperature, the output probability distribution of the teacher model can be smoother, making it easier for the student model to learn. By standardizing the logits of the teacher and the student, the output of the student model is closer to the output of the teacher model, thereby improving the distillation effect. This method helps to suppress overfitting, reduce the impact of label noise, and improve the robustness of the model.

[0063] During the distillation process, the distillation temperature can be dynamically adjusted according to the learning progress of the student model to balance the efficiency of knowledge transfer between the teacher model and the student model. This approach can make the distillation process more flexible and adapt to the learning needs of different stages.

[0064] Attention mechanisms, such as the Convolutional Block Attention Module (CBAM), are used to extract important information from feature maps, and the mean squared error (MSE) loss function is used to measure the feature differences between the teacher and student models. This approach performs well in tasks such as semantic segmentation and can improve the performance of the student model.

[0065] Generally, other technical steps can be used to extend the feature distillation process, such as multi-stage distillation strategy, spatiotemporal collaborative teaching, etc.

[0066] Furthermore, based on the knowledge distillation algorithm, the model features of the feature encoding algorithm are migrated to the student model so as to feature encode the first video data through the student model; based on the distillation loss function, the student model is optimized so that the similarity between the third video feature output by the student model and the second video feature extracted by the feature encoding algorithm is greater than or equal to a preset similarity threshold.

[0067] In one embodiment, the trained CLIP model is used as the teacher model to distill the video clip features. Through distillation, the student model can benefit from the knowledge and generalization ability of the teacher model, thereby improving the performance on smaller datasets, which is particularly useful for video clip retrieval tasks, and can also improve training efficiency and reduce computational costs.

[0068] In one embodiment, the student model is optimized by a distillation loss function to make it as close as possible to the encoding result of the teacher model. The distillation loss function is defined by KL divergence:

[0069]

[0070] Among them, P represents the output probability distribution of the teacher model, and Q represents the output probability distribution of the student model. The video clip obtained by the student model is represented by X V , the text is represented by X Q .

[0071] In one embodiment, the CLIP model is used as a teacher model and its knowledge is distilled into the student model to improve performance. This knowledge distillation method allows the student model to benefit from the knowledge and generalization ability of the teacher model, which helps to speed up the training process and improve retrieval accuracy.

[0072] In this embodiment, by learning the knowledge of the teacher model, the student model's accuracy in the video clip retrieval task is improved. The student model better understands the relationship between video and text through comparative learning, and thus has good retrieval performance on unseen data. Knowledge distillation can accelerate the convergence speed of the student model, allowing it to reach a higher performance level in a shorter time. Compared with directly training large and complex models, the student model of knowledge distillation can reduce the demand for computing resources while maintaining performance.

[0073] S2023. Based on the contrastive learning strategy, train the pre-trained model to perform association learning on the third video feature and the first text feature to obtain the first retrieval model.

[0074] In one embodiment, contrastive learning is to perform multimodal fusion and associative learning of video clip features and text features to facilitate subsequent retrieval.

[0075] In one embodiment, the model is trained using a contrastive learning strategy, and similar video and text features can be closer in feature space and dissimilar features can be further away by designing a proxy task and an objective function, and selecting a suitable loss function (such as cosine similarity loss or contrastive loss). Exemplarily, the proxy task and the objective function can be such that the similarity of positive samples calculated by the pre-trained model is greater than the similarity of negative samples.

[0076] In one embodiment, in multimodal learning, it is ensured that features of different modalities are semantically and structurally aligned to narrow the differences in feature representation between the teacher model and the student model.

[0077] Furthermore, based on the third video feature and the first text feature, first sample data corresponding to the first video data and second sample data corresponding to the first text data are extracted; based on the first sample data and the second sample data, comparative learning is performed on the pre-trained model, and the pre-trained model is trained to calculate positive sample similarity and negative sample similarity until the positive sample similarity is higher than the negative sample similarity, thereby obtaining the first retrieval model.

[0078] In one embodiment, for the pre-trained model, the contrastive learning strategy is trained by calculating the similarity of positive samples and negative samples. Positive samples refer to those sample pairs that are semantically similar or related, while negative samples refer to those sample pairs that are semantically irrelevant or different.

[0079] In one embodiment, by calculating the similarity between pairs of positive samples (e.g., using cosine similarity) and maximizing this value, and calculating the similarity between pairs of negative samples and minimizing this value, the model learns how to map similar samples more closely in feature space, while dissimilar samples are farther apart.

[0080] In one embodiment, the loss function of contrastive learning (such as InfoNCE loss) is used to measure the similarity difference between positive and negative sample pairs. The goal of model training is to minimize the loss function, thereby increasing the similarity of positive samples and reducing the similarity of negative samples. Through optimization algorithms such as back propagation and gradient descent, the model parameters are updated to better distinguish between positive and negative samples.

[0081] In a specific embodiment, for each query text, a positive sample (i.e., a video clip feature related to the query) and multiple negative samples are selected. Similarly, for each video clip, a positive sample and multiple negative samples are selected. The positive sample can be text related to the video clip; the negative sample can be other texts in the same batch, and the negative clips can be other clips of the video and randomly selected clips of other videos. The similarity scores between the positive samples and the negative samples are calculated, with the goal of making the scores of the positive samples higher than those of the negative samples.

[0082] In one embodiment, a contrast loss function is used to train the model so that it can distinguish positive samples from negative samples. The formula is as follows:

[0083]

[0084] Among them, NCE(X V , X Q ) and NCE(X Q , X V ) is the contrast loss between video and text similarity and text and video similarity. The specific formula is as follows:

[0085]

[0086] Among them, τ is a hyperparameter, and NCE(X Q , X V ) is defined to be symmetric.

[0087] In one embodiment, during the training process, the performance of the model needs to be evaluated regularly, and the model's ability to distinguish between positive and negative samples is usually measured using indicators such as accuracy, recall, or F1 score. Based on the evaluation results, the model parameters or training strategies may need to be adjusted to further improve the model performance.

[0088] In one embodiment, after sufficient training iterations, when the similarity of positive samples is stably higher than that of negative samples, the model has obtained better discrimination ability. At this point, the first retrieval model has been trained and can be used for actual video clip retrieval tasks, such as video content search, video recommendation and other application scenarios.

[0089] S203: Based on the retrieval enhancement strategy, optimize and train the first retrieval model to obtain the video clip retrieval model.

[0090] In one embodiment, in order to further improve the quality of retrieval, a retrieval enhancement strategy may be adopted, specifically including increasing the number of negative samples and using a dynamic threshold.

[0091] In one embodiment, increasing the number of negative samples refers to introducing more negative samples in the contrastive learning process. The negative samples include negative texts and negative segments. Negative texts are other texts in the same batch, and negative segments are other segments of the video and randomly selected segments of other videos.

[0092] In one embodiment, using a dynamic threshold refers to dynamically adjusting the similarity threshold according to the difficulty of the query text to balance the recall rate and the precision.

[0093] Furthermore, based on the first video data and the first text data, a video negative sample dataset and a text negative sample dataset are collected; based on a dynamic similarity threshold, the video negative sample dataset and the text negative sample dataset, comparative learning training is performed on the first retrieval model until the positive sample similarity is higher than the negative sample similarity, so as to obtain the video clip retrieval model.

[0094] In one embodiment, a negative sample dataset of video and text is constructed. These negative sample datasets should contain video clips that do not match the query text and text descriptions that do not match the video clips. Negative samples can be generated by various data enhancement techniques, or irrelevant video and text pairs can be selected from existing large-scale datasets.

[0095] In one embodiment, in contrastive learning training, a dynamic similarity threshold can be introduced to improve the discrimination ability of the model. The dynamic threshold can be adjusted according to the performance of the model during the training process to ensure that the model can more accurately distinguish between positive and negative samples. For example, the threshold can be dynamically adjusted according to the performance of the model in the previous training stage to adapt to the training needs of different stages.

[0096] In one embodiment, the first video data and the first text data are used as positive samples, together with the collected video negative sample data set and text negative sample data set, to perform contrastive learning training. During the training process, the model needs to learn to maximize the similarity between positive samples and minimize the similarity between negative samples. During the training process, a variety of loss functions can be used to optimize the performance of the model, such as contrast loss, triplet loss or InfoNCE loss, etc., to help the model better learn to distinguish between positive and negative samples.

[0097] In one embodiment, during the training process, the performance of the model needs to be regularly evaluated, including indicators such as accuracy, recall, and F1 score. Based on the evaluation results, the model parameters or training strategies may need to be adjusted to further improve the model performance.

[0098] In one embodiment, through dynamic similarity threshold and contrastive learning, the model can more accurately match video clips and text queries, improving the relevance of retrieval results. The introduction of negative samples enables the model to learn richer feature representations, making it more stable in the face of noise and interference.

[0099] In one embodiment, a retrieval enhancement strategy is introduced, including increasing the number of negative samples and dynamic threshold adjustment. By increasing the number of negative samples, the performance and robustness of the model are improved. At the same time, the dynamic threshold adjustment automatically adjusts the similarity threshold according to the difficulty of the query text to balance the recall rate and precision.

[0100] In this embodiment, by learning the knowledge of the teacher model, the accuracy of the student model in the video clip retrieval task is improved. The student model better understands the relationship between video and text through comparative learning, and thus has good retrieval performance on unseen data.

[0101] See also Figure 5 , Figure 5 It is a structural schematic diagram of a first embodiment of a video segment retrieval device based on contrastive learning provided in the present application. The video segment retrieval device based on contrastive learning is used to execute the aforementioned video segment retrieval method based on contrastive learning.

[0102] like Figure 5 As shown, the video segment retrieval device 300 based on contrastive learning includes: a data acquisition module 301 , a feature extraction module 302 and a segment retrieval module 303 .

[0103] The data acquisition module 301 is used to acquire the video search text and target video data input by the target user;

[0104] A feature extraction module 302 is used to extract a first text feature corresponding to the video retrieval text and a first video feature corresponding to at least one video segment in the target video data based on a pre-trained video segment retrieval model;

[0105] The segment retrieval module 303 is used to calculate the feature similarity between the first text feature and the first video feature corresponding to each of the video segments based on a contrastive learning strategy, and determine a target video segment that matches the video retrieval text.

[0106] In one embodiment, the video segment retrieval device 300 based on contrastive learning further includes a model training module, including:

[0107] A training data acquisition submodule, used to acquire model training data, wherein the model training data includes first video data and first text data matching the first video data;

[0108] A model training submodule, used to perform model training on the pre-trained model based on the model training data to obtain a first retrieval model;

[0109] The retrieval enhancement training submodule is used to optimize the training of the first retrieval model based on the retrieval enhancement strategy to obtain the video clip retrieval model.

[0110] In one embodiment, the model training submodule includes:

[0111] A feature encoding unit, configured to perform feature encoding on the model training data based on a feature encoding algorithm to obtain a second text feature corresponding to the first text data and a second video feature corresponding to the first video data;

[0112] A feature distillation unit, configured to perform feature distillation on the second video feature based on a knowledge distillation algorithm to obtain a third video feature;

[0113] A contrastive learning unit is used to train the pre-trained model to perform association learning on the third video feature and the first text feature based on the contrastive learning strategy to obtain the first retrieval model.

[0114] In one embodiment, the comparative learning unit includes:

[0115] a sample data extraction subunit, configured to extract first sample data corresponding to the first video data and second sample data corresponding to the first text data based on the third video feature and the first text feature;

[0116] The comparative learning subunit is used to perform comparative learning on the pre-trained model based on the first sample data and the second sample data, train the pre-trained model to calculate the positive sample similarity and the negative sample similarity until the positive sample similarity is higher than the negative sample similarity, and obtain the first retrieval model.

[0117] In one embodiment, the characteristic distillation unit comprises:

[0118] A model migration subunit, configured to migrate the model features of the feature encoding algorithm to the student model based on the knowledge distillation algorithm, so as to perform feature encoding on the first video data through the student model;

[0119] The student model optimization subunit is used to optimize the student model based on the distillation loss function so that the similarity between the third video feature output by the student model and the second video feature extracted by the feature encoding algorithm is greater than or equal to a preset similarity threshold.

[0120] In one embodiment, the retrieval enhancement training submodule includes:

[0121] A negative sample collection unit, configured to collect a video negative sample data set and a text negative sample data set based on the first video data and the first text data;

[0122] The model training unit is used to perform comparative learning training on the first retrieval model based on a dynamic similarity threshold, the video negative sample data set and the text negative sample data set until the positive sample similarity is higher than the negative sample similarity, thereby obtaining the video clip retrieval model.

[0123] In one embodiment, the model training module further includes:

[0124] A data enhancement unit is used to perform data processing on the first video data based on a data enhancement algorithm, wherein the data enhancement algorithm includes video compression and front and back random cropping.

[0125] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned video clip retrieval method embodiment based on contrast learning, and will not be repeated here.

[0126] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 6 Runs on the computer device shown.

[0127] See also Figure 6 , Figure 6 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0128] See also Figure 6 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0129] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any video segment retrieval method based on contrastive learning.

[0130] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0131] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any video segment retrieval method based on contrast learning.

[0132] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will appreciate that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0133] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0134] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0135] Obtain the video search text and target video data input by the target user;

[0136] Based on a pre-trained video segment retrieval model, extracting a first text feature corresponding to the video retrieval text and a first video feature corresponding to at least one video segment in the target video data respectively;

[0137] Based on a contrastive learning strategy, the feature similarity between the first text feature and the first video feature corresponding to each of the video clips is calculated to determine a target video clip that matches the video retrieval text.

[0138] In one embodiment, before obtaining the video search text input by the target user, the processor is further configured to implement:

[0139] Acquire model training data, where the model training data includes first video data and first text data matching the first video data;

[0140] Based on the model training data, the pre-trained model is trained to obtain a first retrieval model;

[0141] Based on the retrieval enhancement strategy, the first retrieval model is optimized and trained to obtain the video clip retrieval model.

[0142] In one embodiment, when the processor performs model training on the pre-trained model based on the model training data to obtain the first retrieval model, it is used to implement:

[0143] Based on a feature encoding algorithm, feature encoding is performed on the model training data to obtain a second text feature corresponding to the first text data and a second video feature corresponding to the first video data;

[0144] Based on a knowledge distillation algorithm, feature distillation is performed on the second video feature to obtain a third video feature;

[0145] Based on the contrastive learning strategy, the pre-trained model is trained to perform association learning on the third video feature and the first text feature to obtain the first retrieval model.

[0146] In one embodiment, when the processor implements the training of the pre-training model to perform association learning on the third video feature and the first text feature based on the contrastive learning strategy to obtain the first retrieval model, it is used to implement:

[0147] Based on the third video feature and the first text feature, extracting first sample data corresponding to the first video data and second sample data corresponding to the first text data;

[0148] Based on the first sample data and the second sample data, the pre-trained model is subjected to comparative learning, and the pre-trained model is trained to calculate positive sample similarity and negative sample similarity until the positive sample similarity is higher than the negative sample similarity, thereby obtaining the first retrieval model.

[0149] In one embodiment, when the processor implements the knowledge distillation algorithm to perform feature distillation on the second video feature to obtain the third video feature, it is configured to implement:

[0150] Based on the knowledge distillation algorithm, the model features of the feature encoding algorithm are transferred to the student model, so as to perform feature encoding on the first video data through the student model;

[0151] Based on the distillation loss function, the student model is optimized so that the similarity between the third video feature output by the student model and the second video feature extracted by the feature encoding algorithm is greater than or equal to a preset similarity threshold.

[0152] In one embodiment, when the processor implements the retrieval enhancement strategy to optimize and train the first retrieval model to obtain the video clip retrieval model, it is used to implement:

[0153] Based on the first video data and the first text data, collecting a video negative sample dataset and a text negative sample dataset;

[0154] Based on the dynamic similarity threshold, the video negative sample data set and the text negative sample data set, the first retrieval model is trained by comparative learning until the positive sample similarity is higher than the negative sample similarity, thereby obtaining the video clip retrieval model.

[0155] In one embodiment, before implementing the step of performing model training on the pre-trained model based on the model training data to obtain the first retrieval model, the processor is further configured to implement:

[0156] The first video data is processed based on a data enhancement algorithm, wherein the data enhancement algorithm includes video compression and front and back random cropping.

[0157] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the video clip retrieval methods based on contrastive learning provided in the embodiments of the present application.

[0158] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0159] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A video clip retrieval method based on contrastive learning, characterized in that: The method comprises: Obtain the video search text and target video data input by the target user; Based on a pre-trained video segment retrieval model, extracting a first text feature corresponding to the video retrieval text and a first video feature corresponding to at least one video segment in the target video data respectively; Based on a contrastive learning strategy, the feature similarity between the first text feature and the first video feature corresponding to each of the video clips is calculated to determine a target video clip that matches the video retrieval text.

2. The video clip retrieval method based on contrastive learning according to claim 1, characterized in that: Before obtaining the video search text input by the target user, the method further includes: Acquire model training data, where the model training data includes first video data and first text data matching the first video data; Based on the model training data, the pre-trained model is trained to obtain a first retrieval model; Based on the retrieval enhancement strategy, the first retrieval model is optimized and trained to obtain the video clip retrieval model.

3. The video clip retrieval method based on contrastive learning according to claim 2, characterized in that: The step of performing model training on the pre-trained model based on the model training data to obtain a first retrieval model includes: Based on a feature encoding algorithm, feature encoding is performed on the model training data to obtain a second text feature corresponding to the first text data and a second video feature corresponding to the first video data; Based on a knowledge distillation algorithm, feature distillation is performed on the second video feature to obtain a third video feature; Based on the contrastive learning strategy, the pre-trained model is trained to perform association learning on the third video feature and the first text feature to obtain the first retrieval model.

4. The video clip retrieval method based on contrastive learning according to claim 3, characterized in that: The step of training the pre-trained model to perform association learning on the third video feature and the first text feature based on the contrastive learning strategy to obtain the first retrieval model includes: Based on the third video feature and the first text feature, extracting first sample data corresponding to the first video data and second sample data corresponding to the first text data; Based on the first sample data and the second sample data, the pre-trained model is subjected to comparative learning, and the pre-trained model is trained to calculate positive sample similarity and negative sample similarity until the positive sample similarity is higher than the negative sample similarity, thereby obtaining the first retrieval model.

5. The video clip retrieval method based on contrastive learning according to claim 3, characterized in that: The step of performing feature distillation on the second video feature based on the knowledge distillation algorithm to obtain the third video feature includes: Based on the knowledge distillation algorithm, the model features of the feature encoding algorithm are transferred to the student model, so as to perform feature encoding on the first video data through the student model; Based on the distillation loss function, the student model is optimized so that the similarity between the third video feature output by the student model and the second video feature extracted by the feature encoding algorithm is greater than or equal to a preset similarity threshold.

6. The video clip retrieval method based on contrastive learning according to claim 2, characterized in that: The step of optimizing and training the first retrieval model based on the retrieval enhancement strategy to obtain the video clip retrieval model includes: Based on the first video data and the first text data, collecting a video negative sample dataset and a text negative sample dataset; Based on the dynamic similarity threshold, the video negative sample data set and the text negative sample data set, the first retrieval model is trained by comparative learning until the positive sample similarity is higher than the negative sample similarity, thereby obtaining the video clip retrieval model.

7. The video clip retrieval method based on contrastive learning according to claim 2, characterized in that: The step of performing model training on the pre-trained model based on the model training data to obtain the first retrieval model further includes: The first video data is processed based on a data enhancement algorithm, wherein the data enhancement algorithm includes video compression and front and back random cropping.

8. A video clip retrieval device based on contrastive learning, characterized in that: The video clip retrieval device based on contrastive learning comprises: A data acquisition module, used to acquire the video search text input by the target user and the target video data; A feature extraction module, used for extracting a first text feature corresponding to the video retrieval text and a first video feature corresponding to at least one video segment in the target video data based on a pre-trained video segment retrieval model; The segment retrieval module is used to calculate the feature similarity between the first text feature and the first video feature corresponding to each of the video segments based on a contrastive learning strategy, and determine a target video segment that matches the video retrieval text.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the video segment retrieval method based on contrastive learning as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the video segment retrieval method based on contrastive learning are implemented as described in any one of claims 1 to 7.

Images