Training method of video clip retrieval model and video clip retrieval method and device

Abstract

The embodiment of the invention relates to a training method of a video clip retrieval model and a video clip retrieval method and device, and the method comprises the steps: obtaining a pre-trained video clip retrieval model, obtaining a first video clip sample, a retrieval tag sample and a real time point location of the retrieval tag sample in the first video clip sample, obtaining a video feature corresponding to the first video clip sample and a retrieval tag feature corresponding to the retrieval tag sample, inputting the video feature and the retrieval tag feature into a video clip retrieval model, and obtaining a prediction time point location, and adjusting model parameters of the video clip retrieval model based on a first loss value between the video feature and the retrieval tag feature and a second loss value between the real time point location and the predicted time point location to obtain a target video clip retrieval model. According to the embodiment of the invention, the target video clip retrieval model of the video clip can be quickly and accurately obtained based on the retrieval tag.

Description

Technical Field

[0001] The present disclosure relates to the field of multimedia processing technology, and in particular to a video clip retrieval model training method, a video clip retrieval method and a device. Background Art

[0002] Typically, on digital video streaming platforms, online video websites, and TV stations, users often hope to obtain video clips of movies or TV series they want to watch through short tag words.

[0003] The existing traditional way to label video clips is either based on images, such as scene classification, or based on a single modality of dialogue text, such as using a text classification model to classify video dialogues. However, it is sometimes difficult to find specific tags based on a single modality, resulting in video clips based on tag retrieval that are not accurate and comprehensive enough. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a video clip retrieval model training method, a video clip retrieval method and a device.

[0005] The disclosed embodiment provides a method for training a video clip retrieval model, comprising: obtaining a pre-trained video clip retrieval model; obtaining a first video clip sample, a retrieval label sample and a real time point of the retrieval label sample in the first video clip sample; obtaining a video feature corresponding to the first video clip sample, and a retrieval label feature corresponding to the retrieval label sample; inputting the video feature and the retrieval label feature into the video clip retrieval model to obtain a predicted time point, and adjusting the model parameters of the video clip retrieval model based on a first loss value between the video feature and the retrieval label feature and a second loss value between the real time point and the predicted time point to obtain a target video clip retrieval model.

[0006] Optionally, the method also includes: obtaining a second video clip sample, retrieving text samples and annotation data; obtaining visual features and text features corresponding to the second video clip sample; adjusting model parameters of the video clip retrieval model to be trained based on the visual features, text features and the annotation data corresponding to the second video clip sample and a preset loss function to obtain the video clip retrieval model.

[0007] Optionally, the adjusting the model parameters of the video clip retrieval model to be trained based on the visual features, text features, the annotation data and a preset loss function corresponding to the second video clip sample to obtain the video clip retrieval model includes: performing text prediction on the text features corresponding to the second video clip sample based on the video clip retrieval model to be trained to obtain a masked text original identifier, calculating a first cross entropy loss value between the masked text original identifier and the actual text identifier in the annotation data based on a masked language model loss function, and adjusting the model parameters of the video clip retrieval model to be trained based on the first cross entropy loss value to obtain the video clip retrieval model; and / or performing video frame prediction on the visual features corresponding to the second video clip sample based on the video clip retrieval model to be trained to obtain a masked video frame original identifier, calculating a second cross entropy loss value between the masked video frame original identifier and the actual video frame identifier in the annotation data based on a masked image model loss function, and adjusting the model parameters of the video clip retrieval model to be trained based on the second cross entropy loss value to obtain the video clip retrieval model; and / or performing text prediction on the text features corresponding to the second video clip sample based on the video clip retrieval model to be trained to obtain a masked video frame original identifier, The method comprises the steps of: performing temporal position prediction on the visual features corresponding to the second video clip sample based on the video subtitle matching model to obtain a predicted time position, calculating a third cross entropy loss value between the predicted time position and the actual time position in the annotated data based on a video subtitle matching model loss function, and adjusting a model parameter of the to-be-trained video clip retrieval model based on the third cross entropy loss value to obtain the video clip retrieval model; and / or performing video frame sequence prediction on the visual features corresponding to the second video clip sample based on the to-be-trained video clip retrieval model to obtain a predicted video frame sequence, and calculating a third cross entropy loss value between the predicted video frame sequence and the actual video frame sequence in the annotated data based on a video frame sequence model loss function. A fourth cross entropy loss value, and based on the fourth cross entropy loss value, adjusting the model parameters of the video clip retrieval model to be trained to obtain the video clip retrieval model; and / or calculating the matching degree of the visual features and text features corresponding to the second video clip sample based on the video clip retrieval model to be trained to obtain the predicted matching degree, calculating the fifth cross entropy loss value between the predicted matching degree and the actual matching degree in the annotated data based on the text-to-video retrieval model loss function, and adjusting the model parameters of the video clip retrieval model to be trained based on the fifth cross entropy loss value to obtain the video clip retrieval model.

[0008] Optionally, obtaining the video features corresponding to the first video clip samples includes: performing frame extraction processing on the first video clip samples at a preset frame extraction frequency to obtain each video frame; performing visual feature extraction on each video frame based on a pre-trained image feature extraction model to obtain visual features corresponding to the first video clip samples; performing optical character recognition on each video frame based on image optical character recognition technology to obtain text information; performing text feature extraction on the text information based on a pre-trained text feature extraction model to obtain text features corresponding to the first video clip samples; and performing feature fusion on the visual features and text features corresponding to the first video clip samples to obtain the video features.

[0009] Optionally, the feature fusion of the visual features and text features corresponding to the first video clip sample to obtain the video feature includes: splicing the visual features and text features corresponding to the first video clip sample according to a preset feature splicing order to obtain the video feature; and / or, weighted averaging the visual features and text features corresponding to the first video clip sample according to preset feature weights to obtain the video feature; and / or, feature selection of the visual features and text features corresponding to the first video clip sample according to a preset screening strategy to obtain the video feature.

[0010] Optionally, the model parameters of the video clip retrieval model are adjusted based on a first loss value between the video feature and the label feature, and a second loss value between the real time point and the predicted time point to obtain a target video clip retrieval model, including: determining a target cross entropy loss value based on the first loss value and the second loss value; and adjusting the model parameters of the video clip retrieval model based on the target cross entropy loss value to obtain the target video clip retrieval model.

[0011] The disclosed embodiment provides a video clip retrieval method, comprising: obtaining a tag to be retrieved; inputting the tag to be retrieved into a target video clip retrieval model to obtain a target video clip; wherein the target video clip retrieval model is obtained according to the training method of the video clip retrieval model described in the aforementioned embodiment.

[0012] The disclosed embodiment also provides a training device for a video clip retrieval model, comprising: a first acquisition module, used to acquire a pre-trained video clip retrieval model; a second acquisition module, used to acquire a first video clip sample, a label sample to be retrieved, and the real time point of the label sample to be retrieved in the first video clip sample; a third acquisition module, used to acquire the video features corresponding to the video clip sample; a fourth acquisition module, used for the label features corresponding to the label sample; a training module, used to input the video features and the retrieved label features into the video clip retrieval model, acquire the predicted time point, and adjust the model parameters of the video clip retrieval model based on a first loss value between the video features and the label features, and a second loss value between the real time point and the predicted time point, to obtain a target video clip retrieval model.

[0013] The disclosed embodiment also provides a video clip retrieval device, comprising: a fifth acquisition module, used to obtain a tag to be retrieved; a retrieval module, used to search in a preset candidate video clip library according to the tag to be retrieved based on the target video clip retrieval model to obtain a target video clip; wherein the target video clip retrieval model is obtained according to the training method of the video clip retrieval model described in the aforementioned embodiment.

[0014] Optionally, the step of searching a preset candidate video segment library based on the target video segment retrieval model according to the to-be-retrieved tag to obtain the target video segment includes:

[0015] Obtaining a tag feature to be retrieved corresponding to the tag to be retrieved;

[0016] Inputting the tag feature to be retrieved and the candidate video feature corresponding to each candidate video segment in the candidate video segment library into the target video segment retrieval model to obtain a target time point;

[0017] The candidate video segment corresponding to the target time point is used as the target video segment.

[0018] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement a training method for a video clip retrieval model and a video clip retrieval method as provided in an embodiment of the present disclosure.

[0019] The embodiments of the present disclosure also provide a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the training method of the video clip retrieval model and the video clip retrieval method provided by the embodiments of the present disclosure.

[0020] The above-mentioned technical solution provided by the embodiment of the present disclosure fine-tunes the model parameters based on the sample data that meets the business needs on the basis of the pre-trained video clip retrieval model, so that the target video clip retrieval model can directly obtain the video clip quickly and accurately based on the retrieval label.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1 A flowchart of a method for training a video clip retrieval model provided in an embodiment of the present disclosure;

[0025] Figure 2 A flowchart of a method for training a video clip retrieval model provided in an embodiment of the present disclosure;

[0026] Figure 3 A schematic diagram of a training process of a video clip retrieval model provided in an embodiment of the present disclosure;

[0027] Figure 4 A schematic diagram of a training process of a video clip retrieval model provided in an embodiment of the present disclosure;

[0028] Figure 5 A schematic diagram of a flow chart of a video clip retrieval method provided by an embodiment of the present disclosure;

[0029] Figure 6 A schematic diagram of a video clip retrieval process provided by an embodiment of the present disclosure;

[0030] Figure 7 A schematic diagram of the structure of a training device for a video clip retrieval model provided in an embodiment of the present disclosure;

[0031] Figure 8 A schematic diagram of the structure of a video clip retrieval device provided in an embodiment of the present disclosure;

[0032] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0034] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0035] In existing video clip retrieval scenarios, users often hope to obtain the video clips of the movies or TV series they want to watch through short label words. Therefore, a label-based video clip retrieval model is needed for video clip retrieval.

[0036] The existing video clip retrieval usually takes a sentence and then locates the relevant video clips in the video. The embodiment of the present disclosure converts the input sentence into a tag word and then locates the relevant video clips in the video. It is understandable that if the video clip retrieval model based on the retrieval text is directly used for tag retrieval, the retrieval effect will be relatively poor. Therefore, the embodiment of the present disclosure fine-tunes the video clip retrieval model under the required tag system, obtains the target video clip retrieval model, and then performs video clip retrieval.

[0037] The training method of the video clip retrieval model of the embodiment of the present disclosure fine-tunes the model parameters based on the sample data that meets the business requirements on the basis of the pre-trained video clip retrieval model, thereby obtaining the target video clip retrieval model, which can directly obtain the video clip quickly and accurately based on the retrieval tag. The following is a detailed explanation:

[0038] Figure 1 The present invention provides a flow chart of a video clip retrieval model training method, which can be applied to electronic devices such as computers, mobile phones, tablet computers, televisions, servers, etc., without limitation. Figure 1 As shown, the method mainly includes the following steps S102 to S108:

[0039] Step S102, obtaining a pre-trained video clip retrieval model.

[0040] In the disclosed embodiment, the video clip retrieval model is a pre-trained model that can retrieve video clips based on a retrieval text (such as a sentence) to obtain video clips.

[0041] In the disclosed embodiments, video clip samples, retrieved text samples and annotated data may be acquired in advance, visual features and text features corresponding to the video clip samples may be acquired, and model parameters of a video clip retrieval model to be trained may be adjusted based on the visual features, text features, annotated data and a preset loss function to obtain a video clip retrieval model.

[0042] Step S104: obtaining the first video clip sample, the search tag sample, and the real time point of the search tag sample in the first video clip sample.

[0043] In the disclosed embodiment, the first video segment sample can be understood as video data for training a video segment retrieval model. Each first video segment sample has a corresponding retrieval label sample, and the retrieval label sample is used to obtain the first video segment sample by inputting the retrieval label sample into the target video segment retrieval model.

[0044] For example, a video may be split into video segment samples X1 to Xn every five seconds or every ten seconds. For example, the search text sample corresponding to the video segment sample X1 may be "trial", so that the video segment sample X1 may be retrieved by inputting the search text sample "trial".

[0045] In the embodiment of the present disclosure, retrieving the real time point of the label sample in the first video clip sample refers to retrieving the actual start time point and the actual end time point marked by the label sample in the first video clip sample.

[0046] Step S106, obtaining video features corresponding to the first video clip sample and retrieval tag features corresponding to the retrieval tag sample.

[0047] In the disclosed embodiment, the first video clip sample includes multimodal data such as visual information, audio and text information, and the visual features, text features, etc. corresponding to the first video clip sample can be obtained according to application needs.

[0048] In the embodiments of the present disclosure, there are many ways to obtain video features corresponding to the first video clip sample. As an example, the first video clip sample is subjected to frame sampling processing at a preset frame sampling frequency to obtain each video frame, visual features are extracted from each video frame based on a pre-trained image feature extraction model to obtain visual features corresponding to the first video clip sample, optical character recognition is performed on each video frame based on image optical character recognition technology to obtain text information, text features are extracted from the text information based on a pre-trained text feature extraction model to obtain text features corresponding to the first video clip sample, and the visual features and text features corresponding to the first video clip sample are feature fused to obtain video features.

[0049] As another example, multiple video frames corresponding to the video clip sample are obtained, and visual features of the multiple video frames are extracted through a convolutional neural network to obtain visual features corresponding to the video clip sample, audio information in the video clip sample is obtained, and the audio information is converted into text information based on speech-to-text technology, text features are extracted from the text information based on a pre-trained text feature extraction model to obtain text features, and visual features and text features corresponding to the first video clip sample are feature fused to obtain video features. The above are only examples, and settings can be selected according to application needs.

[0050] In the embodiments of the present disclosure, there are many ways to obtain the retrieval label features corresponding to the retrieval label samples. As an example, text features are extracted from the retrieval label samples through a neural network or an encoding model to obtain the retrieval label features; as another example, text features are extracted from the retrieval label samples based on a pre-trained text feature extraction model to obtain the retrieval text features. The above are only examples, and the settings can be selected according to application needs.

[0051] Step S108, input the video features and the retrieval label features into the video clip retrieval model, obtain the predicted time point, and adjust the model parameters of the video clip retrieval model based on the first loss value between the video features and the retrieval label features, and the second loss value between the real time point and the predicted time point to obtain the target video clip retrieval model.

[0052] In the disclosed embodiment, the pre-trained video clip retrieval model can process video features and retrieval label features to obtain predicted time points; wherein the predicted time points refer to the starting time points and ending time points of the predicted retrieval label samples in the first video clip sample processed by the video clip retrieval model.

[0053] In the disclosed embodiments, the first loss value of this training can be determined based on the similarity between the video feature and the retrieval label feature, such as performing dot product of the video feature and the retrieval label feature to obtain the similarity, or calculating the cosine value between the video feature and the retrieval label feature as the similarity, and inputting the similarity into a pre-set loss function for model training. The second loss value of this training can also be determined based on the similarity between the real time point and the predicted time point, such as taking the difference between the real time point and the predicted time point as the similarity, and inputting the similarity into a pre-set loss function for model training. Finally, the model parameters of the video clip retrieval model to be trained are adjusted according to the first loss value and the second loss value. For example, the first loss value and the second loss value determine the target cross entropy loss value to adjust the model parameters of the video clip retrieval model, so that the model parameters of the video clip retrieval model can be continuously adjusted based on multiple first video clip samples and retrieval text samples, and finally the target video clip retrieval model is obtained.

[0054] In the above embodiment, the model parameters are fine-tuned based on the sample data that meets the business requirements on the basis of the pre-trained video clip retrieval model, so that the target video clip retrieval model can directly obtain the video clip quickly and accurately based on the retrieval tag.

[0055] In some embodiments, the method further includes: obtaining a second video clip sample, retrieving a text sample and annotated data; obtaining visual features and text features corresponding to the second video clip sample; adjusting the model parameters of the video clip retrieval model to be trained based on the visual features, text features and annotated data corresponding to the second video clip sample and a preset loss function to obtain the video clip retrieval model.

[0056] In the disclosed embodiment, the second video clip sample can be understood as video data for training the video clip retrieval model. Each second video clip sample has a corresponding retrieval text sample, and the retrieval text sample is used to obtain the second video clip sample by inputting the retrieval text sample into the video clip retrieval model.

[0057] For example, a video can be split into video segment samples Y1 to Yn every five seconds or every ten seconds, such as the second video segment sample Y1. The search text sample corresponding to the second video segment sample Y1 can be "a group of sheep running on the grassland", so that the second video segment sample Y1 can be retrieved by inputting the search text sample "a group of sheep running on the grassland".

[0058] In the disclosed embodiment, the annotated data is real data that is pre-annotated for the second video clip sample, including, for example, actual text identifiers, actual video frame identifiers, actual time positions, actual video frame sequences, and actual visual text matching degrees.

[0059] In the embodiment of the present disclosure, the method for obtaining the visual features and text features corresponding to the second video clip sample may be, for example, performing frame extraction processing on the second video clip sample at a preset frame extraction frequency to obtain each video frame, performing visual feature extraction on each video frame based on a pre-trained image feature extraction model to obtain the visual features corresponding to the second video clip sample, obtaining audio information in the second video clip sample, and converting the audio information into text information based on a speech-to-text technology, and performing text feature extraction on the text information based on a pre-trained text feature extraction model to obtain the text features corresponding to the second video clip sample; for another example, obtaining multiple video frames corresponding to the second video clip sample, and performing visual feature extraction on the multiple video frames through a convolutional neural network to obtain the visual features corresponding to the second video clip sample, obtaining audio information in the video clip sample, and converting the audio information into first text information based on a speech-to-text technology, performing optical character recognition on each video frame based on an image optical character recognition technology to obtain second text information, and performing text feature extraction on the first text information and the second text information based on a pre-trained text feature extraction model to obtain the text features corresponding to the second video clip sample.

[0060] In the embodiment of the present disclosure, the model parameters of the video clip retrieval model to be trained are adjusted based on the visual features, text features and annotation data corresponding to the second video clip sample and a preset loss function. There are many ways to obtain the video clip retrieval model, and one or more of them can be selected according to actual application needs.

[0061] Among them, the open source model HERO is used as the video clip retrieval model to be trained, that is, as a pre-training model architecture, and pre-training is performed on data sets such as the second video clip sample, the retrieval text sample and the annotation data to obtain the video clip retrieval model.

[0062] As an example, text prediction is performed on the text features corresponding to the second video clip sample based on the video clip retrieval model to be trained to obtain the masked text original identifier, and the first cross entropy loss value between the masked text original identifier and the actual text identifier in the annotated data is calculated based on the masked language model loss function, and based on the first cross entropy loss value, the model parameters of the video clip retrieval model to be trained are adjusted to obtain the video clip retrieval model.

[0063] In an embodiment of the present disclosure, the masked language model loss function may be an MLM (Masked Language Model) loss function. The MLM loss function is designed based on the task of predicting masked (i.e., randomly hidden) words. In this prediction task, the model receives a sentence in which some words are replaced with special mask tags, and then the model needs to predict the original identifier of the masked word.

[0064] Therefore, the MLM loss function calculates the cross entropy loss value based on the probability distribution predicted by the model and the actual word identity, so the MLM loss function is the average of the cross entropy loss values ​​of all masked words. During the entire pre-training process, by optimizing this MLM loss function, the model can more accurately predict words in a larger context, thereby learning better semantic representations; where words refer to texts such as lines (such as subtitles) in the second video clip.

[0065] For example, the predicted original label of the masked text is "Hello, I like the sea", and the actual text label in the annotated data is "Hello, I like to go to the sea", so the first cross entropy loss value between the original label of the masked text and the actual text label is calculated, and based on the first cross entropy loss value and the preset threshold, the model parameters of the video clip retrieval model to be trained are adjusted to obtain the video clip retrieval model.

[0066] As another example, based on the video clip retrieval model to be trained, video frame prediction is performed on the visual features corresponding to the second video clip sample to obtain the masked video frame original identifier, and the second cross entropy loss value between the masked video frame original identifier and the actual video frame identifier in the annotated data is calculated based on the mask image model loss function, and based on the second cross entropy loss value, the model parameters of the video clip retrieval model to be trained are adjusted to obtain the video clip retrieval model.

[0067] In an embodiment of the present disclosure, the masked image model loss function may be an MFM (Masked Frame Modeling) loss function, which is designed based on the task of predicting masked (i.e., randomly hidden) video frames. In this prediction task, the model receives individual video frames, where some of the video frames are replaced with special mask marks, and then the model needs to predict the original identifier of the masked video frame.

[0068] Therefore, the MLM loss function calculates the cross entropy loss value based on the probability distribution predicted by the model and the actual video frame identification, so the MLM loss function is the average of the cross entropy loss values ​​of all masked video frames. During the entire pre-training process, by optimizing this MLM loss function, the model can more accurately predict video frames in a larger context environment, thereby learning better visual representations.

[0069] For example, the predicted original identifier of the masked video frame is "video frame 1, video frame 2, and video frame 3", and the actual text identifier in the annotation data is also "video frame 1, video frame 2, and video frame 3", so as to calculate the second cross entropy loss value between the original identifier of the masked video frame and the actual video frame identifier, and based on the comparison between the second cross entropy loss value and the preset threshold, adjust the model parameters of the video clip retrieval model to be trained, so as to obtain the video clip retrieval model.

[0070] As another example, the temporal position of the visual features and text features corresponding to the second video clip sample is predicted based on the video clip retrieval model to be trained to obtain the predicted temporal position, and the third cross entropy loss value between the predicted temporal position and the actual temporal position in the annotated data is calculated based on the video subtitle matching model loss function, and based on the third cross entropy loss value, the model parameters of the video clip retrieval model to be trained are adjusted to obtain the video clip retrieval model.

[0071] In the disclosed embodiment, the video-subtitle matching model loss function may be a VSM (Video-Subtitle Matching) loss function, which may enable the model to learn to retrieve which video segment the text should correspond to and within which specific timeline in the video segment. The input of the VSM loss function includes a video segment v, a line s1 in the video segment v, and lines sn in the video segment v except s1.

[0072] For example, if the predicted time position is the starting time position T1 and the ending time position T2, and the actual time position in the annotated data is the starting time position T1 and the ending time position T3, then the third cross entropy loss value between the predicted time position and the actual time position is calculated, and based on the comparison between the third cross entropy loss value and the preset threshold, the model parameters of the video clip retrieval model to be trained are adjusted to obtain the video clip retrieval model.

[0073] As another example, based on the video clip retrieval model to be trained, the video frame sequence is predicted for the visual features corresponding to the second video clip sample to obtain the predicted video frame sequence, and the fourth cross entropy loss value between the predicted video frame sequence and the actual video frame sequence in the annotation data is calculated based on the video frame sequence model loss function, and based on the fourth cross entropy loss value, the model parameters of the video clip retrieval model to be trained are adjusted to obtain the video clip retrieval model.

[0074] In the disclosed embodiment, the video frame order model loss function may be a FOM (Frame Order Modeling) loss function. The input of the FOM loss function is all video frames of a video, all lines of the video clip, and the order of the video frames. For example, 15% of the video frames are randomly selected and their order is disrupted. The purpose of the FOM loss function is to restore the original video sequence.

[0075] For example, the predicted video frame sequence is "video frame 1 to video frame 7", and the actual video frame sequence in the annotated data is also "video frame 1, video frame 3 to 7", so as to calculate the fourth cross entropy loss value between the predicted video frame sequence and the actual video frame sequence, and based on the fourth cross entropy loss value and the preset threshold, the model parameters of the video clip retrieval model to be trained are adjusted to obtain the video clip retrieval model.

[0076] As another example, the matching degree of the visual features and text features corresponding to the second video clip sample is calculated based on the video clip retrieval model to be trained to obtain the predicted matching degree, and the fifth cross-entropy loss value between the predicted matching degree and the actual matching degree in the annotated data is calculated based on the text-to-video retrieval model loss function, and based on the fifth cross-entropy loss value, the model parameters of the video clip retrieval model to be trained are adjusted to obtain the video clip retrieval model.

[0077] In the disclosed embodiment, the text-to-video retrieval model loss function may be a TVR (text-to-video-retrieval) loss function for matching text features with video features, so that the model learns the video features corresponding to the text, thereby retrieving the target video clip based on the label.

[0078] It should be noted that one or more of the aforementioned loss functions can be selected to train the model according to actual training needs to meet different usage requirements.

[0079] In this way, a multimodal video clip retrieval model can be pre-trained so that the parameters of the video clip retrieval model can be fine-tuned on data that meets business needs, thereby training a target video clip retrieval model that can retrieve video clips based on tags, further meeting user needs and improving user retrieval experience.

[0080] In some embodiments, obtaining video features corresponding to the first video clip sample includes: performing frame sampling processing on the first video clip sample at a preset frame sampling frequency to obtain each video frame, performing visual feature extraction on each video frame based on a pre-trained image feature extraction model to obtain visual features corresponding to the first video clip sample, performing optical character recognition on each video frame based on image optical character recognition technology to obtain text information, performing text feature extraction on the text information based on a pre-trained text feature extraction model to obtain text features corresponding to the first video clip sample, and performing feature fusion on the visual features and text features corresponding to the first video clip sample to obtain video features.

[0081] In the embodiment of the present disclosure, the frame extraction frequency is preset, for example, the video clip samples are extracted at one frame per two seconds to obtain each video frame.

[0082] In the disclosed embodiment, an image feature extraction model is pre-trained, for example, a multimodal (text and image) pre-training model CLIP (Contrastive Language-Image Pre-training) model, and each video frame is input into the CLIP model to obtain the visual features corresponding to the video clip samples.

[0083] Therefore, by extracting key frames from the first video clip sample and extracting visual features therefrom based on a pre-trained image feature extraction model, the accuracy and efficiency of obtaining visual features corresponding to the video clip sample are further improved, thereby improving the accuracy and efficiency of model training.

[0084] In the disclosed embodiment, the video clip sample may be further extracted to obtain individual video frames, and optical character recognition may be performed on the video frames using image optical character recognition technology to obtain second text information. The second text information may be understood as text corresponding to the lines (subtitles displayed on the image screen) in the video clip sample, that is, all the lines (subtitles) of the video clip sample are converted into text.

[0085] Among them, a text feature extraction model is pre-trained, such as a bidirectional encoder representation BERT (Bidirectional Encoder Representations from Transformers) model.

[0086] In the disclosed embodiment, the generated video description text features are integrated with the original visual features and text features to form new video features; various feature fusion algorithms can be used, such as splicing, weighted averaging, feature selection, etc.

[0087] In some embodiments, the visual features and text features corresponding to the first video clip sample are feature fused to obtain video features, including: splicing the visual features and text features corresponding to the first video clip sample according to a preset feature splicing order to obtain video features; and / or, weighted averaging the visual features and text features corresponding to the first video clip sample according to preset feature weights to obtain video features; and / or, feature selecting the visual features and text features corresponding to the first video clip sample according to a preset screening strategy to obtain video features.

[0088] Specifically, visual features, text features and video description text features may be spliced ​​in a preset feature splicing order to obtain video features; wherein the feature splicing order is preset, such as splicing visual features and text features in sequence, and the setting is selected specifically according to application needs.

[0089] Specifically, the visual features and text features can be weighted averaged according to preset feature weights to obtain video features; wherein the feature weights can be set according to application needs, for example, the feature weights corresponding to the visual features and text features are A1 and A2, respectively, and the first result of multiplying the visual features and A1 is added to the second result of multiplying the text features and A2, and the added results are averaged to obtain the video features.

[0090] Specifically, feature selection is performed on visual features and text features according to a preset screening strategy to obtain video features; wherein the screening strategy is preset, such as selecting visual features, so that the visual features are used as video features.

[0091] Therefore, one or more methods can be selected to fuse visual features and text features to obtain video features, thereby further improving the accuracy and flexibility of feature fusion, thereby improving the accuracy and flexibility of subsequent model training.

[0092] In some embodiments, based on a first loss value between video features and label features, and a second loss value between real time points and predicted time points, model parameters of a video clip retrieval model are adjusted to obtain a target video clip retrieval model, including: determining a target cross entropy loss value based on the first loss value and the second loss value, and adjusting the model parameters of the video clip retrieval model to obtain a target video clip retrieval model.

[0093] In the disclosed embodiment, the loss function may be pre-set according to training requirements, such as a mean square error loss function, a cross entropy loss function, etc.

[0094] In the disclosed embodiment, the video features and label features can be processed by dot multiplication or the like to obtain the similarity between the video features and the label features, and the similarity is input into a preset loss function for calculation to obtain a first loss value. The similarity between the real time point and the predicted time point can also be calculated, and the similarity is input into a preset loss function for calculation to obtain a second loss value. A target cross entropy loss value is determined based on the comparison between the first loss value and the second loss value, or is calculated by other calculation formulas to obtain a target cross entropy loss value. It can be understood that when the target cross entropy loss value is less than a certain threshold, the training can be stopped. If the target cross entropy loss value is greater than or equal to the threshold, the model parameters of the video clip retrieval model to be trained need to be adjusted, and then the model training is continued until the target cross entropy loss value is less than the threshold to obtain the target video clip retrieval model.

[0095] In this way, the target video segment retrieval model can be accurately and quickly obtained, thereby improving the accuracy of subsequent video segment retrieval.

[0096] Based on the description of the above embodiments, video clip retrieval is a task that combines natural language and computer vision. Its purpose is to find the most matching video clip from a number of video clips and the time point at which the content described in the retrieval text occurs based on a given retrieval text. In this task, the video clip retrieval model needs to understand and process multiple types of data, including text, video images, etc. The retrieval text includes plot description text and dialogue text. The video image refers to the image content of the film and television drama. Usually, a model is used to extract the features of a given text, and then extract the multimodal fusion features of the video. The similarity of these two features is compared to determine which video clip the text corresponds to.

[0097] The difference between the video clip retrieval of the target video clip retrieval model of the disclosed embodiment and the above-mentioned video clip retrieval is that the retrieval tag used to retrieve the video clip in the disclosed embodiment is a very short phrase, such as trial, chase, etc. Based on an existing video clip retrieval model, the disclosed embodiment fine-tunes the model parameters on data that meets business needs, thereby training a target video clip retrieval model that can directly retrieve video clips based on retrieval tags.

[0098] Figure 2 A flowchart of a method for training a video clip retrieval model provided in an embodiment of the present disclosure is provided. The method mainly includes the following steps S202 to S212:

[0099] Step S202, obtain a second video clip sample, retrieve a text sample and annotated data, obtain visual features and text features corresponding to the second video clip sample, adjust the model parameters of the video clip retrieval model to be trained based on the visual features, text features and annotated data corresponding to the second video clip sample and a preset loss function, and obtain a video clip retrieval model.

[0100] Step S204: obtaining the first video clip sample, the search tag sample, and the real time point of the search tag sample in the first video clip sample.

[0101] Step S206, performing frame extraction processing on the first video clip sample according to a preset frame extraction frequency to obtain each video frame, and extracting visual features from each video frame based on a pre-trained image feature extraction model to obtain visual features corresponding to the first video clip sample.

[0102] Step S208, performing optical character recognition on each video frame based on image optical character recognition technology to obtain text information, performing text feature extraction on the text information based on a pre-trained text feature extraction model to obtain text features corresponding to the first video clip sample.

[0103] Step S210: splicing the visual features and text features corresponding to the first video clip sample according to a preset feature splicing order to obtain video features.

[0104] Step S212: Input the video features and the retrieval label features into the video clip retrieval model, obtain the predicted time point, determine the target cross entropy loss value based on the first loss value between the video features and the label features, and the second loss value between the real time point and the predicted time point, adjust the model parameters of the video clip retrieval model, and obtain the target video clip retrieval model.

[0105] Specifically, training data is prepared, including a second video clip sample, a retrieval text sample, and annotated data. The open source model HERO is used as a pre-training model architecture, and pre-training is performed through the training data to obtain a video clip retrieval model.

[0106] Furthermore, based on the trained video clip retrieval model, the model parameters are fine-tuned to train a target video clip retrieval model. Specifically, training data is prepared, which is usually a first video clip sample of a few seconds to 5 minutes and a retrieval label sample corresponding to the video, as well as the starting time point and ending time point of the retrieval label sample in the first video clip sample, that is, the real time point.

[0107] Furthermore, through the TVR loss function, the retrieval label features are matched with the video features, so that the model learns which video feature the retrieval label sample corresponds to, and through the time point loss function, the retrieval label sample can find the video clip corresponding to the time more accurately.

[0108] As an example, Figure 3 As shown, a second video clip sample is input, and at the same time, the second video clip sample is extracted at a rate of one frame per two seconds to obtain each video frame, such as 150 frames of images. Then, an open source model, such as the CLIP model, is used to extract visual features of each image to obtain visual features of each video frame, such as 150 frames of images; the second video clip sample is processed using open source OCR (Optical Character Recognition) technology to obtain N lines of dialogue (text information), and an open source model, such as the BERT model, is used to extract features of the N lines of dialogue to obtain N text features; wherein N is a positive integer, and the visual features and text features of each video frame are fused to obtain video features; the video features, the retrieval text samples, and the annotated data are input into the target platform (such as the HERO architecture) and the model is trained in combination with the loss function to obtain a video clip retrieval model.

[0109] As another example, Figure 4 As shown, a first video clip sample is input, and the first video clip sample is extracted at a rate of one frame per two seconds to obtain each video frame. Then, an open source model such as the CLIP model is used to extract visual features of each image to obtain visual features of each video frame. The first video clip sample is processed using open source OCR (Optical Character Recognition) technology to obtain N lines (text information), and an open source model such as the BERT model is used to extract features of the N lines to obtain N text features. Wherein, N is a positive integer, and the visual features and text features of each video frame are fused to obtain video features. The video features, the retrieval label samples, and the real time points of the retrieval label samples in the first video clip samples are input into the video clip retrieval model and the model is trained in combination with the loss function to obtain a target video clip retrieval model.

[0110] Therefore, on the basis of the pre-trained video clip retrieval model, the model parameters are fine-tuned based on the sample data that meets the business needs, so that the target video clip retrieval model can directly obtain the video clip quickly and accurately based on the retrieval label.

[0111] Figure 5 The following is a flow chart of a video clip retrieval method provided by an embodiment of the present disclosure. The method mainly includes the following steps S502 to S504:

[0112] Step S502, obtaining tags to be searched.

[0113] Step S504: searching in a preset candidate video segment library according to the to-be-searched tags based on the target video segment retrieval model to obtain the target video segment.

[0114] In the embodiment of the present disclosure, the target video clip retrieval model is obtained by the training method of the video clip retrieval model described in the aforementioned embodiment.

[0115] In this embodiment, the tag to be retrieved is the content input by the user according to the actual video clip retrieval demand. Based on the target video clip retrieval model, a search is performed in a preset candidate video clip library according to the tag to be retrieved, so that the target video clip matching the tag to be retrieved can be obtained; wherein, the candidate video clip library has multiple candidate video clips, and each candidate video clip has corresponding candidate video features (such as visual features, text features, etc.); in order to further improve the retrieval efficiency, the visual features corresponding to the candidate video clip can be extracted in advance for each candidate video clip through a pre-trained image feature extraction model, and the text information of the candidate video clip can be identified, and the text features corresponding to the text information can be extracted through a pre-trained text feature extraction model to obtain the text features corresponding to the video clip.

[0116] Specifically, based on the target video segment retrieval model, a search is performed in a preset candidate video segment library according to the tag to be retrieved. There are many ways to obtain the target video segment. As a possible implementation method, the tag feature to be retrieved corresponding to the tag to be retrieved is obtained, and the tag feature to be retrieved and the candidate video feature corresponding to each candidate video segment in the candidate video segment library are input into the target video segment retrieval model to obtain the target time point, and the candidate video segment corresponding to the target time point is used as the target video segment.

[0117] As a possible implementation method, the tag to be retrieved is input into the target video segment retrieval model to obtain the feature of the tag to be retrieved, and the similarity is calculated between the candidate video features corresponding to each candidate video segment in the video segment library, and the candidate video segment corresponding to the similarity greater than a preset similarity threshold is used as the target video segment, or the candidate video segment corresponding to the maximum similarity is used as the target video segment, etc.

[0118] Specifically, when a user initiates a video clip retrieval request, the retrieval tag input by the user and the corresponding integrated video features of all video clips in the video library are input into the target video clip retrieval model, and the most relevant target video clip is returned.

[0119] For example, Figure 6As shown, all video clips in the video clip are obtained, and at the same time, the video clips are extracted at a rate of one frame per two seconds to obtain each video frame, and then the visual features of each image are extracted using an open source model such as the CLIP model to obtain the visual features of each video frame; the video clips are processed using open source OCR (Optical Character Recognition) technology to obtain text information, and open source models such as the BERT model are used to extract features of the text information to obtain text features; the visual features and text features of each video frame are fused to obtain feature expressions of each video; each video feature and retrieval tag are input into a target video clip retrieval model to determine the target video clip from all video clips.

[0120] Therefore, on the basis of the pre-trained video clip retrieval model, the model parameters are fine-tuned based on the sample data that meets the business needs, so that the target video clip retrieval model can directly obtain the video clip quickly and accurately based on the retrieval label.

[0121] Corresponding to the aforementioned video segment retrieval model training method, the present disclosure further provides a video segment retrieval model training device. Figure 7 A schematic diagram of the structure of a video clip retrieval model training device provided in an embodiment of the present disclosure, the device can be implemented by software and / or hardware, and the device includes:

[0122] The first acquisition module 702 is used to acquire a pre-trained video clip retrieval model.

[0123] The second acquisition module 704 is used to acquire the first video segment sample, the tag sample to be retrieved, and the real time point of the tag sample to be retrieved in the first video segment sample.

[0124] The third acquisition module 706 is used to acquire the video features corresponding to the video clip samples.

[0125] The fourth acquisition module 708 is used for obtaining label features corresponding to the label samples.

[0126] The training module 710 is used to input the video features and the retrieval label features into the video clip retrieval model, obtain the predicted time point, and adjust the model parameters of the video clip retrieval model based on the first loss value between the video features and the label features, and the second loss value between the real time point and the predicted time point to obtain the target video clip retrieval model.

[0127] The above-mentioned device provided by the embodiment of the present disclosure fine-tunes the model parameters based on the sample data that meets the business needs on the basis of the pre-trained video clip retrieval model, so that the target video clip retrieval model can directly obtain the video clip quickly and accurately based on the retrieval label.

[0128] In some embodiments, the device also includes: a sixth acquisition module, used to acquire a second video clip sample, a retrieval text sample and annotation data; a seventh acquisition module, used to acquire visual features and text features corresponding to the second video clip sample; a training adjustment module, used to adjust the model parameters of the video clip retrieval model to be trained based on the visual features, text features and the annotation data corresponding to the second video clip sample and a preset loss function to obtain the video clip retrieval model.

[0129] In some embodiments, the training adjustment module is specifically used to: perform text prediction on the text features corresponding to the second video clip sample based on the video clip retrieval model to be trained to obtain a masked text original identifier, calculate a first cross entropy loss value between the masked text original identifier and the actual text identifier in the annotated data based on a masked language model loss function, and adjust the model parameters of the video clip retrieval model to be trained based on the first cross entropy loss value to obtain the video clip retrieval model; and / or perform video frame prediction on the visual features corresponding to the second video clip sample based on the video clip retrieval model to be trained to obtain a masked video frame original identifier, calculate a second cross entropy loss value between the masked video frame original identifier and the actual video frame identifier in the annotated data based on a masked image model loss function, and adjust the model parameters of the video clip retrieval model to be trained based on the second cross entropy loss value to obtain the video clip retrieval model; and / or perform time position prediction on the visual features and text features corresponding to the second video clip sample based on the video clip retrieval model to be trained to obtain a predicted time position, and calculate the second cross entropy loss value between the masked video frame original identifier and the actual video frame identifier in the annotated data based on the masked image model loss function, and adjust the model parameters of the video clip retrieval model to be trained based on the second cross entropy loss value to obtain the video clip retrieval model. The loss function calculates a third cross entropy loss value between the predicted time position and the actual time position in the annotated data, and based on the third cross entropy loss value, adjusts the model parameters of the video clip retrieval model to be trained to obtain the video clip retrieval model; and / or based on the video clip retrieval model to be trained, performs video frame sequence prediction on the visual features corresponding to the second video clip sample to obtain the predicted video frame sequence, calculates a fourth cross entropy loss value between the predicted video frame sequence and the actual video frame sequence in the annotated data based on the video frame sequence model loss function, and adjusts the model parameters of the video clip retrieval model to be trained based on the fourth cross entropy loss value to obtain the video clip retrieval model; and / or based on the video clip retrieval model to be trained, performs matching calculation on the visual features and text features corresponding to the second video clip sample to obtain the predicted matching degree, calculates a fifth cross entropy loss value between the predicted matching degree and the actual matching degree in the annotated data based on the text-to-video retrieval model loss function, and adjusts the model parameters of the video clip retrieval model to be trained based on the fifth cross entropy loss value to obtain the video clip retrieval model.

[0130] In some embodiments, the third acquisition module 706 includes: a frame extraction unit, which is used to extract frames from the first video clip sample at a preset frame extraction frequency to obtain each video frame; a first extraction unit, which is used to extract visual features from the each video frame based on a pre-trained image feature extraction model to obtain visual features corresponding to the first video clip sample; a recognition unit, which is used to perform optical character recognition on the each video frame based on image optical character recognition technology to obtain text information; a second extraction unit, which is used to perform text feature extraction on the text information based on a pre-trained text feature extraction model to obtain text features corresponding to the first video clip sample; and a fusion unit, which is used to fuse the visual features and text features corresponding to the first video clip sample to obtain the video features.

[0131] In some embodiments, the fusion unit is specifically used to: splice the visual features and text features corresponding to the first video clip sample according to a preset feature splicing order to obtain the video features; and / or, perform weighted averaging processing on the visual features and text features corresponding to the first video clip sample according to preset feature weights to obtain the video features; and / or, perform feature selection on the visual features and text features corresponding to the first video clip sample according to a preset screening strategy to obtain the video features.

[0132] In some embodiments, the training module is specifically used to: input the video features and the retrieval tag features into the video clip retrieval model, obtain the predicted time point, and determine the target cross entropy loss value based on the first loss value and the second loss value; adjust the model parameters of the video clip retrieval model based on the target cross entropy loss value to obtain the target video clip retrieval model.

[0133] The training device for the video clip retrieval model provided in the embodiments of the present disclosure can execute the training method for the video clip retrieval model provided in any embodiment of the present disclosure, and has the functional modules and beneficial effects corresponding to the execution method.

[0134] Corresponding to the aforementioned video segment retrieval method, the embodiment of the present disclosure further provides a video segment retrieval device, Figure 8 This is a schematic diagram of the structure of a video clip retrieval device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware, and the device includes:

[0135] The fifth acquisition module 802 is used to acquire the tag to be searched.

[0136] The retrieval module 804 is used to search in a preset candidate video segment library according to the to-be-retrieved tags based on the target video segment retrieval model to obtain the target video segment.

[0137] The video clip retrieval model is obtained according to the training method of the video clip retrieval model described in the aforementioned embodiment.

[0138] In some embodiments, the retrieval module 804 is specifically used to obtain the tag features to be retrieved corresponding to the tag to be retrieved; input the tag features to be retrieved and the candidate video features corresponding to each candidate video segment in the candidate video segment library into the target video segment retrieval model to obtain the target time point; and use the candidate video segment corresponding to the target time point as the target video segment.

[0139] The above-mentioned device provided by the embodiment of the present disclosure fine-tunes the model parameters based on the sample data that meets the business needs on the basis of the pre-trained video clip retrieval model, so that the target video clip retrieval model can directly obtain the video clip quickly and accurately based on the retrieval label.

[0140] The video clip retrieval device provided in the embodiments of the present disclosure can execute the video clip retrieval method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0141] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0142] An embodiment of the present disclosure provides an electronic device, which includes: a storage device on which a computer program is stored; and a processing device for executing the computer program in the storage device to implement the steps of any method in the present disclosure.

[0143] Reference below Fig. 9 , which shows a schematic diagram of the structure of an electronic device 900 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0144] like Fig. 9As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0145] Typically, the following devices may be connected to the I / O interface 905: an input device 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 908 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0146] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0147] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the image processing method provided by the embodiments of the present disclosure. The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present disclosure, the programming languages ​​including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0148] In addition, the embodiments of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the training method of the video clip retrieval model and the video clip retrieval method provided by the embodiments of the present disclosure.

[0149] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0150] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the video clip retrieval model training method and the video clip retrieval method in the embodiments of the present disclosure.

[0151] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0152] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0153] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0154] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0155] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0156] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a video clip retrieval model, characterized in that: The method comprises: Get a pre-trained video clip retrieval model; Obtaining a first video clip sample, a retrieval tag sample, and a real time point position of the retrieval tag sample in the first video clip sample; Obtaining video features corresponding to the first video clip sample and retrieval tag features corresponding to the retrieval tag sample; The video features and the retrieval label features are input into the video clip retrieval model to obtain a predicted time point, and the model parameters of the video clip retrieval model are adjusted based on a first loss value between the video features and the retrieval label features, and a second loss value between the real time point and the predicted time point to obtain a target video clip retrieval model.

2. The method according to claim 1, characterized in that: The method further comprises: Obtaining a second video clip sample, retrieving text samples and annotating data; Acquire visual features and text features corresponding to the second video clip sample; Based on the visual features, text features, the annotation data and a preset loss function corresponding to the second video clip sample, the model parameters of the video clip retrieval model to be trained are adjusted to obtain the video clip retrieval model.

3. The method according to claim 2, characterized in that The method of adjusting the model parameters of the video segment retrieval model to be trained based on the visual features, text features, the annotation data and the preset loss function corresponding to the second video segment sample to obtain the video segment retrieval model includes: Based on the video clip retrieval model to be trained, text prediction is performed on the text features corresponding to the second video clip sample to obtain a masked text original identifier, a first cross entropy loss value between the masked text original identifier and the actual text identifier in the annotated data is calculated based on the masked language model loss function, and based on the first cross entropy loss value, model parameters of the video clip retrieval model to be trained are adjusted to obtain the video clip retrieval model; and / or Based on the video clip retrieval model to be trained, video frame prediction is performed on the visual features corresponding to the second video clip sample to obtain a masked video frame original identifier, a second cross entropy loss value between the masked video frame original identifier and the actual video frame identifier in the annotated data is calculated based on the mask image model loss function, and based on the second cross entropy loss value, model parameters of the video clip retrieval model to be trained are adjusted to obtain the video clip retrieval model; and / or Based on the video segment retrieval model to be trained, the visual features and text features corresponding to the second video segment sample are predicted in time position to obtain the predicted time position, and based on the video subtitle matching model loss function, a third cross entropy loss value between the predicted time position and the actual time position in the annotated data is calculated, and based on the third cross entropy loss value, the model parameters of the video segment retrieval model to be trained are adjusted to obtain the video segment retrieval model; and / or Based on the video clip retrieval model to be trained, the visual features corresponding to the second video clip sample are predicted to obtain a predicted video frame sequence, and a fourth cross entropy loss value between the predicted video frame sequence and the actual video frame sequence in the annotated data is calculated based on the video frame sequence model loss function, and based on the fourth cross entropy loss value, the model parameters of the video clip retrieval model to be trained are adjusted to obtain the video clip retrieval model; and / or Based on the video clip retrieval model to be trained, the matching degree of the visual features and the text features corresponding to the second video clip sample is calculated to obtain a predicted matching degree, and based on the text-to-video retrieval model loss function, a fifth cross-entropy loss value between the predicted matching degree and the actual matching degree in the annotated data is calculated, and based on the fifth cross-entropy loss value, the model parameters of the video clip retrieval model to be trained are adjusted to obtain the video clip retrieval model.

4. The method according to claim 1, characterized in that The obtaining of the video feature corresponding to the first video clip sample includes: Performing frame extraction processing on the first video clip samples according to a preset frame extraction frequency to obtain video frames; Extracting visual features from each video frame based on a pre-trained image feature extraction model to obtain visual features corresponding to the first video clip sample; Performing optical character recognition on each video frame based on image optical character recognition technology to obtain text information; Performing text feature extraction on the text information based on a pre-trained text feature extraction model to obtain text features corresponding to the first video clip sample; The visual features and text features corresponding to the first video clip sample are fused to obtain the video features.

5. The method according to claim 4, characterized in that The step of fusing the visual features and text features corresponding to the first video clip sample to obtain the video features includes: splicing the visual features and text features corresponding to the first video clip sample according to a preset feature splicing order to obtain the video features; and / or, Performing weighted averaging processing on the visual features and text features corresponding to the first video clip sample according to preset feature weights to obtain the video features; and / or, Feature selection is performed on the visual features and text features corresponding to the first video clip sample according to a preset screening strategy to obtain the video features.

6. The method according to claim 1, characterized in that The method of adjusting the model parameters of the video segment retrieval model based on the first loss value between the video feature and the label feature, and the second loss value between the real time point and the predicted time point to obtain the target video segment retrieval model includes: Determining a target cross entropy loss value according to the first loss value and the second loss value; The model parameters of the video clip retrieval model are adjusted based on the target cross entropy loss value to obtain the target video clip retrieval model.

7. A video clip retrieval method, characterized in that: The method comprises: Get the tags to be retrieved; Based on the target video segment retrieval model, searching in a preset candidate video segment library according to the to-be-retrieved tags, to obtain the target video segment; The target video clip retrieval model is obtained according to the training method of the video clip retrieval model described in any one of claims 1-6.

8. The method according to claim 7, characterized in that The method of searching a preset candidate video segment library based on the target video segment retrieval model according to the to-be-retrieved tag to obtain the target video segment includes: Obtaining a tag feature to be retrieved corresponding to the tag to be retrieved; Input the tag feature to be retrieved and the candidate video feature corresponding to each candidate video segment in the candidate video segment library into the target video segment retrieval model to obtain the target time point; The candidate video segment corresponding to the target time point is used as the target video segment.

9. A training device for a video clip retrieval model, characterized in that: include: A first acquisition module is used to acquire a pre-trained video clip retrieval model; A second acquisition module is used to acquire a first video clip sample, a tag sample to be retrieved, and a real time point position of the tag sample to be retrieved in the first video clip sample; A third acquisition module, used to acquire video features corresponding to the first video clip sample; A fourth acquisition module, used for label features corresponding to the label samples; A training module is used to input the video features and the retrieval label features into the video clip retrieval model, obtain a predicted time point, and adjust the model parameters of the video clip retrieval model based on a first loss value between the video features and the label features, and a second loss value between the real time point and the predicted time point, so as to obtain a target video clip retrieval model.

10. A video clip retrieval device, characterized in that: include: The fifth acquisition module is used to acquire the tag to be retrieved; A retrieval module, configured to search a preset candidate video segment library based on the target video segment retrieval model and the to-be-retrieved tag to obtain the target video segment; The target video clip retrieval model is obtained according to the training method of the video clip retrieval model described in any one of claims 1-6.

11. An electronic device, characterized in that: The electronic device comprises: a storage device having a computer program stored thereon; A processing device for executing the computer program in the storage device to implement the steps of the video clip retrieval model training method described in any one of claims 1 to 6, or the video text retrieval method described in claim 7 or 8.

12. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which is used to execute the steps of the video clip retrieval model training method described in any one of claims 1 to 6, or the video text retrieval method described in claim 7 or 8.

Images