Training method of video retrieval model and video retrieval method and device

Abstract

The embodiment of the invention relates to a training method of a video retrieval model and a video retrieval method and device, and the method comprises the steps: obtaining a video sample and a retrieval text sample, obtaining a visual feature, a subtitle text feature and an audio feature corresponding to the video sample, configuring initial weight parameters corresponding to the visual feature, the text feature and the audio feature, performing feature fusion on the visual features, the subtitle text features and the audio features based on the initial weight parameters to obtain video features, and adjusting model parameters and the initial weight parameters of a to-be-trained video retrieval model when a loss value between the video features and retrieval text features corresponding to the retrieval text samples is greater than or equal to a preset threshold value, and obtaining a video retrieval model until the adjusted loss value is smaller than a preset threshold value. According to the video retrieval model obtained through training, the weight parameters corresponding to the visual features, the subtitle text features and the audio features can be accurately distributed, and therefore the retrieval result of video retrieval based on the video retrieval model is more accurate and comprehensive.

Description

Technical Field

[0001] The present disclosure relates to the field of multimedia processing technology, and in particular to a video retrieval model training method, a video retrieval method and a device. Background Art

[0002] Typically, on digital video streaming platforms, online video websites, and TV stations, users often hope to obtain videos of movies or TV series they want to watch through simple keywords or short text descriptions.

[0003] However, existing video retrieval models usually rely solely on video images or dialogue information during the training process, resulting in poor performance of the trained video retrieval models, which in turn affects the final video retrieval results. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a video retrieval model training method, a video retrieval method and a device.

[0005] The disclosed embodiment provides a method for training a video retrieval model, comprising: obtaining a video sample and a retrieval text sample; obtaining visual features, subtitle text features and audio features corresponding to the video sample, and configuring initial weight parameters corresponding to the visual features, the subtitle text features and the audio features; inputting the video features and the retrieval text sample into a video retrieval model to be trained, and when a loss value between the video features and the retrieval text features corresponding to the retrieval text sample is greater than or equal to a preset threshold, adjusting the model parameters and the initial weight parameters of the video retrieval model to be trained until the adjusted loss value is less than the preset threshold, thereby obtaining a video retrieval model.

[0006] Optionally, obtaining the video sample includes: obtaining a video to be processed, and processing the video to be processed according to a preset video splitting frequency to obtain multiple video clips as the video samples; and / or obtaining a video identifier, and obtaining at least one video clip in a preset video clip library as the video sample based on the video identifier.

[0007] Optionally, obtaining the visual features corresponding to the video samples includes: performing frame sampling processing on the video samples at a preset frame sampling frequency to obtain each video frame; and performing visual feature extraction on each video frame based on a pre-trained image feature extraction model to obtain the visual features corresponding to the video samples.

[0008] Optionally, obtaining subtitle text features corresponding to the video sample includes: obtaining audio information in the video sample, and converting the audio information into first text information based on speech-to-text technology; and / or performing optical character recognition on each video frame based on image optical character recognition technology to obtain second text information; performing text feature extraction on the first text information and / or the second text information based on a pre-trained text feature extraction model to obtain subtitle text features corresponding to the video sample.

[0009] Optionally, obtaining the audio features corresponding to the video samples includes: extracting audio information from the video samples based on a preset audio processing tool to obtain audio information; and extracting audio features from the audio information based on a preset audio model to obtain the audio features.

[0010] Optionally, the visual features, the text features and the audio features are subjected to feature fusion based on the initial weight parameters to obtain video features, including: obtaining a first weight coefficient corresponding to the visual features, a second weight coefficient corresponding to the text features and a third weight coefficient corresponding to the audio features based on the initial weight parameters; and performing weighted averaging processing on the visual features, the text features and the audio features according to the first weight coefficient, the second weight coefficient and the third weight coefficient to obtain the video features.

[0011] Optionally, the video features and the retrieval text samples are input into the video retrieval model to be trained, and when the loss value between the video features and the retrieval text features corresponding to the retrieval text samples is greater than or equal to a preset threshold, the model parameters and the initial weight parameters of the video retrieval model to be trained are adjusted until the adjusted loss value is less than the preset threshold, thereby obtaining a video retrieval model, including: calculating the similarity between the video features and the retrieval text features; determining a first loss value based on the similarity and a preset contrast loss function, determining a second loss value based on the similarity and a preset matching loss function, and determining a third loss value based on the similarity and a preset generative loss function; and when it is determined that the total loss value is greater than or equal to a preset threshold based on the first loss value, the second loss value, and the third loss value, adjusting the model parameters and the initial weight parameters of the video retrieval model to be trained until the adjusted loss value is less than the preset threshold, thereby obtaining the video retrieval model.

[0012] The disclosed embodiment also provides a video retrieval method, which includes: obtaining a text to be retrieved; searching a preset candidate video library based on the text to be retrieved based on a video retrieval model to obtain a target video; wherein the video retrieval model is obtained according to the training method of the video retrieval model described in the aforementioned embodiment.

[0013] Optionally, the video retrieval model-based search is performed in a preset candidate video library according to the text to be retrieved to obtain a target video, including: obtaining the features to be retrieved corresponding to the text to be retrieved, and obtaining the candidate visual features, candidate text features and candidate audio features corresponding to each candidate video in the video library; calculating through the video retrieval model based on the candidate weight parameters corresponding to the candidate visual features, the candidate text features and the candidate audio features and the features to be retrieved to obtain a target similarity; sorting according to all the target similarities, and obtaining the candidate video corresponding to the maximum target similarity as the target video.

[0014] The disclosed embodiment also provides a training device for a video retrieval model, comprising: a first acquisition module, used to acquire video samples and retrieval text samples; a second acquisition module, used to acquire visual features corresponding to the video samples; a third acquisition module, used to acquire subtitle text features corresponding to the video samples; a fourth acquisition module, used to acquire audio features corresponding to the video samples; a configuration module, used to configure weight parameters corresponding to the visual features, the subtitle text features and the audio features; a fusion module, used to perform feature fusion on the visual features, the subtitle text features and the audio features based on the weight parameters to obtain video features; a training module, used to input the video features and the retrieval text samples into a video retrieval model to be trained, and when the loss value between the video features and the retrieval text features corresponding to the retrieval text samples is greater than or equal to a preset threshold, adjust the model parameters and the initial weight parameters of the video retrieval model to be trained until the adjusted loss value is less than the preset threshold to obtain a video retrieval model.

[0015] The disclosed embodiment also provides a video retrieval device, comprising: a fifth acquisition module, used to acquire the text to be retrieved; a retrieval module, used to search in a preset candidate video library according to the text to be retrieved based on a video retrieval model to obtain a target video; wherein the video retrieval model is obtained according to the training method of the video retrieval model described in the aforementioned embodiment.

[0016] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement a video retrieval model training method and a video retrieval method as provided in an embodiment of the present disclosure.

[0017] The embodiments of the present disclosure also provide a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the video retrieval model training method and video retrieval method provided by the embodiments of the present disclosure.

[0018] The above-mentioned technical solution provided by the embodiment of the present disclosure obtains video features for training a video retrieval model by fusing multimodal features such as visual features, text features and audio features according to certain weight parameters, and continuously adjusts the weight parameters during the model training process to determine the optimal weight parameters of visual features, text features and audio features to obtain a video retrieval model. Since the video features combine multimodal information such as visual information, dialogue information and audio information of the video, the retrieval results of video retrieval based on the video retrieval model are more accurate and comprehensive.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] Figure 1 A flowchart of a method for training a video retrieval model provided in an embodiment of the present disclosure;

[0023] Figure 2 A flowchart of a method for training a video retrieval model provided in an embodiment of the present disclosure;

[0024] Figure 3 A schematic diagram of a training process of a video retrieval model provided in an embodiment of the present disclosure;

[0025] Figure 4 A schematic diagram of a flow chart of a video retrieval method provided by an embodiment of the present disclosure;

[0026] Figure 5 A schematic diagram of a video retrieval process provided by an embodiment of the present disclosure;

[0027] Figure 6 A schematic diagram of the structure of a video retrieval model training device provided in an embodiment of the present disclosure;

[0028] Figure 7 A schematic diagram of the structure of a video retrieval device provided in an embodiment of the present disclosure;

[0029] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0031] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0032] In existing video retrieval scenarios, users often hope to find the movies or TV series they want to watch through simple keywords or short text descriptions. Therefore, an efficient and time-saving video retrieval model is needed for video retrieval.

[0033] Existing video retrieval models mainly rely on the video's picture information and dialogue information for training and acquisition separately, resulting in poor performance of the trained video retrieval model, which in turn affects the final video retrieval results.

[0034] Specifically, full-modal film and television drama plot retrieval, that is, video retrieval methods can make full use of all the information in the video, such as vision, text and audio; in a typical film and television drama plot, there is rich visual information (such as scenes, characters, actions, etc.), text information (such as dialogues, narration, etc.), and audio information (such as background music, sound effects, etc.), which together constitute a complete plot and bring a three-dimensional sensory experience to the audience. However, in existing video retrieval technologies, usually only visual and text information are considered, and audio information is ignored. Moreover, there is no unified standard for the representation and processing of audio information, which makes the use of audio information complicated and difficult.

[0035] However, in actual usage scenarios, audio information is sometimes a key determinant. For example, in a musical, the main plot is often conveyed through songs; in a horror movie, music and sound effects are important means of creating atmosphere and suspense. In these scenarios, if audio information is ignored, relevant videos may not be accurately retrieved.

[0036] Therefore, how to effectively use audio information to improve and enhance the performance of video retrieval is a key issue to be solved by the present disclosure. The present disclosure embodiment uses a pre-trained model to integrate the features of all modalities so that all modal information can be expressed in a unified feature space, and uses various loss function methods to ensure the robustness and performance of the model, thereby improving the accuracy and efficiency of video retrieval; that is, audio information is supplemented in the training video retrieval model to make the retrieval results more comprehensive and accurate.

[0037] Specifically, the disclosed embodiment proposes a method for training a video retrieval model, by fusing the visual features, subtitle text features and audio features corresponding to the video sample according to certain weights to obtain video features for subsequent video retrieval model training, and continuously adjusting the weight parameters during the model training process to determine the optimal weight parameters of the visual features, subtitle text features and audio features to obtain a video retrieval model, thereby fusing the full-modal feature information of the video in a feature space, so that the trained video retrieval model can assign corresponding weights to the features of different modalities, thereby making the retrieval results based on the video retrieval model more accurate and comprehensive. The following is a detailed explanation:

[0038] Figure 1 The present invention provides a flow chart of a video retrieval model training method, which can be applied to an electronic device including a central processing unit, such as a computer, a mobile phone, a tablet computer, a television, a server, etc., without limitation. Figure 2 As shown, the method mainly includes the following steps S102 to S108:

[0039] Step S102, obtaining video samples and retrieving text samples.

[0040] In the disclosed embodiment, the video sample can be understood as video data used for training the video retrieval model. Each video sample has a corresponding retrieval text sample, and the retrieval text sample is used to obtain the video sample by inputting the retrieval text sample into the video retrieval model.

[0041] In the disclosed embodiment, the video sample may be a video or a video segment in a video. For example, a video may be split into video segment samples X1 to Xn every five seconds or ten seconds. For example, the search text sample corresponding to the video segment sample X1 may be "The male protagonist is sunbathing on the beach", so that the video segment sample X1 may be retrieved by inputting the search text sample "The male protagonist is sunbathing on the beach".

[0042] Step S104, obtaining visual features, subtitle text features and audio features corresponding to the video sample, and configuring weight parameters corresponding to the visual features, subtitle text features and audio features.

[0043] In the disclosed embodiment, the video sample includes multimodal data such as visual information, audio and text information, and the visual features, subtitle text features and audio features corresponding to the video sample can be obtained according to application needs.

[0044] In the disclosed embodiments, there are many ways to obtain visual features corresponding to video samples. As an example, the video samples are subjected to frame extraction processing according to a preset frame extraction frequency to obtain each video frame, and visual features are extracted from each video frame based on a pre-trained image feature extraction model to obtain visual features corresponding to the video sample; as another example, multiple video frames corresponding to the video sample are obtained, and visual features are extracted from the multiple video frames through a convolutional neural network to obtain visual features corresponding to the video sample. The above are only examples, and the settings can be selected according to application needs.

[0045] In the disclosed embodiments, there are many ways to obtain subtitle text features corresponding to video samples. As an example, audio information in the video sample is obtained, and the audio information is converted into first text information based on speech-to-text technology, and text features are extracted from the first text information based on a pre-trained text feature extraction model to obtain subtitle text features corresponding to the video sample; as another example, audio information in the video sample is obtained, and the audio information is converted into first text information based on speech-to-text technology, and optical character recognition is performed on each video frame based on image optical character recognition technology to obtain second text information, and text features are extracted from the first text information and / or the second text information based on a pre-trained text feature extraction model to obtain subtitle text features corresponding to the video sample. The above are only examples, and the settings can be selected according to application needs.

[0046] In the embodiments of the present disclosure, there are many ways to obtain audio features corresponding to video samples. As an example, audio information is extracted from video samples based on a preset audio processing tool to obtain audio information, and audio features are extracted from audio information based on a preset audio model to obtain audio features. As another example, video samples are input into a pre-trained large model to generate audio features. The above are only examples, and settings can be selected according to application needs.

[0047] In the embodiments of the present disclosure, initial weight parameters corresponding to visual features, subtitle text features and audio features are configured, such as a first weight coefficient corresponding to visual features, a second weight coefficient corresponding to subtitle text features and a third weight coefficient corresponding to audio features; wherein the sum of the first weight coefficient, the second weight coefficient and the third weight coefficient is 1; that is, a certain weight coefficient can be randomly assigned to each visual feature, subtitle text feature and audio feature, and subsequently, based on the training process, the weight parameters corresponding to the visual features, subtitle text features and audio features are continuously adjusted, so that the actual weight parameters corresponding to the visual features, subtitle text features and audio features can be determined, so that the video retrieval results based on the trained video retrieval model are more accurate and comprehensive.

[0048] Step S106, based on the initial weight parameters, the visual features, the subtitle text features and the audio features are fused to obtain the video features.

[0049] In the embodiments of the present disclosure, video features can be understood as key information of the video, including visual information, audio information, text information, and the like.

[0050] In the disclosed embodiment, there are many ways to obtain video features by fusing visual features, subtitle text features and audio features based on initial weight parameters. As an example, visual features, subtitle text features and audio features are weighted averaged according to weight parameters to obtain video features; as another example, visual features, subtitle text features and audio features are spliced ​​according to initial weight parameters and a preset feature splicing order to obtain video features. The above are only examples, and settings can be selected according to application needs.

[0051] Step S108, input the video features and the retrieval text samples into the video retrieval model to be trained, and when the loss value between the video features and the retrieval text features corresponding to the retrieval text samples is greater than or equal to a preset threshold, adjust the model parameters and initial weight parameters of the video retrieval model to be trained until the adjusted loss value is less than the preset threshold, thereby obtaining the video retrieval model.

[0052] In the embodiments of the present disclosure, there are many ways to obtain the retrieval text features corresponding to the retrieval text sample. As an example, the retrieval text features are extracted from the retrieval text sample by a neural network or an encoding model to obtain the retrieval text features; as another example, the retrieval text features are extracted from the retrieval text sample based on a pre-trained text feature extraction model to obtain the retrieval text features. The above are only examples, and the settings can be selected according to application needs.

[0053] In the disclosed embodiments, the similarity between the video features and the retrieval text features can be determined based on the similarity, for example, by performing a dot product between the video features and the retrieval text features to obtain the similarity, or by calculating the cosine value between the video features and the retrieval text features as the similarity, etc., and the similarity can be input into a plurality of loss functions of the pre-set model training to determine the loss value of this training. When the loss value is greater than or equal to a preset threshold (which can be set according to the needs of the actual application scenario), the model parameters and initial weight parameters of the video retrieval model to be trained are adjusted based on the loss value, so that the model parameters and initial weight parameters of the video retrieval model to be trained can be continuously adjusted based on multiple video samples and retrieval text samples until the adjusted loss value is less than the preset threshold, and finally the video retrieval model is obtained.

[0054] Among them, it can be understood that the weight parameters are also parameters that are continuously adjusted during model training, so that the actual weight parameters corresponding to the visual features, text features and audio features can be determined, so that the video retrieval results based on the trained video retrieval model are more accurate and comprehensive.

[0055] In the above embodiment, the visual features, text features and audio features corresponding to the video samples are fused according to certain weights to obtain video features for subsequent video retrieval model training, and the weight parameters are continuously adjusted during the model training process to determine the optimal weight parameters of the visual features, text features and audio features to obtain the video retrieval model. Thus, the full-modal feature information of the video is fused in a feature space, so that the trained video retrieval model can assign corresponding weights to features of different modalities, thereby making the retrieval results based on the video retrieval model more accurate and comprehensive.

[0056] In some embodiments, obtaining a video sample includes: obtaining a video to be processed, and processing the video to be processed according to a preset video splitting frequency to obtain multiple video segments as video samples; and / or obtaining a video identifier, and obtaining at least one video segment in a preset video segment library as a video sample based on the video identifier.

[0057] In the embodiment of the present disclosure, a video splitting frequency is preset, such as five seconds, or ten seconds, so that the video to be processed is processed according to the video splitting frequency to obtain multiple video clips as video samples.

[0058] In the disclosed embodiment, the video identifier can uniquely identify a video clip, and the video identifier and the corresponding video clip can be pre-associated and stored in a video clip library, so as to obtain at least one video clip as a video sample in the preset video clip library based on the video identifier.

[0059] Therefore, the video clips can be used as video samples, so as to subsequently perform video clip retrieval based on the video retrieval model, thereby improving the efficiency and effect of video clip retrieval.

[0060] In some embodiments, obtaining visual features corresponding to video samples includes: performing frame sampling on the video samples at a preset frame sampling frequency to obtain each video frame, and extracting visual features from each video frame based on a pre-trained image feature extraction model to obtain visual features corresponding to the video samples.

[0061] In the embodiment of the present disclosure, the frame extraction frequency is preset, for example, the video samples are extracted at one frame every two seconds to obtain each video frame.

[0062] In the disclosed embodiment, an image feature extraction model is pre-trained, for example, a multimodal (text and image) pre-training model CLIP (Contrastive Language-Image Pre-training) model, and each video frame is input into the CLIP model to obtain the visual features corresponding to the video sample.

[0063] Therefore, by extracting key frames from video samples and extracting visual features therefrom based on a pre-trained image feature extraction model, the accuracy and efficiency of obtaining visual features corresponding to the video samples can be further improved, thereby improving the accuracy and efficiency of model training.

[0064] In some embodiments, obtaining subtitle text features corresponding to a video sample includes: obtaining audio information in the video sample, and converting the audio information into first text information based on speech-to-text technology; and / or performing optical character recognition on each video frame based on image optical character recognition technology to obtain second text information; performing text feature extraction on the first text information and / or the second text information based on a pre-trained subtitle text feature extraction model to obtain subtitle text features corresponding to the video sample.

[0065] It is understandable that the subtitle text features include text features corresponding to dialogues and narrations in video samples. There may be some audio information in the video samples without dialogue information (displayed on the screen subtitles), and there may be some videos that only have dialogue information without corresponding audio information.

[0066] In the disclosed embodiment, the audio information of the video sample is converted into first text information through speech-to-text technology. The first text information can be understood as the text corresponding to the audio in the video sample, that is, the audio of the video sample is completely converted into text.

[0067] In the disclosed embodiment, the video sample can also be extracted to obtain individual video frames, and optical character recognition can be performed on each video frame using image optical character recognition technology to obtain second text information. The second text information can be understood as the text corresponding to the lines (subtitles displayed on the image screen) in the video sample, that is, all the lines (subtitles) of the video sample are converted into text.

[0068] In the disclosed embodiment, in order to improve processing efficiency, the first text information or the second text information can be selected to be input into a pre-trained text feature extraction model for text feature extraction to obtain subtitle text features corresponding to the video sample. In order to improve processing accuracy, the first text information and the second text information can be all input into a pre-trained text feature extraction model for text feature extraction to obtain subtitle text features corresponding to the video sample.

[0069] Among them, a text feature extraction model is pre-trained, such as a bidirectional encoder representation BERT (Bidirectional Encoder Representations from Transformers) model.

[0070] Therefore, the subtitle text features corresponding to the video sample can be determined by combining the audio and / or subtitles at the same time, further improving the accuracy and efficiency of subsequent model training.

[0071] In some embodiments, obtaining audio features corresponding to video samples includes: extracting audio information from video samples based on a preset audio processing tool to obtain audio information, and extracting audio features from the audio information based on a preset audio model to obtain audio features.

[0072] In the embodiments of the present disclosure, the preset audio processing tool may be FFmpeg, which may perform audio information extraction on video samples in real time to obtain audio information, wherein the audio information is background music and sound effect information in the video samples, and audio features are extracted from the audio information based on a preset audio model to obtain audio features such as Mel-Frequency Cepstrum (MFCC).

[0073] Therefore, after extracting the audio information of the video sample through the audio processing tool and extracting the audio features of the audio information through the audio model, the audio features are obtained, and the audio features are also used as part of the features of the video information. Then, subsequent video retrieval model training is carried out. The video retrieval model trained by combining visual information, dialogue information and audio information makes the retrieval results more accurate and comprehensive.

[0074] In some embodiments, visual features, text features and audio features are fused based on initial weight parameters to obtain video features, including: obtaining a first weight coefficient corresponding to the visual features, a second weight coefficient corresponding to the text features and a third weight coefficient corresponding to the audio features based on the initial weight parameters; and performing weighted averaging of the visual features, text features and audio features according to the first weight coefficient, the second weight coefficient and the third weight coefficient to obtain video features.

[0075] Specifically, the visual features, text features and audio features can be weighted averaged according to the corresponding weight parameters to obtain video features; wherein the initial weight parameters can be set according to application needs, for example, the weight parameters corresponding to the visual features, text features and audio features are the first weight coefficient A1, the second weight coefficient A2 and the third weight coefficient A3, respectively, and the first result of multiplying the visual features by the first weight coefficient A1 is added to the second result of the text features and the second weight coefficient A2, and the third result of multiplying the audio features by the third weight coefficient A3 is added, and finally the added results are averaged to obtain video features.

[0076] Therefore, visual features, text features and audio features can be fused based on weight parameters to obtain video features, further improving the accuracy and flexibility of feature fusion, thereby improving the accuracy and flexibility of subsequent model training.

[0077] In some embodiments, video features and retrieval text samples are input into a video retrieval model to be trained, and when the loss value between the video features and the retrieval text features corresponding to the retrieval text samples is greater than or equal to a preset threshold, the model parameters and initial weight parameters of the video retrieval model to be trained are adjusted until the adjusted loss value is less than the preset threshold, thereby obtaining a video retrieval model, including: calculating the similarity between the video features and the retrieval text features; determining a first loss value based on the similarity and a preset contrast loss function, determining a second loss value based on the similarity and a preset matching loss function, and determining a third loss value based on the similarity and a preset generative loss function; determining that the total loss value is greater than or equal to a preset threshold based on the first loss value, the second loss value, and the third loss value, adjusting the model parameters and initial weight parameters of the video retrieval model to be trained until the adjusted loss value is less than the preset threshold, thereby obtaining a video retrieval model.

[0078] In the disclosed embodiments, a loss function may be pre-set according to training requirements, such as a contrast loss function. The contrast loss function is a dimensionality reduction learning method that can bring similar samples closer together and push dissimilar samples farther apart. Therefore, when the correlation between video features and retrieved text features is relatively large, the loss value obtained by calculating through the contrast loss function is relatively small, thereby further improving the accuracy of the model. The contrast loss function may be, for example, a mean absolute error loss function. Another example is that the loss function is a matching loss function. The matching loss function is a function used to measure the difference between the model's predicted results and the actual results. The model can be optimized by minimizing the difference between the predicted values ​​and the true values, thereby improving the performance of the model in the retrieval matching task. The matching loss function may be, for example, a mean square error loss function. Another example is that the loss function is a generative loss function. The generative loss function may evaluate the difference between the generated results and the true results of the model, thereby guiding the learning process of the model so that it can generate more realistic and diverse data. The generative loss function may be, for example, a cross entropy loss function.

[0079] In the disclosed embodiment, multiple loss values ​​can be determined by multiple loss functions, and different weights are assigned according to the effects of different loss functions to perform weighted summation on the multiple loss values ​​to obtain a total loss value and compare it with a preset threshold. When the total loss value is greater than or equal to the preset threshold, it is necessary to adjust the model parameters and the weight parameters corresponding to the visual features, subtitle text features, and audio features. As an example, when the total loss value is greater than or equal to the preset threshold, one or more weight coefficients of the visual features, subtitle text features, and audio features can be adjusted. For example, the first weight coefficient corresponding to the visual feature is adjusted from r1 to r2, the second weight coefficient corresponding to the subtitle text feature is adjusted from r3 to r4, and the third weight coefficient corresponding to the audio feature is adjusted from r5 to r6. After adjusting the model parameters and the weight parameters corresponding to the visual features, subtitle text features, and audio features, continue to make a judgment based on the total loss value between the video features and the retrieval text features corresponding to the retrieval text sample and the preset threshold. If the total loss value is greater than or equal to the preset threshold, continue to adjust the model parameters and weight parameters until the adjusted loss value is less than the preset threshold to obtain a video retrieval model.

[0080] It can also be understood that the final weight parameter determined when the total loss value is less than the preset threshold value can determine the final weight coefficient corresponding to the visual features, subtitle text features and audio features of the video sample, so that in subsequent real-time retrieval, the retrieval results can be determined based on the final weight parameters corresponding to the visual features, subtitle text features and audio features.

[0081] In the disclosed embodiment, the video features and the retrieval text features can be processed by dot multiplication or the like to obtain the similarity between the video features and the retrieval text features, and the similarity can be input into various pre-set loss functions for calculation to obtain multiple loss values, and the total loss value, i.e., the final loss value, can be determined based on the multiple loss values. It can be understood that when the final loss value is less than a certain threshold, the training can be stopped. If the final loss value is greater than or equal to the threshold, the model parameters and initial weight parameters of the video retrieval model to be trained need to be adjusted, and then the model training is continued until the final loss value is less than the threshold to obtain the video retrieval model.

[0082] In this way, the video retrieval model can be accurately and quickly obtained, thereby improving the accuracy of subsequent video retrieval.

[0083] Based on the description of the aforementioned embodiments, we can understand common problems in video retrieval, such as the fact that existing video retrieval methods mainly rely on visual and text information, while ignoring audio information, which results in that in actual video retrieval, some audio-related plots are difficult to be accurately retrieved; for example, some special clips in film and television dramas that mainly use music or sound effects to advance the plot.

[0084] To solve this problem, the present invention proposes a training method based on a full-modal video model, where full-modal information includes visual information, text information, and audio information. By pre-training the model on a large amount of data, integrating full-modal feature information and expressing it in a feature space, the features of different modalities can be effectively matched, thereby improving the accuracy of video retrieval.

[0085] Specifically, we first preprocess the video samples to extract visual features, subtitle text features, and audio features. Then, we use the pre-trained model to learn the contribution of various features to the plot, and integrate the feature information of all modalities into a feature space through comparison loss function, matching loss function, generative loss function, etc., so that the video retrieval model can assign corresponding weights to the features of different modalities, thereby obtaining a video retrieval model.

[0086] It should be noted that after obtaining the video retrieval model, the video samples and retrieval text samples can be used again to fine-tune the video retrieval model to obtain the final video retrieval model, which can not only more comprehensively utilize the various modal information in the video samples, but also improve the accuracy and efficiency of video retrieval.

[0087] Figure 2 The present invention provides a flow chart of a video retrieval model training method, which mainly includes the following steps S202 to S214:

[0088] Step S202, obtaining video samples and retrieving text samples.

[0089] Step S204, performing frame extraction processing on the video sample according to a preset frame extraction frequency to obtain each video frame, and extracting visual features from each video frame based on a pre-trained image feature extraction model to obtain visual features corresponding to the video sample.

[0090] Step S206, obtaining audio information from the video sample, and converting the audio information into first text information based on speech-to-text technology, performing optical character recognition on each video frame based on image optical character recognition technology to obtain second text information, performing text feature extraction on the first text information and the second text information based on a pre-trained text feature extraction model to obtain subtitle text features corresponding to the video sample.

[0091] Step S208: extracting audio information from the video sample based on a preset audio processing tool to obtain audio information, and extracting audio features from the audio information based on a preset audio model to obtain audio features.

[0092] Step S210, based on the weight parameters, obtain the first weight coefficient corresponding to the visual feature, the second weight coefficient corresponding to the subtitle text feature, and the third weight coefficient corresponding to the audio feature, and perform weighted averaging processing on the visual feature, the subtitle text feature, and the audio feature according to the first weight coefficient, the second weight coefficient, and the third weight coefficient to obtain the video feature.

[0093] Step S212, calculate the similarity between the video feature and the retrieved text feature, determine a first loss value based on the similarity and a preset contrast loss function, determine a second loss value based on the similarity and a preset matching loss function, and determine a third loss value based on the similarity and a preset generative loss function.

[0094] Step 214, when it is determined based on the first loss value, the second loss value and the third loss value that the total loss value is greater than or equal to a preset threshold, the model parameters and initial weight parameters of the video retrieval model to be trained are adjusted until the adjusted loss value is less than the preset threshold, thereby obtaining the video retrieval model.

[0095] Specifically, all modal information in a video sample: visual information, audio information, and textual information are considered to be complementary to each other, and then they are fused together for video retrieval; among them, audio information is an important component of video retrieval in many practical usage scenarios, and ignoring it will lead to inaccurate retrieval results.

[0096] Therefore, by preprocessing video samples to extract various modal features, the pre-trained model can learn the shared representation of these features; throughout the process, the model will learn how to best map and fuse all modal features through contrast loss functions, matching loss functions, generative loss functions, etc. Models with this mapping and fusion capability can unify various features into a feature space, which enables the model to accurately evaluate the relationship between different modal features and perform accurate video retrieval based on this.

[0097] At the same time, in order to adapt to new data and tasks, the video retrieval model can also be fine-tuned. That is, after inputting new text to be retrieved and the corresponding video, the video retrieval model will automatically adjust its parameters to adapt to the new data, thereby further improving the accuracy and efficiency of the retrieval. This flexible adjustment and optimization strategy is conducive to the video retrieval model maintaining good performance in ever-changing practical tasks.

[0098] First, the video sample is processed: the given video sample is processed to extract the visual features and subtitle text features of the video sample; for the visual features, image processing technology can be used to extract key video frames from the video sample and extract visual features from them; for the subtitle text features, speech recognition technology can be used to convert the audio into text information, and extract the subtitle text features from the text information; then, audio features are generated, such as extracting audio features from the video sample based on a preset audio processing tool to obtain audio features.

[0099] That is to say, the video samples are first preprocessed, including frame extraction, audio extraction, and dialogue text extraction. For the visual part, visual feature extractors such as convolutional neural networks are used to complete frame feature extraction; for the audio part, audio feature extraction technology such as Mel-frequency cepstral coefficient (MFCC) is used for feature extraction; for the text part, natural language processing technology such as word embedding models such as word2vec, or Bert is used to extract text features.

[0100] Furthermore, the generated audio features, visual features and subtitle text features are integrated to form new video features; for visual features, audio features and subtitle text features, firstly, a deep learning model is trained separately for each modality so that it can be expressed in a feature space, and then, in the loss function design, contrast loss function, matching loss function, generative loss function, etc. are used to train the video retrieval model so that the video retrieval model can process information of various modalities in a feature space; wherein, during the model training process, weight coefficients are dynamically assigned according to the contribution of the features of each modality to the final retrieval results; for example, the weight coefficient of visual information may reflect the contribution of visual information to the retrieval results, and then this weight coefficient is assigned to the visual feature accordingly.

[0101] Therefore, when a user provides a search text query, the degree of match is calculated by comparing it with the full-modal features of all videos, and the degree of match is used as the basis for sorting. The most relevant videos are returned, and the corresponding model parameters and weight parameters can be fine-tuned based on user feedback in the future.

[0102] As an example, take the video sample as a video clip sample, such as Figure 3 As shown, a video clip sample is input, and the video clip sample is extracted at a rate of one frame per two seconds to obtain each video frame, such as 150 frames of images. Then, an open source model, such as the CLIP model, is used to extract the visual features of each image to obtain the visual features of each video frame, such as 150 frames of images; the video clip sample is processed by an audio processing tool, such as FFmpeg, to generate audio information, and the features of the audio information are extracted by an open source audio model to obtain audio features; the video clip sample is processed by using open source OCR (Optical Character Recognition) technology to obtain N lines of dialogue (text information), and the features of the N lines of dialogue are extracted by using an open source model, such as the BERT model, to obtain N text features; wherein N is a positive integer, and the visual features, audio features, and text features of each video frame are fused to obtain video features; the video features and the retrieved text samples are input into the target platform (such as the HERO architecture) and the model is trained in combination with the loss function to obtain a video retrieval model.

[0103] Therefore, by fusing the visual features, subtitle text features and audio features corresponding to the video samples according to certain weights, the video features are obtained for the subsequent training of the video retrieval model, and the weight parameters are continuously adjusted during the model training process to determine the optimal weight parameters of the visual features, text features and audio features to obtain the video retrieval model. Thus, the full modal feature information of the video is fused in a feature space, so that the trained video retrieval model can assign corresponding weights to the features of different modalities, so that the retrieval results based on the video retrieval model are more accurate and comprehensive.

[0104] Figure 4 The following is a flow chart of a video retrieval method provided by an embodiment of the present disclosure. The method mainly includes the following steps S402 to S404:

[0105] Step S402, obtaining the text to be searched.

[0106] Step S404: searching in a preset candidate video library according to the text to be searched based on the video retrieval model to obtain the target video.

[0107] In the embodiment of the present disclosure, the video retrieval model is obtained by the training method of the video retrieval model described in the aforementioned embodiment.

[0108] In this embodiment, the text to be retrieved is content input by the user according to actual video retrieval requirements. By inputting the text to be retrieved into the video retrieval model, the target video matching the text to be retrieved can be obtained.

[0109] As a possible implementation method, the features to be retrieved corresponding to the text to be retrieved are obtained, and the candidate visual features, candidate text features and candidate audio features corresponding to each candidate video in the video library are obtained; the target similarity is obtained by calculating the candidate weight parameters and the features to be retrieved based on the candidate visual features, candidate text features and candidate audio features through the video retrieval model; all target similarities are sorted, and the candidate video corresponding to the maximum target similarity is obtained as the target video.

[0110] Specifically, when a user initiates a video retrieval request, the search text entered by the user and the integrated video features corresponding to all videos in the video library are input into the video retrieval model to return the most relevant target video.

[0111] For example, taking the video as a video clip, such as Figure 5As shown, all videos in the video are obtained, and at the same time, the video is extracted at a rate of one frame per two seconds to obtain each video frame, and then the visual features of each image are extracted using an open source model such as the CLIP model to obtain the visual features of each video frame; the video samples are processed by an audio processing tool such as FFmpeg to generate audio information, and the features of the audio information are extracted using an open source audio model to obtain audio features; the video is processed using open source OCR (Optical Character Recognition) technology to obtain text information, and the features of the text information are extracted using an open source model such as the BERT model to obtain text features; the visual features, audio features and text features of each video frame are fused to obtain the features of each video; the features of each video and the search text are input into the video retrieval model to determine the target video from all videos.

[0112] Therefore, the retrieval results of video retrieval based on the video retrieval model that combines multimodal features such as visual features, text features and audio features are more accurate and comprehensive.

[0113] Corresponding to the aforementioned video retrieval model training method, the present disclosure further provides a video retrieval model training device. Figure 6 A schematic diagram of the structure of a video retrieval model training device provided in an embodiment of the present disclosure, the device can be implemented by software and / or hardware, and the device includes:

[0114] The first acquisition module 602 is used to acquire video samples and retrieve text samples.

[0115] The second acquisition module 604 is used to acquire the visual features corresponding to the video sample.

[0116] The third acquisition module 606 is used to acquire the subtitle text features corresponding to the video sample.

[0117] The fourth acquisition module 608 is used to acquire the audio features corresponding to the video sample.

[0118] The configuration module 610 is used to configure initial weight parameters corresponding to the visual features, the subtitle text features and the audio features.

[0119] The fusion module 612 is used to fuse the visual features, the subtitle text features and the audio features based on the initial weight parameters to obtain video features.

[0120] The training module 614 is used to input the video features and the retrieval text samples into the video retrieval model to be trained, and adjust the model parameters and the initial weight parameters of the video retrieval model to be trained when the loss value between the video features and the retrieval text features corresponding to the retrieval text samples is greater than or equal to a preset threshold, until the adjusted loss value is less than the preset threshold, so as to obtain a video retrieval model.

[0121] The above-mentioned device provided by the embodiment of the present disclosure obtains video features for training a video retrieval model by fusing multimodal features such as visual features, subtitle text features and audio features according to certain weight parameters, and continuously adjusts the weight parameters during the model training process to determine the optimal weight parameters of visual features, subtitle text features and audio features to obtain a video retrieval model. Since the video features combine multimodal information such as visual information, dialogue information and audio information of the video, the retrieval results of video retrieval based on the video retrieval model are more accurate and comprehensive.

[0122] In some embodiments, the first acquisition module 602 is specifically used to: obtain a video to be processed, and process the video to be processed according to a preset video splitting frequency to obtain multiple video segments as the video samples; and / or obtain a video identifier, and based on the video identifier, obtain at least one video segment in a preset video segment library as the video sample, and obtain a retrieval text sample.

[0123] In some embodiments, the second acquisition module 604 is specifically used to: perform frame extraction processing on the video sample at a preset frame extraction frequency to obtain each video frame; perform visual feature extraction on each video frame based on a pre-trained image feature extraction model to obtain visual features corresponding to the video sample.

[0124] In some embodiments, the third acquisition module 606 is specifically used to: obtain audio information in the video sample, and convert the audio information into first text information based on speech-to-text technology; and / or perform optical character recognition on each video frame based on image optical character recognition technology to obtain second text information; perform text feature extraction on the first text information and / or the second text information based on a pre-trained text feature extraction model to obtain subtitle text features corresponding to the video sample.

[0125] In some implementations, the fourth acquisition module 608 is specifically used to: extract audio information from the video sample based on a preset audio processing tool to obtain audio information; and extract audio features from the audio information based on a preset audio model to obtain the audio features.

[0126] In some embodiments, the fusion module 612 is specifically used to: obtain a first weight coefficient corresponding to the visual feature, a second weight coefficient corresponding to the subtitle text feature, and a third weight coefficient corresponding to the audio feature based on the initial weight parameters; and perform weighted averaging of the visual feature, the subtitle text feature, and the audio feature according to the first weight coefficient, the second weight coefficient, and the third weight coefficient to obtain the video feature.

[0127] In some embodiments, the training module 614 is specifically used to: calculate the similarity between the video features and the retrieval text features; determine a first loss value based on the similarity and a preset contrast loss function, determine a second loss value based on the similarity and a preset matching loss function, and determine a third loss value based on the similarity and a preset generative loss function; adjust the model parameters and the initial weight parameters of the video retrieval model to be trained when the total loss value is greater than a preset threshold based on the first loss value, the second loss value, and the third loss value, until the adjusted loss value is less than the preset threshold, thereby obtaining the video retrieval model.

[0128] The video retrieval model training device provided in the embodiments of the present disclosure can execute the video retrieval model training method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0129] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0130] Corresponding to the aforementioned video retrieval method, the embodiment of the present disclosure further provides a video retrieval device, Figure 7 This is a schematic diagram of the structure of a video retrieval device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware. The device includes:

[0131] The fifth acquisition module 702 is used to acquire the text to be searched.

[0132] The retrieval module 704 is used to search in a preset candidate video library according to the text to be retrieved based on a video retrieval model to obtain a target video.

[0133] Wherein, the video retrieval model is obtained according to the training method of the video retrieval model described in the aforementioned embodiment.

[0134] In some embodiments, the retrieval module 704 is specifically used to: obtain the features to be retrieved corresponding to the text to be retrieved, and obtain the candidate visual features, candidate text features and candidate audio features corresponding to each candidate video in the video library; calculate the target similarity based on the candidate weight parameters corresponding to the candidate visual features, the candidate text features and the candidate audio features and the features to be retrieved through the video retrieval model; sort according to all the target similarities, and obtain the candidate video corresponding to the maximum target similarity as the target video.

[0135] The above-mentioned device provided by the embodiment of the present disclosure makes the retrieval results of video retrieval based on the video retrieval model based on multimodal features such as visual features, text features and audio features more accurate and comprehensive.

[0136] The video retrieval device provided in the embodiments of the present disclosure can execute the video retrieval method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0137] An embodiment of the present disclosure provides an electronic device, which includes: a storage device on which a computer program is stored; and a processing device for executing the computer program in the storage device to implement the steps of any method in the present disclosure.

[0138] Reference below Figure 8 , which shows a schematic diagram of the structure of an electronic device 800 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0139] like Figure 8 As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0140] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0141] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0142] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the image processing method provided by the embodiments of the present disclosure. The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present disclosure, the programming languages ​​including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0143] In addition, the embodiments of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the video retrieval model training method and video retrieval method provided by the embodiments of the present disclosure.

[0144] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0145] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the video retrieval model training method and video retrieval method in the embodiments of the present disclosure.

[0146] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0147] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0148] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0149] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0150] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0151] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video retrieval model training method, characterized in that: The method comprises: Obtain video samples and retrieve text samples; Acquire visual features, subtitle text features, and audio features corresponding to the video sample, and configure initial weight parameters corresponding to the visual features, the subtitle text features, and the audio features; Based on the initial weight parameters, the visual features, the subtitle text features and the audio features are subjected to feature fusion to obtain video features; The video features and the retrieval text samples are input into the video retrieval model to be trained, and when the loss value between the video features and the retrieval text features corresponding to the retrieval text samples is greater than or equal to a preset threshold, the model parameters and the initial weight parameters of the video retrieval model to be trained are adjusted until the adjusted loss value is less than the preset threshold, thereby obtaining a video retrieval model.

2. The method according to claim 1, characterized in that: The obtaining of the video sample comprises: Obtaining a video to be processed, and processing the video to be processed according to a preset video splitting frequency to obtain a plurality of video segments as the video samples; and / or, A video identifier is obtained, and based on the video identifier, at least one video segment is obtained from a preset video segment library as the video sample.

3. The method according to claim 1, characterized in that The obtaining of visual features corresponding to the video sample includes: Performing frame extraction processing on the video samples according to a preset frame extraction frequency to obtain video frames; Visual features are extracted from each video frame based on a pre-trained image feature extraction model to obtain visual features corresponding to the video sample.

4. The method according to claim 3, characterized in that: The obtaining of subtitle text features corresponding to the video sample includes: Acquire audio information in the video sample, and convert the audio information into first text information based on speech-to-text technology; and / or, Performing optical character recognition on each video frame based on image optical character recognition technology to obtain second text information; Based on a pre-trained text feature extraction model, text feature extraction is performed on the first text information and / or the second text information to obtain subtitle text features corresponding to the video sample.

5. The method according to claim 1, characterized in that The obtaining of the audio features corresponding to the video sample includes: Extracting audio information from the video sample based on a preset audio processing tool to obtain audio information; The audio features are extracted from the audio information based on a preset audio model to obtain the audio features.

6. The method according to claim 1, characterized in that The step of fusing the visual features, the subtitle text features, and the audio features based on the initial weight parameters to obtain video features includes: Based on the initial weight parameter, obtain a first weight coefficient corresponding to the visual feature, a second weight coefficient corresponding to the subtitle text feature, and a third weight coefficient corresponding to the audio feature; The visual feature, the subtitle text feature and the audio feature are weighted averaged according to the first weight coefficient, the second weight coefficient and the third weight coefficient to obtain the video feature.

7. The method according to claim 1, characterized in that The video features and the retrieval text samples are input into the video retrieval model to be trained, and when the loss value between the video features and the retrieval text features corresponding to the retrieval text samples is greater than or equal to a preset threshold, the model parameters and the initial weight parameters of the video retrieval model to be trained are adjusted until the adjusted loss value is less than the preset threshold, thereby obtaining a video retrieval model, including: Calculating the similarity between the video feature and the retrieved text feature; Determine a first loss value based on the similarity and a preset contrast loss function, determine a second loss value based on the similarity and a preset matching loss function, and determine a third loss value based on the similarity and a preset generative loss function; When the total loss value is determined to be greater than or equal to a preset threshold based on the first loss value, the second loss value and the third loss value, the model parameters and the initial weight parameters of the video retrieval model to be trained are adjusted until the adjusted loss value is less than the preset threshold, thereby obtaining the video retrieval model.

8. A video retrieval method, characterized in that: The method comprises: Get the text to be retrieved; Based on the video retrieval model, searching in a preset candidate video library according to the text to be retrieved to obtain a target video; Wherein, the video retrieval model is obtained according to the training method of the video retrieval model described in any one of claims 1-6.

9. The method according to claim 8, characterized in that The video retrieval model is used to search in a preset candidate video library according to the text to be retrieved to obtain a target video, including: Acquire the to-be-retrieved features corresponding to the to-be-retrieved text, and acquire candidate visual features, candidate text features, and candidate audio features corresponding to each candidate video in the video library; The video retrieval model is used to calculate the target similarity based on the candidate weight parameters corresponding to the candidate visual features, the candidate text features and the candidate audio features and the features to be retrieved; All the target similarities are sorted, and the candidate video corresponding to the maximum target similarity is obtained as the target video.

10. A video retrieval model training device, characterized in that: include: A first acquisition module, used to acquire video samples and retrieve text samples; A second acquisition module, used to acquire visual features corresponding to the video sample; A third acquisition module is used to obtain the subtitle text features corresponding to the video sample; A fourth acquisition module, used to acquire audio features corresponding to the video sample; A configuration module, used to configure initial weight parameters corresponding to the visual features, the subtitle text features and the audio features; A fusion module, used for fusing the visual features, the subtitle text features and the audio features based on the initial weight parameters to obtain video features; A training module is used to input the video features and the retrieval text samples into the video retrieval model to be trained, and when the loss value between the video features and the retrieval text features corresponding to the retrieval text samples is greater than or equal to a preset threshold, adjust the model parameters and the initial weight parameters of the video retrieval model to be trained until the adjusted loss value is less than the preset threshold, thereby obtaining a video retrieval model.

11. A video retrieval device, characterized in that: include: The fifth acquisition module is used to acquire the text to be searched; A retrieval module, used to search in a preset candidate video library according to the text to be retrieved based on a video retrieval model to obtain a target video; Wherein, the video retrieval model is obtained according to the training method of the video retrieval model described in any one of claims 1-6.

12. An electronic device, characterized in that: The electronic device comprises: a storage device having a computer program stored thereon; A processing device for executing the computer program in the storage device to implement the steps of the video retrieval model training method described in any one of claims 1 to 7, or the video retrieval method described in claim 8 or 9.

13. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which is used to execute the steps of the video retrieval model training method described in any one of claims 1 to 7, or the video retrieval method described in claim 8 or 9.

Images