Video retrieval method, apparatus, device, and storage medium

Abstract

The present disclosure relates to a video retrieval method, device, equipment and storage medium. In the embodiments of the present application, the sentences in the long text to be retrieved are subjected to vectorization processing to obtain a first sentence vector corresponding to each sentence, and a second sentence vector reflecting the context relationship between the sentences is generated according to the time sequence relationship between the sentences in the long text to be retrieved and the first sentence vector corresponding to each sentence. The second sentence vector of each sentence in the long text to be retrieved can reflect the information of other sentences, so that the video information composed of the video clips retrieved based on the second sentence vector in the long text to be retrieved can better match the theme of the long text to be retrieved, and the efficiency and accuracy of video retrieval are improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of video retrieval, and particularly relates to a video retrieval method, device, equipment and storage medium BACKGROUND

[0002] With the rapid development of the Internet, video has become an increasingly important information transmission carrier and is widely used in application software or social networks. At present, users often retrieve video content of interest based on text information.

[0003] A traditional video retrieval scheme retrieves a single sentence, that is, one video content is retrieved for one sentence. However, for long text containing multiple sentences, directly applying the traditional video retrieval scheme does not achieve an ideal retrieval effect, for example, the retrieved video has a low degree of matching with the theme of the long text. SUMMARY

[0004] The present disclosure provides a video retrieval method, device, equipment and storage medium to at least solve the problem that the video retrieval effect is not ideal for long text containing multiple sentences. The technical scheme of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a video retrieval method is provided, including: obtaining a long text to be retrieved, the long text to be retrieved including at least two sentences, and a time sequence relationship existing between the at least two sentences; performing vectorization processing on each sentence according to a word included in the sentence to obtain a first sentence vector corresponding to the sentence; performing association processing on the first sentence vector corresponding to each sentence and first sentence vectors corresponding to other sentences according to the time sequence relationship to obtain a second sentence vector corresponding to the sentence, the other sentences including all or part of the sentences except the sentence in the at least two sentences; performing retrieval in a video vector library according to the second sentence vectors respectively corresponding to the at least two sentences to obtain video clips respectively adapted to the at least two sentences; and generating video information corresponding to the long text to be retrieved according to the video clips respectively adapted to the at least two sentences.

[0006] Optionally, the vectorization processing on the at least two sentences respectively according to the words included in the sentences to obtain the first sentence vectors respectively corresponding to the at least two sentences includes: obtaining word vectors corresponding to a plurality of words included in each sentence; inputting the word vectors corresponding to the plurality of words into a sentence vector generation network layer in a video retrieval model; and performing synthesis processing on the word vectors corresponding to the plurality of words by the sentence vector generation network layer to obtain the first sentence vector corresponding to each sentence.

[0007] Optionally, according to the time sequence relationship, the first sentence vector corresponding to each sentence is associated with the first sentence vectors corresponding to other sentences to obtain a second sentence vector corresponding to the sentence, including: inputting the first sentence vector corresponding to each sentence into a sentence vector association network layer in the video retrieval model according to the time sequence relationship, and performing association processing on the first sentence vector corresponding to each sentence and the first sentence vectors corresponding to other sentences in the sentence vector association network layer to obtain a second sentence vector corresponding to each sentence.

[0008] Optionally, the video vector library includes existing video clips and corresponding video vectors, and the second sentence vectors corresponding to the at least two sentences are respectively searched in the video vector library to obtain video clips adapted to the at least two sentences respectively, including: inputting the second sentence vector corresponding to each sentence into a video retrieval network layer in the video retrieval model, calculating the similarity between each second sentence vector and the video vectors in the video vector library through the video retrieval network layer, and taking the existing video clip corresponding to the video vector satisfying the set similarity condition as the video clip adapted to the second sentence vector, thereby obtaining the video clips adapted to the at least two sentences respectively.

[0009] Optionally, the method further includes: preselecting existing video clips; for any existing video clip, obtaining a plurality of image frames in the existing video clip, inputting the plurality of image frames into a video vector generation network layer in the video retrieval model, performing feature extraction on the plurality of image frames to obtain feature vectors corresponding to the plurality of image frames respectively, and fusing the feature vectors corresponding to the plurality of image frames respectively to obtain a video vector corresponding to the existing video clip.

[0010] Optionally, it further includes: obtaining sentence samples and corresponding video clip samples; for each sentence sample, generating a sentence vector of the sentence sample and a video vector of the video clip sample corresponding to the sentence sample as a set of training samples; inputting the sentence vectors in the training samples as the input of the video retrieval model, inputting the video vectors in the training samples as the expected model output for multiple model training, and calculating the loss function of each training, adjusting the model parameters according to the loss function of each training, until the loss function meets the set requirements, to obtain the video retrieval model; wherein the loss function is the difference information between the actual model output and the expected model output.

[0011] According to a second aspect of the embodiments of the present disclosure, a video retrieval device is provided, comprising an acquisition module, a processing module, an association module, a retrieval module and a generation module; the acquisition module is configured to acquire a long text to be retrieved, the long text to be retrieved comprising at least two sentences, and there being a time sequence relationship between the at least two sentences; the processing module is configured to perform vectorization processing on each sentence according to the words contained in the sentence, to obtain a first sentence vector corresponding to each sentence; the association module is configured to perform association processing on the first sentence vector corresponding to each sentence and the first sentence vectors corresponding to other sentences according to the time sequence relationship, to obtain a second sentence vector corresponding to the sentence, the other sentences comprising all or part of the sentences other than the sentence in the at least two sentences; the retrieval module is configured to perform retrieval in a video vector library according to the second sentence vectors corresponding to the at least two sentences respectively, to obtain video clips adapted to the at least two sentences respectively; and the generation module is configured to generate video information corresponding to the long text to be retrieved according to the video clips adapted to the at least two sentences respectively.

[0012] Optionally, the processing module is specifically configured to: acquire word vectors corresponding to a plurality of words included in each sentence; and input the word vectors corresponding to the plurality of words into a sentence vector generation network layer in a video retrieval model, to perform synthesis processing on the plurality of word vectors through the sentence vector generation network layer, to obtain the first sentence vector corresponding to each sentence.

[0013] Optionally, the association module is specifically configured to: input the first sentence vector corresponding to each sentence into a sentence vector association network layer in a video retrieval model according to the time sequence relationship, and perform association processing on the first sentence vector corresponding to each sentence and the first sentence vectors corresponding to other sentences in the sentence vector association network layer, to obtain the second sentence vector corresponding to each sentence.

[0014] Optionally, the video vector library comprises existing video clips and video vectors corresponding to the existing video clips, and the retrieval module is specifically configured to: input the second sentence vector corresponding to each sentence into a video retrieval network layer in a video retrieval model, to calculate the similarity between each second sentence vector and the video vectors in the video vector library through the video retrieval network layer, to take the existing video clip corresponding to the video vector satisfying a set similarity condition as the video clip adapted to the sentence corresponding to the second sentence vector, and to thereby obtain the video clips adapted to the at least two sentences respectively.

[0015] Optionally, the video retrieval apparatus further comprises a selection module, the selection module is configured to preselect the existing video segments; and the processing module is further configured to: for any existing video segment, acquire a plurality of image frames in the existing video segment, input the plurality of image frames into a video vector generation network layer in the video retrieval model, perform feature extraction on the plurality of image frames to obtain a plurality of feature vectors corresponding to the plurality of image frames respectively, and fuse the plurality of feature vectors corresponding to the plurality of image frames respectively to obtain a video vector corresponding to the existing video segment.

[0016] Optionally, the acquisition module is further configured to acquire sentence samples and video segment samples corresponding to the sentence samples; the generation module is further configured to: for each sentence sample, generate a sentence vector of the sentence sample, and generate a video vector of a video segment sample corresponding to the sentence sample as a training sample; and the processing module is further configured to: take the sentence vector in the training sample as an input of the video retrieval model, perform multiple model training by taking the video vector in the training sample as an expected model output, calculate a loss function of each training, adjust model parameters according to the loss function of each training, and obtain the video retrieval model when the loss function meets a set requirement.

[0017] According to a third aspect of the embodiments of the present disclosure, a video retrieval device is provided, comprising a memory and a processor; the memory is configured to store a computer program; the processor is coupled with the memory and is configured to execute the computer program to implement the steps in the video retrieval method provided by the embodiments of the present disclosure.

[0018] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which causes a processor to implement the steps in the video retrieval method provided by the embodiments of the present disclosure when the computer program is executed by the processor.

[0019] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, which comprises a computer program, and causes a processor to implement the steps in the video retrieval method provided by the embodiments of the present disclosure when the computer program is executed by the processor.

[0020] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:

[0021] In the embodiments of the present application, the sentences in the long text to be searched are subjected to vectorization processing to obtain a first sentence vector corresponding to each sentence, and a second sentence vector reflecting the context relationship between the sentences is generated according to the time sequence relationship between the sentences in the long text to be searched and the first sentence vector corresponding to each sentence. The second sentence vector of each sentence in the long text to be searched can reflect the information of other sentences, so that the video information composed of the video clips retrieved based on the second sentence vector of the long text to be searched can better match the theme of the long text to be searched, improving the efficiency and accuracy of video retrieval.

[0022] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure, and do not constitute an improper limitation on the present disclosure.

[0024] Figure 1 is a flowchart of a video retrieval method according to an exemplary embodiment.

[0025] Figure 2 is a block diagram of a video retrieval model according to an exemplary embodiment.

[0026] Figure 3 is a flowchart of a video retrieval device according to an exemplary embodiment.

[0027] Figure 4 is a flowchart of a video retrieval device according to an exemplary embodiment. DETAILED DESCRIPTION

[0028] In order to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings.

[0029] It should be noted that the terms "first", "second" and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all the embodiments consistent with the present disclosure. On the contrary, they are only examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0030] Figure 1 is a flowchart of a video retrieval method according to an exemplary embodiment, as shown in Figure 1 The video retrieval method includes the following steps, as shown in

[0031] 101. Obtain a long text to be retrieved, the long text to be retrieved including at least two sentences, and there being a time sequence relationship between the at least two sentences;

[0032] 102. Perform vectorization processing on each sentence according to the words contained in the sentence, to obtain a first sentence vector corresponding to each sentence;

[0033] 103. According to the time sequence relationship, perform association processing on the first sentence vector corresponding to each sentence and the first sentence vectors corresponding to other sentences, to obtain a second sentence vector corresponding to the sentence, the other sentences including all or part of the sentences other than the sentence among the at least two sentences;

[0034] 104. Perform retrieval in a video vector library respectively according to the second sentence vectors corresponding to the at least two sentences, to obtain video clips respectively adapted to the at least two sentences;

[0035] 105. Generate video information corresponding to the long text to be retrieved according to the video clips respectively adapted to the at least two sentences.

[0036] In this embodiment, the long text to be retrieved refers to a text that needs to retrieve video information according to the long text, the long text to be retrieved including at least two sentences, and there being a time sequence relationship between the at least two sentences, which can reflect the time order of the appearance of the sentences. The manner of obtaining the long text to be retrieved is not limited. For example, the video retrieval method is implemented on an electronic terminal, the electronic terminal is installed with a video retrieval APP, the video retrieval APP shows a text input interface to a user, the text input interface includes an editing control, and in response to a triggering operation of the editing control by the user, a long text input by the user is obtained, and the long text is taken as the long text to be retrieved. For another example, the video retrieval method is implemented on an electronic terminal, the electronic terminal is provided with a graphical user interface, the graphical user interface includes a video retrieval interface, the video retrieval interface includes a text import control, and in response to a triggering operation of the text import control by the user, a file that can be imported is displayed, a text file selected by the user is parsed to obtain a long text, and the long text is taken as the long text to be retrieved.

[0037] In this embodiment, after obtaining the long text to be searched, for any sentence contained in the long text to be searched, each sentence contains words, and according to the words contained in each sentence, the sentence is vectorized to obtain a sentence vector corresponding to the sentence. In order to distinguish and describe, the sentence vector obtained by vectorizing the sentence is called the first sentence vector, so that at least two sentence vectors corresponding to the at least two sentences contained in the long text to be searched can be obtained. Wherein, the vectorization processing refers to the processing mode of converting the sentence into a real number vector. For example, the vectorization of the sentence can be to split the sentence into words (which can be one word, two words or multiple words), to pre-set the vector of each word, the dimension of the vector is the size of the dictionary, the current position of the word is represented by 1, and the other positions are represented by 0, and the frequency of the word appearing in the sentence is used as the weight, and the sentence vector is calculated according to the weight of each word.

[0038] In this embodiment, according to the time sequence relationship between the at least two sentences contained in the long text to be searched, the first sentence vector corresponding to each sentence and the first sentence vector corresponding to other sentences can be associated to obtain a second sentence vector corresponding to the sentence, wherein the other sentences include all or part of the sentences in the at least two sentences in the long text to be searched, except the sentence; the second sentence vector corresponding to each sentence reflects the context relationship between the sentence and other sentences, so as to obtain the information of other sentences for each sentence. Wherein, the association processing can be a weighted multiplication operation, or can be input into a recurrent neural network (RNN) model to perform a recurrent convolution operation, so as to realize the operation of associating the first sentence vector corresponding to each sentence with the first sentence vector corresponding to other sentences.

[0039] For example, the order of each sentence in the time sequence can be determined according to the time sequence relationship between at least two sentences, for each sentence, one or more sentences before the sentence in the time sequence are regarded as other sentences, and the first sentence vector corresponding to each sentence is associated with the first sentence vector corresponding to the other sentence. For example, a long text includes three sentences, namely, sentence A1, sentence A2 and sentence A3, and the time sequence relationship of the three sentences is: sentence A1, sentence A2 and sentence A3. According to the time sequence relationship, there is no sentence before sentence A1, and therefore the first sentence vector of sentence A1 is associated to obtain a second sentence vector which is the same as the first sentence vector. The first sentence vector corresponding to sentence A2 is associated, which means that the first sentence vectors of sentence A1 and sentence A2 are associated to obtain a second sentence vector corresponding to sentence A2, and the second sentence vector corresponding to sentence A2 can reflect the context relationship between sentence A1 and sentence A2. The first sentence vector corresponding to sentence A3 is associated, which means that the first sentence vectors of sentence A1, sentence A2 and sentence A3 are associated to obtain a second sentence vector corresponding to sentence A3, and the second sentence vector corresponding to sentence A3 can reflect the context relationship between sentence A1 and sentence A3 and between sentence A2 and sentence A3.

[0040] In this embodiment, after obtaining the second sentence vectors corresponding to the at least two sentences respectively, the second sentence vectors corresponding to the at least two sentences respectively are searched in the video vector library to obtain video clips respectively adapted to the at least two sentences. For example, the video library includes a plurality of video clips, the image features of each shot in each video clip or the image features of each key frame in each video clip are extracted, the image features extracted for each video clip are converted into a feature vector to obtain a feature vector corresponding to the video clip, wherein the image features extracted for each video clip can include but are not limited to color features, texture features, shape features, spatial relationship features, etc.; each second sentence vector is matched with the feature vectors corresponding to the plurality of video clips included in the video vector library to obtain a feature vector adapted to each second sentence vector, and the video clip corresponding to the feature vector is taken as a video clip matched with the second sentence vector corresponding to the second sentence vector, thereby obtaining video clips respectively adapted to the at least two sentences in the long text to be searched. The length of the video clip is not limited, for example, the length of the video clip can be 2s, 3s, 5s or 6s, etc.

[0041] In this embodiment, after obtaining the video segments respectively adapted to the at least two sentences, the video information corresponding to the long text to be retrieved can be generated according to the video segments respectively adapted to the at least two sentences. For example, the video information corresponding to the long text to be retrieved is generated by splicing the video segments respectively adapted to the at least two sentences according to the time sequence relationship between the at least two sentences. For another example, the video information corresponding to the long text to be retrieved is generated by splicing the video segments respectively adapted to the at least two sentences according to the time sequence relationship between the at least two sentences and adding a transition effect at the splicing position, so that the video information corresponding to the long text to be retrieved is more natural and smooth in time sequence effect.

[0042] In the embodiments of the present application, the sentences in the long text to be retrieved are subjected to vectorization processing to obtain a first sentence vector corresponding to each sentence, and a second sentence vector reflecting the context relationship between the sentences is generated according to the time sequence relationship between the sentences in the long text to be retrieved and the first sentence vector corresponding to each sentence. The second sentence vector of each sentence in the long text to be retrieved can reflect the information of other sentences, so that the video information composed of the video segments retrieved based on the second sentence vector of the long text to be retrieved can better match the theme of the long text to be retrieved, and the efficiency and accuracy of video retrieval are improved.

[0043] In an optional embodiment, a video retrieval model can be used to retrieve the long text to be retrieved to obtain the video information corresponding to the long text to be retrieved. The implementation of the video retrieval model is not limited. A specific implementation can be found in Figure 2 In Figure 2 , the video retrieval model includes a sentence vector generation network layer, a sentence vector association network layer, a video vector generation network layer, and a video retrieval network layer.

[0044] The sentence vector generation network layer is configured to generate a first sentence vector corresponding to each sentence according to the word vectors corresponding to the words included in each sentence. Any network layer or model that can generate a first sentence vector corresponding to each sentence according to the word vectors corresponding to the words included in each sentence is applicable to the embodiments of the present application. For example, the sentence vector generation network layer can be implemented as an RNN or a word to vector (Word2vec).

[0045] In an optional embodiment, the implementation of vectorizing the at least two sentences respectively according to the words included in each sentence to obtain the first sentence vectors corresponding to the at least two sentences respectively is not limited. The following is an exemplary description.

[0046] Example X1: Obtain the word vectors corresponding to the words included in each sentence. For example, a word vector query table can be established in advance, which includes each word and the word vector corresponding to the word. First, the words included in each sentence in the to-be-retrieved long text can be determined, and the word vector corresponding to each word can be obtained based on the query table. The word vectors corresponding to the words included in each sentence are input into the sentence vector generation network layer in the video retrieval model, the word vectors corresponding to the words are synthesized by the sentence vector generation network layer, and the first sentence vector corresponding to each sentence is obtained. The sentence vector generation network layer can be implemented as an RNN, as shown in Figure 2 As shown in the figure, the RNN is used as the sentence vector generation network layer to generate the first sentence vector, which can take into account the time sequence relationship of the words in the sentence and / or the word frequency of the words, thereby improving the accuracy of the sentence vector generation.

[0047] Example X2: Obtain the word vectors corresponding to the words included in each sentence, and perform weighted averaging on the word vectors corresponding to the words included in each sentence to obtain the first sentence vector corresponding to the sentence.

[0048] In the embodiment, the implementation of associating the first sentence vector corresponding to each sentence with the first sentence vectors corresponding to other sentences to obtain the second sentence vector corresponding to the sentence according to the time sequence relationship is not limited, and the following exemplary description is made.

[0049] Example Y1: According to the time sequence relationship, the first sentence vector corresponding to each sentence is input into a sentence vector association network layer in the video retrieval model, and in the sentence vector association network layer, the first sentence vector corresponding to each sentence is associated with the second sentence vectors corresponding to other sentences to obtain the second sentence vector corresponding to each sentence. Wherein, the sentence vector association network layer can be implemented as RNN, as shown in the following formula: Figure 2 For any sentence in the long text to be retrieved, other sentences are implemented as all or part of the sentences except the any sentence.

[0050] In the case where other sentences are implemented as all sentences except the any sentence, for example, the long text B0 to be retrieved includes the sentence B1, the sentence B2, the sentence B3 and the sentence B4, in the process of calculating the second sentence vector corresponding to the sentence B3, according to the time sequence relationship, the first sentence vectors corresponding to the sentence B1, the sentence B2 and the sentence B4 are respectively associated with the first sentence vector corresponding to the sentence B3 to obtain the second sentence vector corresponding to the sentence B3.

[0051] In the case where other sentences are implemented as all sentences except the any sentence, for example, other sentences are implemented as all sentences before the any sentence, or one sentence before the any sentence, etc. If other sentences are implemented as one sentence before the any sentence, the long text B0 to be retrieved includes the sentence B1, the sentence B2, the sentence B3 and the sentence B4, in the process of calculating the second sentence vector corresponding to the sentence B3, according to the time sequence relationship, the first sentence vectors corresponding to the sentence B2 and the sentence B3 are associated to obtain the second sentence vector corresponding to the sentence B3. If other sentences are implemented as all sentences before the any sentence, the long text B0 to be retrieved includes the sentence B1, the sentence B2, the sentence B3 and the sentence B4, in the process of calculating the second sentence vector corresponding to the sentence B3, according to the time sequence relationship, the first sentence vectors corresponding to the sentence B1 and the sentence B2 are respectively associated with the first sentence vector corresponding to the sentence B3 to obtain the second sentence vector corresponding to the sentence B3.

[0052] Example Y2: Without using the sentence vector association network layer, directly associating the first sentence vector corresponding to each sentence with the first sentence vectors corresponding to other sentences to obtain the second sentence vector corresponding to the sentence, for example, the association processing can be implemented as weighted multiplication.

[0053] In the present embodiment, the implementation of retrieving the video segments respectively adapted to the at least two sentences by the second sentence vectors respectively corresponding to the at least two sentences in the video vector library is not limited, and is exemplarily described below.

[0054] Example Z1: The video vector library includes the existing video segments and the video vectors corresponding to the existing video segments. The second sentence vector corresponding to each sentence is input into the video retrieval network layer in the video retrieval model (such as the video retrieval network layer shown in FIG. 6), the similarity between each second sentence vector and the video vector in the video vector library is calculated by the video retrieval network layer, the existing video segment corresponding to the video vector satisfying the set similarity condition is taken as the video segment adapted to the sentence corresponding to the second sentence vector, and thus the video segments respectively adapted to the at least two sentences are obtained. Figure 2

[0055] The set similarity condition can be that the similarity exceeds a similarity threshold value, for example, the similarity threshold value is 80%, 90% or 95%, etc. For example, if there are multiple video vectors whose similarity with the second sentence vector exceeds the set similarity threshold value, one of the multiple video vectors can be randomly selected, and the video segment corresponding to the video vector is taken as the video segment adapted to the sentence corresponding to the second sentence vector. Alternatively, the set similarity condition can be the highest similarity. The similarity between the video vector in the video vector library and the second sentence vector is calculated, and the video vector with the highest similarity is selected, and the video segment corresponding to the video vector is taken as the video segment adapted to the sentence corresponding to the second sentence vector.

[0056] The second sentence vector and the video vector can be respectively added with weights, and the similarity between each second sentence vector and the video vector in the video vector library is calculated according to the respective added weight coefficients, which is not limited. The weights respectively added to the second sentence vector and the video vector can be obtained by training the video retrieval network layer.

[0057] Example Z2: The video vector library includes the existing video segments and the video vectors corresponding to the existing video segments. Without using the video retrieval network layer, the similarity between each second sentence vector and the video vector in the video vector library is directly calculated, for example, the cosine similarity, Euclidean distance or Manhattan distance between the second sentence vector and the video vector is calculated, and the calculated value is taken as the similarity between the second sentence vector and the video vector. The existing video segment corresponding to the video vector satisfying the set similarity condition is taken as the video segment adapted to the sentence corresponding to the second sentence vector, and thus the video segments respectively adapted to the at least two sentences are obtained. ​

[0058] In an optional embodiment, the method provided in this application further includes: generating video vectors corresponding to video segments based on video segments in a video vector library. Specifically, existing video segments are pre-selected, and the selection method for existing video segments is not limited. For example, an independent video segment corresponding to a sentence can be directly selected, and this independent video segment is used as an existing video segment. Another example is to obtain an existing long text and its corresponding existing video, and segment the existing video according to the time segment corresponding to each sentence in the existing long text to obtain multiple existing video segments. After selecting existing video segments, for any given existing video segment, multiple image frames are obtained from that existing video segment, and the multiple image frames are input into the video vector generation network layer (e.g., ...) in the video retrieval model. Figure 2 As shown, feature extraction is performed on multiple image frames to obtain feature vectors corresponding to each image frame. These feature vectors are then fused to obtain the video vector corresponding to the existing video segment. The image features in the multiple image frames can be brightness, edges, texture, and color, or they can be matrices, histograms, or principal components corresponding to the image frames. The feature vector corresponding to an image frame is formed by combining one or more features of the image frame, and this feature vector represents the image frame. If the feature vector consists of one feature, it is a one-dimensional vector; if it consists of n features, it is an n-dimensional vector, where n ≥ 2 and n is a positive integer.

[0059] This can be achieved by using methods such as max pooling, average pooling, or minimum pooling to fuse the feature vectors corresponding to multiple image frames, thus obtaining the video vector corresponding to the existing video segment. For example, if video segment C consists of 10 image frames, each image frame corresponds to a 512-dimensional feature vector, meaning each feature vector has 512 feature values, then by using max pooling, the maximum feature value from the 10 image frames is selected as the feature value of the video vector in each of the 512 dimensions, thereby obtaining the video vector corresponding to the existing video segment.

[0060] In an optional embodiment, a process of training the video retrieval model is further included, specifically, obtaining a sentence sample and a corresponding video segment sample, for example, a long text sample and a corresponding video sample can be obtained, the video sample is divided according to a time segment corresponding to each sentence sample in the long text sample to obtain a video segment sample corresponding to the sentence sample; after obtaining the sentence sample and the corresponding video segment sample, a sentence vector of the sentence sample is generated, and a video vector of the video segment sample corresponding to the sentence sample is generated as a set of training samples, wherein the implementation of generating the sentence vector of the sentence sample can refer to the aforementioned process of generating the first sentence vector and the second sentence vector, which will not be described here again, wherein the sentence vector of the sentence sample corresponds to the aforementioned second sentence vector, and the process of generating the video vector can refer to the aforementioned process, which will not be described here again; the sentence vector in the training sample is taken as the input of the video retrieval model, the video vector in the training sample is taken as the expected model output for multiple model training, and the loss function of each training is calculated, the model parameters are adjusted according to the loss function of each training, and the video retrieval model is obtained when the loss function meets the set requirement; wherein the loss function is the difference information between the actual model output and the expected model output, and the loss function can adopt a maximum margin (Max-Margin) function.

[0061] Figure 3 is a block diagram of a video retrieval device according to an exemplary embodiment. Referring to Figure 3 The device comprises an obtaining module 31, a processing module 32, an association module 33, a retrieval module 34 and a generating module 35.

[0062] The obtaining module 31 is configured to obtain a long text to be retrieved, wherein the long text to be retrieved comprises at least two sentences, and the at least two sentences have a time sequence relationship.

[0063] The processing module 32 is configured to perform vectorization processing on each sentence according to the words contained in the sentence to obtain a first sentence vector corresponding to the sentence.

[0064] The association module 33 is configured to perform association processing on the first sentence vector corresponding to each sentence and the first sentence vectors corresponding to other sentences according to the time sequence relationship to obtain a second sentence vector corresponding to the sentence, wherein the other sentences include all or part of the sentences other than the sentence in the at least two sentences.

[0065] The retrieval module 34 is configured to perform retrieval in a video vector library according to the second sentence vectors respectively corresponding to the at least two sentences to obtain video segments respectively adapted to the at least two sentences.

[0066] The generating module 35 is configured to generate video information corresponding to the long text to be searched according to the video segments respectively adapted to the at least two sentences.

[0067] In an optional embodiment, the processing module is specifically configured to: acquire word vectors corresponding to a plurality of words included in each sentence; input the word vectors corresponding to the plurality of words into a sentence vector generation network layer in the video search model, and perform synthesis processing on the plurality of word vectors through the sentence vector generation network layer to obtain a first sentence vector corresponding to each sentence.

[0068] In an optional embodiment, the association module is specifically configured to: input the first sentence vector corresponding to each sentence into a sentence vector association network layer in the video search model according to the time sequence relationship, and perform association processing on the first sentence vector corresponding to each sentence and the first sentence vectors corresponding to other sentences in the sentence vector association network layer to obtain a second sentence vector corresponding to each sentence.

[0069] In an optional embodiment, the video vector library includes existing video segments and video vectors corresponding to the existing video segments, and the searching module is specifically configured to: input the second sentence vector corresponding to each sentence into a video search network layer in the video search model, calculate similarity between each second sentence vector and the video vectors in the video vector library through the video search network layer, and take an existing video segment corresponding to a video vector satisfying a set similarity condition as a video segment adapted to the second sentence vector corresponding to the second sentence vector, so as to obtain the video segments respectively adapted to the at least two sentences.

[0070] In an optional embodiment, the video search device further includes a selection module, and the selection module is configured to preselect the existing video segments. The processing module is further configured to: for any existing video segment, acquire a plurality of image frames in the existing video segment, input the plurality of image frames into the video vector generation network layer in the video search model, perform feature extraction on the plurality of image frames to obtain feature vectors respectively corresponding to the plurality of image frames, and fuse the feature vectors respectively corresponding to the plurality of image frames to obtain a video vector corresponding to the existing video segment.

[0071] In an optional embodiment, the obtaining module is further configured to obtain sentence samples and corresponding video clip samples; the generating module is further configured to generate, for each sentence sample, a sentence vector of the sentence sample and a video vector of the video clip sample corresponding to the sentence sample as a set of training samples; the processing module is further configured to perform multiple model training by taking the sentence vector in the training sample as the input of the video retrieval model and taking the video vector in the training sample as the expected model output, and to calculate a loss function of each training, and to adjust the model parameters according to the loss function of each training until the loss function meets the set requirement to obtain the video retrieval model; wherein the loss function is the difference information between the actual model output and the expected model output.

[0072] Figure 4 A structural schematic diagram of a video retrieval device is provided for another exemplary embodiment of the present application. As shown in the figure, the device includes a memory 44 and a processor 45. Figure 4

[0073] The memory 44 is configured to store computer programs and can be configured to store various data to support operations on the computing platform. Examples of these data include instructions of any application program or method for operating on the computing platform, contact data, phonebook data, messages, pictures, videos, etc.

[0074] The memory 44 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0075] The processor 45 is coupled to the memory 44 and is configured to execute the computer programs in the memory 44 for: obtaining a long text to be retrieved, the long text to be retrieved including at least two sentences, the at least two sentences having a time sequence relationship; performing vectorization processing on each sentence according to the words contained in the sentence to obtain a first sentence vector corresponding to the sentence; performing association processing on the first sentence vector corresponding to each sentence and the first sentence vectors corresponding to other sentences according to the time sequence relationship, to obtain a second sentence vector corresponding to the sentence, the other sentences including all or part of the sentences other than the sentence among the at least two sentences; performing retrieval in a video vector library according to the second sentence vectors respectively corresponding to the at least two sentences to obtain video clips respectively adapted to the at least two sentences; and generating video information corresponding to the long text to be retrieved according to the video clips respectively adapted to the at least two sentences.

[0076] ​In an optional embodiment, the processor 45 is specifically configured to: obtain word vectors corresponding to the plurality of words included in each sentence; input the word vectors corresponding to the plurality of words into a sentence vector generation network layer in the video retrieval model, and perform synthesis processing on the word vectors corresponding to the plurality of words through the sentence vector generation network layer to obtain a first sentence vector corresponding to each sentence.

[0077] In an optional embodiment, the processor 45 is specifically configured to: input the first sentence vector corresponding to each sentence into a sentence vector association network layer in the video retrieval model according to the time sequence relationship, and perform association processing on the first sentence vector corresponding to each sentence and the first sentence vectors corresponding to other sentences in the sentence vector association network layer to obtain a second sentence vector corresponding to each sentence.

[0078] In an optional embodiment, the video vector library includes existing video clips and corresponding video vectors, and the processor 45 is specifically configured to: input the second sentence vector corresponding to each sentence into a video retrieval network layer in the video retrieval model, calculate the similarity between each second sentence vector and the video vectors in the video vector library through the video retrieval network layer, and take the existing video clip corresponding to the video vector that meets the set similarity condition as the video clip adapted to the sentence corresponding to the second sentence vector, thereby obtaining the video clip adapted to each of the at least two sentences.

[0079] In an optional embodiment, the processor 45 is further configured to: preselect an existing video clip; for any existing video clip, obtain a plurality of image frames in the existing video clip, input the plurality of image frames into a video vector generation network layer in the video retrieval model, perform feature extraction on the plurality of image frames to obtain feature vectors corresponding to the plurality of image frames respectively, and fuse the feature vectors corresponding to the plurality of image frames respectively to obtain a video vector corresponding to the existing video clip.

[0080] In an optional embodiment, the processor 45 is further configured to: obtain a sentence sample and a corresponding video clip sample;

[0081] For each sentence sample, a sentence vector of the sentence sample is generated, and a video vector of a video segment sample corresponding to the sentence sample is generated, as a set of training samples; the sentence vector in the training sample is taken as an input of the video retrieval model, the video vector in the training sample is taken as an expected model output for multiple model training, and a loss function of each training is calculated, according to the loss function of each training, the model parameters are adjusted, until the loss function meets the set requirement, the video retrieval model is obtained; wherein the loss function is difference information between an actual model output and an expected model output.

[0082] Further, as shown in Figure 4 , the video retrieval device further includes a communication component 46, a display 47, a power supply component 48, an audio component 49, and other components. Figure 4 Some components are only schematically shown in the Figure 4 , and it does not mean that the video retrieval device only includes the components shown in the Figure 4 . It should be noted that the components in the dashed box in the are optional components, not mandatory components, and the specific components can be determined according to the product form of the video retrieval device.

[0083] Correspondingly, the embodiment of the present application also provides a computer readable storage medium storing a computer program, when the computer program is executed by a processor, the processor can implement each step in the method shown in Figure 1 .

[0084] Correspondingly, the embodiment of the present application also provides a computer program product storing a computer program, when the computer program is executed by a processor, the processor can implement each step in the method shown in Figure 1 .

[0085] The communication component in the above Figure 4 is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G, or a combination thereof. In an example embodiment, the communication component receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra wide band (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0086] The communication component in the above Figure 4The display in the electronic device includes a screen, which can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding movement, but also detect duration and pressure related to the touching or sliding movement.

[0087] The above Figure 4 The power component in the electronic device provides power to various components of the device in which the power component is located. The power component can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the device in which the power component is located.

[0088] The above Figure 4 The audio component in the electronic device can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) that is configured to receive an external audio signal when the device in which the audio component is located is in an operation mode such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in a memory or transmitted via the communication component. In some embodiments, the audio component also includes a speaker to output audio signals.

[0089] Those skilled in the art understand that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer usable program code.

[0090] The present application is described with reference to the flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the flow Figure 1 The apparatus for performing the function specified in the flow or flows and / or blocks. Figure 1 The apparatus for performing the function specified in the flow or flows and / or blocks.

[0091] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0093] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0094] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory, etc. The memory is an example of computer readable media.

[0095] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0096] It should also be noted that the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0097] The above description is merely illustrative of the application, and not restrictive. Various modifications and changes can become apparent to those skilled in the art. Incorporating any modification, equivalent substitution, improvement, etc. within the spirit and principle of the application, shall be included in the scope of the claims of the application.

Claims

1. A video retrieval method, characterized in that, include: Obtain the long text to be retrieved, wherein the long text to be retrieved includes at least two sentences and there is a temporal relationship between the at least two sentences; Based on the words contained in each sentence, each sentence is vectorized to obtain the first sentence vector corresponding to each sentence; Based on the temporal relationship, the first sentence vector corresponding to each sentence is associated with the first sentence vectors corresponding to other sentences to obtain the second sentence vector corresponding to that sentence. The other sentences include all or part of the sentences other than that sentence in the at least two sentences. Based on the second sentence vectors corresponding to the at least two sentences respectively, a search is performed in the video vector library to obtain the video segments that each of the at least two sentences is adapted to. Based on the video segments adapted to each of the at least two sentences, video information corresponding to the long text to be retrieved is generated.

2. The method according to claim 1, characterized in that, Based on the words contained in each sentence, the at least two sentences are vectorized to obtain the first sentence vector corresponding to each of the at least two sentences, including: Obtain the word vectors corresponding to the multiple words included in each sentence; The word vectors corresponding to the multiple words are input into the sentence vector generation network layer in the video retrieval model. The sentence vector generation network layer synthesizes the word vectors corresponding to the multiple words to obtain the first sentence vector corresponding to each sentence.

3. The method according to claim 1, characterized in that, Based on the temporal relationship, the first sentence vector corresponding to each sentence is associated with the first sentence vectors corresponding to other sentences to obtain the second sentence vector corresponding to that sentence, including: According to the temporal relationship, the first sentence vector corresponding to each sentence is input into the sentence vector association network layer in the video retrieval model. In the sentence vector association network layer, the first sentence vector corresponding to each sentence is associated with the first sentence vectors corresponding to other sentences to obtain the second sentence vector corresponding to each sentence.

4. The method according to claim 1, characterized in that, The video vector library includes existing video segments and their corresponding video vectors. Based on the second sentence vectors corresponding to the at least two sentences, a search is performed in the video vector library to obtain the video segments suitable for each of the at least two sentences, including: The second sentence vector corresponding to each sentence is input into the video retrieval network layer in the video retrieval model. The similarity between each second sentence vector and the video vector in the video vector library is calculated by the video retrieval network layer. The existing video segments corresponding to the video vectors that meet the set similarity conditions are used as video segments that are adapted to the sentences corresponding to the second sentence vectors, thereby obtaining the video segments adapted to each of the at least two sentences.

5. The method according to claim 4, characterized in that, The method further includes: Pre-select existing video clips; For any existing video segment, multiple image frames are obtained from the existing video segment. The multiple image frames are input into the video vector generation network layer in the video retrieval model. Feature extraction is performed on the multiple image frames to obtain feature vectors corresponding to the multiple image frames respectively. The feature vectors corresponding to the multiple image frames are fused to obtain the video vector corresponding to the existing video segment.

6. The method according to any one of claims 2-5, characterized in that, Also includes: Obtain sentence samples and their corresponding video clip samples; For each sentence sample, generate the sentence vector for that sentence sample, and generate the video vector for the corresponding video segment sample, as a set of training samples; The sentence vectors in the training samples are used as input to the video retrieval model, and the video vectors in the training samples are used as the expected model output for multiple model trainings. The loss function is calculated for each training, and the model parameters are adjusted according to the loss function for each training until the loss function meets the set requirements, thus obtaining the video retrieval model. The loss function is the difference information between the actual model output and the expected model output.

7. A video retrieval device, characterized in that, include: The module includes an acquisition module, a processing module, an association module, a retrieval module, and a generation module. The acquisition module is used to acquire the long text to be retrieved, which includes at least two sentences and there is a temporal relationship between the at least two sentences; The processing module is used to vectorize each sentence according to the words contained in each sentence to obtain the first sentence vector corresponding to each sentence; The association module is used to associate the first sentence vector corresponding to each sentence with the first sentence vectors corresponding to other sentences according to the temporal relationship, so as to obtain the second sentence vector corresponding to the sentence. The other sentences include all or part of the sentences other than the sentence in the at least two sentences. The retrieval module is used to perform retrieval in the video vector library according to the second sentence vectors corresponding to the at least two sentences respectively, and obtain the video segments that each of the at least two sentences is adapted to. The generation module is used to generate video information corresponding to the long text to be retrieved based on the video segments adapted to each of the at least two sentences.

8. The apparatus according to claim 7, characterized in that, The processing module is specifically used for: Obtain the word vectors corresponding to the multiple words included in each sentence; The word vectors corresponding to the multiple words are input into the sentence vector generation network layer in the video retrieval model. The sentence vector generation network layer synthesizes the word vectors corresponding to the multiple words to obtain the first sentence vector corresponding to each sentence.

9. The apparatus according to claim 8, characterized in that, The association module is specifically used for: According to the temporal relationship, the first sentence vector corresponding to each sentence is input into the sentence vector association network layer in the video retrieval model. In the sentence vector association network layer, the first sentence vector corresponding to each sentence is associated with the first sentence vectors corresponding to other sentences to obtain the second sentence vector corresponding to each sentence.

10. The apparatus according to claim 9, characterized in that, The video vector library includes existing video clips and their corresponding video vectors. The retrieval module is specifically used for: The second sentence vector corresponding to each sentence is input into the video retrieval network layer in the video retrieval model. The similarity between each second sentence vector and the video vector in the video vector library is calculated by the video retrieval network layer. The existing video segments corresponding to the video vectors that meet the set similarity conditions are used as video segments that are adapted to the sentences corresponding to the second sentence vectors, thereby obtaining the video segments adapted to each of the at least two sentences.

11. The apparatus according to claim 10, characterized in that, The video retrieval device further includes: a selection module; The selection module is used to pre-select existing video segments; The processing module is further configured to: for any existing video segment, acquire multiple image frames from the existing video segment, input the multiple image frames into the video vector generation network layer in the video retrieval model, extract features from the multiple image frames to obtain feature vectors corresponding to the multiple image frames respectively, and fuse the feature vectors corresponding to the multiple image frames respectively to obtain the video vector corresponding to the existing video segment.

12. The apparatus according to any one of claims 8-11, characterized in that, The acquisition module is also used to: acquire sentence samples and their corresponding video segment samples; The generation module is also used to: generate a sentence vector for each sentence sample and generate a video vector for the corresponding video segment sample, as a set of training samples; The processing module is further configured to: use sentence vectors in the training samples as input to the video retrieval model, use video vectors in the training samples as the expected model output for multiple model trainings, calculate the loss function for each training, and adjust the model parameters according to the loss function for each training until the loss function meets the set requirements, thereby obtaining the video retrieval model; wherein, the loss function is the difference information between the actual model output and the expected model output.

13. A video retrieval device, characterized in that, include: Memory and processor; The memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps of the method according to any one of claims 1-6.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method according to any one of claims 1-6.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method according to any one of claims 1-6.

Images