Video retrieval method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of machine learning, and discloses a video retrieval method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring video data and text data related to the video data; encoding video data by using a video encoder to obtain video encoded data, and encoding text data by using a first text encoder and a second text encoder to obtain a first text code and a second text code; according to the first text code, the second text code and the video code data, performing calculation by using a multi-modal model to obtain video clip correlation value data; obtaining a plurality of saliency vectors, and calculating to obtain a saliency mark according to the plurality of saliency vectors, the video clip correlation value data and the video coding data; and inputting the saliency mark and the video data into a retrieval model to obtain target video data. A multi-modal model technology is applied to video clip retrieval, so that a user can retrieve video clips conforming to text semantics, and creation is more efficient and convenient.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a video retrieval method, apparatus, computer equipment and storage medium. Background Art

[0002] With the rapid rise of online media technologies such as live broadcast promotion, digital humans, and short videos, from scientific and technological enterprises in the fields of smart healthcare and smart cities to financial institutions such as banks, insurance companies, and securities companies, there is a growing interest in using these new technologies for product promotion, customer marketing, public relations maintenance, and other activities. However, relying on traditional video production models to produce media data in the form of product explanations, video explanations, and promotional videos often requires retrieving a large amount of material for editing, and the accuracy and convenience of searching by human memory or file names are often limited, which makes it impossible for creators to efficiently and conveniently use the large amount of video materials accumulated in the past, especially the fragments in some long videos cannot be effectively used, resulting in problems such as slow video production, high creation threshold, and low material utilization efficiency. Summary of the invention

[0003] The present application discloses a video retrieval method, apparatus, computer equipment and storage medium, which solve the problem of limited accuracy and convenience in users' retrieval of video clip data.

[0004] In a first aspect, the present application provides a video retrieval method, comprising:

[0005] Acquire video data and text data related to the video data;

[0006] Encoding the video data using a video encoder to obtain video encoded data, and encoding the text data using a first text encoder and a second text encoder to obtain a first text code and a second text code;

[0007] Obtaining video segment related value data by calculating using a multimodal model according to the first text code, the second text code and the video code data;

[0008] Acquire a plurality of saliency vectors, and calculate a saliency mark according to the plurality of saliency vectors, the video segment related value data and the video encoding data;

[0009] The saliency mark and the video data are input into a retrieval model to obtain target video data.

[0010] In a second aspect, the present application provides a model training device, comprising:

[0011] A data information acquisition module, used to acquire video data and text data related to the video data;

[0012] A data feature extraction module, used to encode the video data using a video encoder to obtain video encoding data, and to encode the text data using a first text encoder and a second text encoder to obtain a first text encoding and a second text encoding;

[0013] A data feature interaction module, configured to calculate and obtain video segment related value data using a multimodal model according to the first text code, the second text code and the video code data;

[0014] A significant mark calculation module is used to obtain a plurality of significant vectors, and calculate a significant mark according to the plurality of significant vectors, the video segment related value data and the video encoding data;

[0015] The target data retrieval module is used to input the significant mark and the video data into a retrieval model to obtain target video data.

[0016] In a third aspect, the present application provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, a video retrieval method provided in any embodiment of the present application is implemented.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the processor implements a video retrieval method as provided in any embodiment of the present application.

[0018] The above-mentioned video retrieval method, device, computer equipment and storage medium obtain video data and text data related to the video data, encode the text data to obtain a first text code and a second text code, and encode the video data to obtain video code data; according to the first text code, the second text code and the video code data, use a multimodal model to calculate the attention weight; according to the attention weight and the video code data, calculate the video clip related value data; obtain multiple salient vectors, and calculate the salient mark according to the video clip related value data and the video code data; input the salient mark and the video data into the retrieval model to obtain the target video data. Through the application of multimodal technologies such as feature extraction and feature interaction, users can search for the required video through text descriptions, so as to carry out creation and editing work more efficiently and conveniently.

[0019] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a schematic flow chart of the steps of a video retrieval method provided by an embodiment of the present application;

[0022] Figure 2 This is a schematic flow chart of the steps of a data feature extraction method provided by an embodiment of the present application;

[0023] Figure 3 is a schematic block diagram of a neural network model provided in one embodiment of the present application;

[0024] Figure 4 is a schematic flow chart of steps of a method for calculating a significant mark provided in one embodiment of the present application;

[0025] Figure 5 is a schematic block diagram of a neural network model provided in one embodiment of the present application;

[0026] Figure 6 is a schematic flow chart of the steps of a video retrieval method provided by an embodiment of the present application;

[0027] Figure 7 It is a schematic flow chart of the steps of a model training method provided by an embodiment of the present application;

[0028] Figure 8 is a schematic block diagram of a video retrieval device provided in one embodiment of the present application;

[0029] Fig. 9 It is a schematic block diagram of the structure of a computer device provided in one embodiment of the present application.

[0030] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0033] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0034] It should be understood that, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "third" are used to distinguish between identical or similar items with substantially identical functions and effects. For example, the first data and the third data are only used to distinguish different data, and their sequence is not limited. Those skilled in the art can understand that words such as "first" and "third" do not limit the quantity and execution order, and words such as "first" and "third" do not necessarily limit them to be different.

[0035] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0036] To facilitate understanding of the embodiments of the present application, some words involved in the embodiments of the present application are briefly explained below.

[0037] 1. BERT (Bidirectional Encoder Representations from Transformers): A pre-trained language representation model based on the Transformer architecture. Its core feature is that it can capture the bidirectional contextual information of the text, so it performs well in various natural language processing (NLP) tasks. Its input representation often includes Token Embeddings that convert words in the text into word embedding vectors, Segment Embeddings that distinguish the two sentences in a sentence pair and are usually used for question-answering tasks, and Position Embeddings that provide the model with information about the position of words in the sentence.

[0038] 2. Transformer: The Transformer model consists of an encoder and a decoder, both of which contain multiple identical layers. Each layer consists of a self-attention sublayer and a feed-forward sublayer, with residual connections and layer normalization between these sublayers.

[0039] 3. Token Embedding Layer: In the Transformer model, the token embedding layer is the first step in processing the input data. This layer converts the words in the text into vectors of fixed dimensions to capture the semantic information of the words.

[0040] 4. Positional Encoding Layer: An important component in Transformer, it provides the model with information about the position of words in the sequence. This is because the self-attention mechanism of Transformer itself does not contain information about the order of words. Positional encoding is usually implemented by adding a vector of the same dimension as the word embedding, which contains information about the position of the word.

[0041] 5. Multi-Head Attention Layer: One of the components of Transformer, used to learn information in different representation subspaces in parallel, allowing the model to focus on different parts of the input sequence at the same time, thereby better capturing complex dependencies.

[0042] 6. Self-Attention Layer: One of the components of Transformer, it allows the model to capture the dependency between any two positions within the sequence when processing sequence data. It is widely used in the fields of natural language processing (NLP) and computer vision (CV).

[0043] 7. Residual Layer: A key component in Transformer, it helps the model to effectively transfer information in deep networks by adding a skip connection, which helps solve the gradient disappearance problem in deep networks.

[0044] 8. Batch Normalization: In machine learning, it refers to a technique used to improve the speed and stability of model training. It is usually located after each sublayer in the model, and reduces internal covariate shift by normalizing the input of the module to prevent changes in the input distribution of the network layer from adversely affecting the training process.

[0045] 9.Softmax: A commonly used activation function, the Softmax function is used to convert the raw scores of the output layer into probability distributions that represent the model's prediction confidence for each possible output.

[0046] 10. Activation layer: An important part of the neural network. It introduces nonlinear characteristics so that the network can learn and perform more complex tasks. You can use activation functions such as ReLU, Sigmoid, Leaky ReLU, Softmax, etc. to introduce nonlinear factors, thereby enhancing the expression ability of deep neural networks. Among them, ReLU is a non-saturated activation function that can alleviate the problem of gradient disappearance caused by the large number of layers in deep neural networks and accelerate the convergence speed.

[0047] 11. ViT (Vision Transformer) model: A deep learning model based on the Transformer architecture for processing visual tasks. Its core idea is to extend the Transformer model in natural language processing (NLP) to the field of computer vision and use the attention mechanism to capture features and patterns in images.

[0048] 12.ViViT: A ViT model that introduces local inductive bias in video Transformers, including a video encoder that can convert video data into video coded data.

[0049] 13. Video Swin Transformer: A ViT model that improves the quality of video representation by combining large-scale pre-trained visual prior knowledge and video-level temporal bias.

[0050] 14.MVit: A multi-scale visual ViT model for video and image recognition.

[0051] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0052] From scientific and technological enterprises in the fields of smart healthcare and smart cities to financial institutions such as banks, insurance companies, and securities companies, there is a growing interest in using online media technologies such as virtual live broadcasts, digital humans, and short videos for product promotion, customer marketing, and public relations maintenance. However, relying on traditional video production models to produce media data in the form of product explanations, video explanations, and promotional videos often requires retrieving a large amount of material for editing, and the accuracy and convenience of searching by human memory or file names are often limited, which makes it impossible for creators to efficiently and conveniently use the large amount of video materials accumulated in the past, especially the fragments in some long videos cannot be effectively used, resulting in problems such as slow video production, high threshold for creation, and low efficiency of material utilization. At the same time, it also makes it difficult for customers to search and learn about product knowledge on their own, and they cannot directly locate the key paragraphs in the product introduction video that meet their needs.

[0053] In actual business, insurance and other financial institutions will shoot introduction or marketing videos for their own financial products. However, insurance and other financial products often have many clauses and complex content, which generally require long material shooting or splitting into multiple episodes after shooting. However, even if it is split into multiple episodes, in order to ensure professionalism, compliance and completeness, the description and explanation of each episode is relatively lengthy. Because creators, editors and investors often have insufficient understanding of professional terms, or the questions are more specific, it is difficult to find the part that they are most interested in and need to know the most based on the video name and its summary, and accurately find the related parts, which causes inconvenience and difficulty in creation, learning and understanding.

[0054] Machine learning technology is being widely used in the fields of information retrieval and data search. Among them, multimodal model technology has been widely used in the fields of autonomous driving, medical image analysis, computer-aided design and creation, and has great potential to continuously empower video retrieval and artistic creation. It has the opportunity to enable users to search for the required videos through text descriptions, thereby more conveniently carrying out creation, editing and investor education.

[0055] In addition, although the existing multimodal model-based video retrieval technologies have their own advantages, they are still mainly in the academic and laboratory stage. Therefore, limited by the laboratory's computing environment and data acquisition capabilities, the existing multimodal model-based video retrieval technologies are often trained based on standardized data sets for specific scenes such as movies, housework, and driving. They have limited generalization capabilities and there are problems with inappropriate use of technologies such as model distillation. Although they adapt to the limited computing environment in the laboratory, they affect the accuracy, generalization ability, flexibility and robustness of the model. Therefore, although the existing technology has achieved good results on specific data sets, it cannot well meet the specific needs of financial scenarios such as product marketing and customer consulting in terms of data processing, model architecture and training process, such as the need for professionalism, rigor and completeness of retrieval results and the need for accuracy, robustness and security of commercial artificial intelligence.

[0056] To solve the above problems, the present application embodiment proposes a video retrieval method. Figure 1 , Figure 1 It is a schematic flowchart of the steps of a video retrieval method provided in an embodiment of the present application.

[0057] like Figure 1 As shown, the video retrieval method specifically includes steps S101 to S105.

[0058] S101: Acquire video data and text data related to the video data.

[0059] Exemplarily, the video data may include one or more of financial video data, product video data, music video data, health video data, promotional video data, introduction video data, educational video data, documentary video data, film and television video data, and short video data.

[0060] Furthermore, the text data may include one or more of the text data related to the video data, such as query information, description information and subtitle information of the video data, for example, question information on insurance laws and regulations, question information on insurance product terms and consulting information on the current insurance asset management portfolio.

[0061] S102: Encode the video data using a video encoder to obtain video encoding data, and encode the text data using a first text encoder and a second text encoder to obtain a first text encoding and a second text encoding.

[0062] Exemplarily, the video data may be encoded using a video encoder having a model with video encoding function, such as ViViT, MViT, or Video Swin Transformer.

[0063] It can be understood that the video encoding data may include multiple video feature expression vectors, and the video feature expression vectors may correspond one-to-one to each frame in the video data.

[0064] The video data or video encoding data may also include annotated data obtained by annotating the corresponding text data, so as to facilitate the subsequent training and improvement of the model. The video data, video encoding data or annotated data may also include time information, contextual information and event information.

[0065] Exemplarily, the video data can be frame extracted to obtain multiple video frames; the multiple video frames can be input into a video encoder to obtain video feature expression vectors corresponding to the video frames; the video feature expression vectors can be labeled according to the correlation between the corresponding video frames and the text data to obtain video encoding data, wherein the video encoding data includes multiple video feature expression vectors and corresponding labeled data.

[0066] It should be noted that the first text encoding may include multiple first text feature expression vectors, and the second text encoding may include multiple second text feature expression vectors, wherein the first text feature expression vectors and the second text feature expression vectors may correspond one-to-one, and the first text feature expression vectors may correspond one-to-one to each sentence in the text data.

[0067] In some embodiments, see Figure 2 , Figure 2 1 is a schematic flow chart of the steps of a data feature extraction method provided in an embodiment of the present application. The data feature extraction method can be used to implement the above step S102.

[0068] like Figure 2 As shown, the data feature extraction method specifically includes steps S102a to S102c.

[0069] It is understandable that steps S102a to S102c may be implemented using a multimodal model, which may include a first text encoder, a second text encoder, and a video encoder.

[0070] S102a, inputting text data into a first text encoder to obtain a first text code.

[0071] Exemplarily, the text data can be encoded using an encoder of a model with a text encoding function, such as BERT, T5, CLIP, or GPT, to obtain a first text encoding.

[0072] S102b, input the first text code into the second text encoder to obtain a feature text code.

[0073] Among them, the second text encoder may include one or more self-attention modules.

[0074] See also Figure 3 , Figure 3 It is a module schematic block diagram of a neural network model provided in an embodiment of the present application. The neural network model can be used as a second text encoder to implement the above step S102b.

[0075] like Figure 3 As shown, the neural network model may include multiple self-attention modules, wherein each self-attention module includes a normalization layer, a self-attention layer, a residual processing layer and a feedforward network layer.

[0076] Among them, the self-attention layer can be a multi-head self-attention layer.

[0077] Exemplarily, for a self-attention module, the input vector of the self-attention module can be normalized to obtain an intermediate vector A; the intermediate vector A is copied three times to obtain a Q vector, a K vector, and a V vector; the multi-head self-attention algorithm is used to calculate the intermediate vector B based on the Q vector, the K vector, and the V vector; the intermediate vector C is obtained by calculating the average value based on the intermediate vector B and the intermediate vector A; the intermediate vector C is input into the feedforward neural network to obtain an intermediate vector D; the intermediate vector D is added to the intermediate vector B and normalized to obtain an intermediate vector E.

[0078] The normalization method may be ReLU and / or SoftMax, and the feedforward neural network may be fully connected.

[0079] It should be noted that if the neural network model includes multiple self-attention modules, the intermediate vector E is used as the input vector of the next self-attention module and input into the next self-attention module; if the neural network model only includes one self-attention module or the self-attention module is the last self-attention module of the neural network model, the intermediate vector E is the feature text encoding.

[0080] It should be noted that before the first text code is input into the second text encoder, it may include obtaining an initial virtual vector; concatenating the initial virtual vector with the first text code, and then inputting the vector into the first self-attention module of the second text encoder to obtain a virtual vector.

[0081] The method for obtaining the initial virtual vector includes initializing the value of the initial virtual vector according to Gaussian distribution. The dimensions of the initial virtual vector and the virtual vector may be consistent with the dimensions of the first text code.

[0082] It is understandable that traditional methods often do not take into account the different importance of different words in text data. For example, in the fields of finance and insurance, modal particles and auxiliary words are not very helpful in determining semantics and video retrieval, while specific nouns, verbs and adverbs are more helpful in determining semantics and video retrieval. The second text encoder used in this application takes advantage of the attention mechanism, so that the feature text encoding can focus more on key words in the text data, capture richer semantic and contextual information, and improve the generalization ability and accuracy of the video retrieval method of this application.

[0083] Furthermore, the neural network model can include three self-attention modules. An appropriate number of self-attention modules also avoids the waste of computing resources and overfitting problems caused by stacking too many self-attention modules.

[0084] S102c, concatenating the first text code and the characteristic text code to obtain a second text code.

[0085] Specifically, the method of concatenating the first text code with the feature text code may include simple concatenation and weighted summation.

[0086] It should be noted that the concatenated vector can contain the original text features and the high-level features extracted by the self-attention mechanism, thereby providing a more comprehensive text representation, helping the model to capture more subtle language features and contextual relationships, and thus improving the performance of the model in video retrieval tasks.

[0087] It should be noted that, in some embodiments, the first text code and the characteristic text code may not be concatenated, and the characteristic text code may be directly used as the second text code to obtain the second text code.

[0088] Exemplarily, video data may be input into a video encoder to obtain video encoding data; text data may be input into a first text encoder to obtain a first text encoding; the first text encoding may be input into a second text encoder to obtain a characteristic text encoding. The characteristic text encoding may be used as the second text encoding to obtain the second text encoding.

[0089] S103: Calculate and obtain video segment related value data using a multimodal model according to the first text code, the second text code and the video code data.

[0090] Exemplarily, the multimodal model may include one or more of a first text encoder, a second text encoder, and a video encoder. The first text encoder, the second text encoder, or the video encoder may be part of the multimodal model or may be used independently.

[0091] Exemplarily, the multimodal model may also include three linear mapping layers of pQ, pK and pV, and a self-attention layer. The multimodal model may also include one or more self-attention layers, wherein each self-attention layer may include three linear mapping layers of pQ, pK and pV.

[0092] It should be noted that the self-attention layer may include a multi-head self-attention layer and / or an adaptive attention layer.

[0093] Exemplarily, the first text code can be input into the linear mapping layer pV to obtain V distribution; the second text code can be input into the linear mapping layer pK to obtain K distribution; the video encoding data can be input into the linear mapping layer pQ to obtain Q distribution; the Q distribution, K distribution and V distribution can be input into the adaptive attention layer to calculate the attention weight W; based on the attention weight W, the video clip related value data can be calculated.

[0094] Among them, Q distribution, K distribution and V distribution are input into the adaptive attention layer to calculate the attention weight W. The corresponding calculation formula may include:

[0095]

[0096] Among them, L q It can represent the number of codes or word tokens contained in the first text code, L d It can represent the number of codes or virtual vectors included in the feature text encoding, L q +L d Then can represent the number of codes included in the second text encoding, ⊙ can represent the dot product, h can represent the number of latent vector dimensions of the linear mapping layer, and W i，j It can represent the attention weight corresponding to the i-th video frame and the j-th first text encoding, V j It can represent the vector in the V distribution corresponding to the jth code in the first text code, Q i It can represent the vector in the Q distribution corresponding to the i-th video frame in the video coding data, K j It can represent the vector in the K distribution corresponding to the jth code in the first text code, K k It can represent the vector in the K distribution corresponding to the k-th code of the feature text code.

[0097] Furthermore, we can use the calculation formula According to the attention weight W, the video clip related value data is calculated

[0098] Exemplarily, the video segment related value data corresponding to the i-th video frame is

[0099] S104: Obtain multiple saliency vectors, and calculate saliency labels according to the video segment related value data and the video encoding data.

[0100] It is understandable that the saliency mark can be used to express the degree of relevance between each video frame and the corresponding text in the video encoding data.

[0101] Exemplarily, obtaining a plurality of salient vectors P and calculating the salient mark according to the video segment related value data and the video coding data may include: obtaining a plurality of salient vectors P and calculating the salient mark according to the video feature expression vector v and the mean vector V corresponding to each video frame in the video coding data. ctx , multiple salient vectors P and video clip related value data Calculate the correlation matrix; use the Softmax or ReLU algorithm to process the correlation matrix and calculate the candidate weight vector C; sort the values ​​of each dimension in the candidate weight vector C to determine the set of the largest top K values ​​{C (1) , C (2) , ..., C (K)}; According to the mean vector V ctx , each value in the set and its corresponding salient vector P are calculated to obtain the salient mark T. The corresponding calculation formula may include:

[0102]

[0103] Among them, T can represent a significant mark, L v It can represent the total number of video frames or video feature expression vectors corresponding to each video frame in the video coding data, L p It can represent the total number of significant vectors, P j The above L can be expressed as p The jth salient vector among the salient vectors, C j It can be expressed as. The mean vector V ctx It can be calculated by arithmetic mean, geometric mean or weighted mean based on the video feature expression vector v corresponding to each video frame in the video coding data. The number of dimensions of the candidate weight vector can be consistent with the number of the above-mentioned multiple significant vectors P. K is a positive integer, and K is less than L p .

[0104] Furthermore, the mean vector V ctx The arithmetic mean, geometric mean or weighted mean applied in the calculation of can be the average value calculated for the value of each video feature expression vector in each dimension, so the mean vector V ctx The number of dimensions can be consistent with that of the video feature expression vector.

[0105] It should be noted that when obtaining multiple significant vectors, the number of significant vectors L p It can be between 5 and 15, and the dimension of the saliency vector can be consistent with the number of dimensions of the video feature expression vector v.

[0106] See also Figure 4 , Figure 4 is a schematic flow chart of the steps of a significant mark calculation method provided in an embodiment of the present application, which can be used to implement the above step S104. Figure 4 As shown, the significant mark calculation method specifically includes steps S104a to S104d.

[0107] S104a, calculating a correlation matrix according to the video coding data and the plurality of salient vectors.

[0108] Exemplarily, the mean vector V is calculated based on the video feature expression vector v corresponding to each video frame in the video encoding data. ctx ; Based on multiple salient vectors P and video clip related value data and the mean vector V ctx , and the correlation matrix is ​​calculated.

[0109] S104b, calculating and obtaining a candidate weight vector according to the video segment correlation value data and the correlation matrix.

[0110] Exemplarily, the correlation matrix is ​​processed using the Softmax function, and the processing result is compared with the video clip correlation value data. The transpose of is used for matrix multiplication (Multiplication), and then the matrix multiplication results are summed by column to obtain the candidate weight vector.

[0111] S104c, determining and calculating an instantaneous description vector according to the candidate weight vector and the plurality of salient vectors.

[0112] Exemplarily, the dimension with the highest value in the candidate weight vector is obtained, and its value is used as a one-dimensional vector, and the inner product is calculated with the significant vector corresponding to the ordinal number of the corresponding dimension to obtain the instantaneous description vector.

[0113] The dimension with the highest value in the candidate weight vector can also be obtained by changing to multiple dimensions with the highest values. If multiple dimensions with the highest values ​​in the candidate weight vector are obtained, the values ​​of the multiple dimensions are respectively used as a matrix with 1 row and 1 column to calculate the inner product with the significant vector corresponding to the ordinal number of the corresponding dimension, and multiple instantaneous description vectors are obtained respectively.

[0114] S104d. Calculate and obtain one or more significant tags according to the instantaneous description vector.

[0115] For example, the instantaneous description vector and the mean vector V ctx Sum them up and get the significant mark.

[0116] Exemplarily, multiple instantaneous description vectors are respectively combined with the mean vector V ctx Add together to get multiple significant marks.

[0117] S105: Input the saliency mark and the video data into the retrieval model to obtain the target video data.

[0118] Exemplarily, the saliency marker is input into a multi-layer perception layer to obtain a mapping vector; the mapping vector is concatenated with the video data to obtain an input vector for inputting into a retrieval model; the input vector is input into the retrieval model to obtain one or more time period data; based on the one or more time period data, the video data is processed to obtain target video data.

[0119] Processing the video data according to the one or more time period data to obtain the target video data may include using a Hungarian matching algorithm to segment the video data according to the one or more time period data, and evaluating and screening to obtain the target video data.

[0120] It is understandable that the output of the retrieval model can be target video data or time period data. For computer devices such as edge computing devices, mobile communication devices, and server devices, the actual performance of the neural network model will be different due to hardware computing power, system design, and other aspects. Therefore, according to the needs and actual conditions, the time period data can be obtained on one device, and then the video data can be processed by another device according to the time period data to obtain the target video data. It can make any video retrieval method provided in the embodiment of the present application more efficient and accurate.

[0121] See also Figure 5 , Figure 5 is a schematic block diagram of another neural network model provided in an embodiment of the present application. The neural network model can be used as a retrieval model to implement the above step S105. Figure 5 As shown, the retrieval model may include at least a multi-layer perceptron (MLP), a fully connected layer, an encoder module and a decoder module.

[0122] Among them, the encoder module and the decoder module can be the encoder module and the decoder module of the Transformer architecture such as Transformer, DETR or BERT.

[0123] Furthermore, the multi-layer perception layer can use activation functions including Sigmoid, Tanh, ReLU, etc.

[0124] Furthermore, the self-attention layer may include a multi-head self-attention layer, a cross-attention layer and / or an adaptive self-attention layer.

[0125] Furthermore, the feed-forward network layer can be fully connected.

[0126] Furthermore, the retrieval model may also include a position encoder and / or a residual processing layer.

[0127] It can be understood that by adding a position encoder, the computer device can learn deep grammatical and semantic features and capture long-distance dependencies based on the position information of words in sentences or video frames in video data, thereby improving the model's understanding and generation capabilities, and further improving the accuracy and robustness of the video retrieval method of the embodiment of the present application.

[0128] Exemplarily, the saliency mark is input into the perception layer to obtain a mapping vector; the mapping vector is concatenated with the video coding data to obtain an encoder input vector; the input vector is input into the encoder module to obtain a position coding vector and a feature coding vector; multiple auxiliary vectors are obtained and added to the position coding vectors respectively to obtain a decoder input vector; the encoder input vector is copied to obtain vector K and vector V; the decoder input vector is used as vector Q, and is input into the decoder module together with vector K and vector V to obtain time period data or target video data.

[0129] The position coding vector may be obtained by performing position coding on the input vector using a position encoder in the encoder module, and the position coding method may include absolute position coding or relative position coding.

[0130] The dimensions of the multiple auxiliary vectors are consistent with the position encoding, the initial multiple auxiliary vectors may all be zero vectors, and the number of auxiliary vectors may be between 10 and 15.

[0131] By using multiple auxiliary vectors to represent the key content in the video, it is possible to prevent overlap and duplication of the target video data retrieved, and to reasonably allocate the model's attention. The number of auxiliary vectors is between 10 and 15, which can semantically extract multiple key contents in the video data without increasing the model's inference computing power consumption too much, thereby improving the versatility of the model and the comprehensiveness of the retrieval method, and preventing missed detection and false detection.

[0132] In some embodiments, see Figure 6 , Figure 6 FIG. 1 is a schematic flow chart of another video retrieval method provided in an embodiment of the present application. The video retrieval method can be used to implement the above step S105. Figure 6As shown, the video retrieval method specifically includes steps S105a to S105e.

[0133] S105a, inputting the saliency mark into the multi-layer perception layer, and concatenating the output of the multi-layer perception layer with the video data to obtain an encoder input vector.

[0134] Exemplarily, the multilayer perceptron (MLP module or MLP layer for short) may include an activation layer and a fully connected layer using activation functions such as Sigmoid, Tanh or ReLU.

[0135] It is understandable that the multi-layer perception layer can be part of the retrieval model or an independent multi-layer perception layer. The multi-layer perception layer can be deployed on the computer device where the retrieval model is located, or it can be independently deployed on another computer device and connected to the computer device where the retrieval model is located through communication technology. Through distributed deployment, the design advantages of different computer devices can be better combined, thereby achieving the effect of improving the robustness and fluency of the system, reducing computing power overhead and saving manpower and material resources.

[0136] S105b, input the encoder input vector to the encoder module, and encode to obtain a feature encoding vector and a position encoding vector.

[0137] like Figure 5 As shown, the encoder module may include a position encoder, a self-attention layer, a residual processing layer, and a feedforward network layer.

[0138] Exemplarily, the encoder input vector is input to the position encoder of the encoder module to obtain a position encoding vector. The position encoding formula may include:

[0139]

[0140] Among them, the dimension of the position encoding vector is the same as the total number of dimensions of the encoder input vector, PE can represent the position encoding vector, pos can represent the position, and d model It can represent the total number of dimensions of the position encoding vector or encoder input vector, and i can represent the dimension.

[0141] Furthermore, the encoder input vector is input into the encoder module to obtain a feature encoding vector.

[0142] It should be noted that the encoder module may be an encoder based on an open source architecture such as Transformer, BERT, etc., and the encoder module may include a position encoder. The encoder module may be part of a retrieval model.

[0143] It is understandable that the position encoder enables the model to better capture deep semantic and contextual information, thereby improving the accuracy of the video retrieval method.

[0144] S105c. Obtain multiple initialization vectors, and add the position encoding vector to the multiple initialization vectors respectively to obtain multiple auxiliary vectors.

[0145] Exemplarily, a zero vector is used as an initialization vector, 12 initialization vectors are obtained, and 12 position codes are obtained, which are added to the 12 initialization vectors respectively to obtain 12 auxiliary vectors.

[0146] It is understandable that the number of auxiliary vectors or initialization vectors can be adjusted according to the specific type of video data. For example, in the financial industry, professional commentary video data generally contains 10 to 15 significant contents, so the number of auxiliary vectors or initialization vectors can be set to be within the range of 10 to 15. If it is set too small, the granularity of the division of video significance may be too large and the precision is insufficient. If it is set too large, the granularity is too fine, and the results will overlap, reducing the clarity and completeness of the search results.

[0147] S105d, input the encoder input data, the feature encoding vector and multiple auxiliary vectors into the decoder module to obtain multiple decoder output vectors.

[0148] Among them, the decoder module may include decoders based on open source architectures such as Transformer and BERT.

[0149] Further, such as Figure 5 As shown, the decoder module may include a multi-head self-attention layer, a residual processing layer, and a feed-forward network layer.

[0150] It is understandable that the decoder module may also include a fully connected layer.

[0151] S105e, inputting multiple decoder output vectors into the fully connected layer, and using the Hungarian matching algorithm to match and obtain the target video data.

[0152] Exemplarily, multiple decoder output vectors can be input into a fully connected layer to obtain multiple time period data; the inner products of the multiple time period data and the feature encoding vector are calculated respectively to obtain multiple hit degree values; based on the multiple hit degree values, X time period data with the largest corresponding hit degree values ​​are determined; based on the X time period data, the video data is cut to obtain the target video data.

[0153] Wherein, X is a positive integer.

[0154] It can be understood that the fully connected layer can be either a fully connected layer included in the retrieval model or an independent fully connected layer.

[0155] It can be understood that by using the Hungarian matching algorithm, the processed target video data can be prevented from overlapping with each other under the premise of high relevance to the text, thereby ensuring the accuracy of the video retrieval method and the availability of the retrieval results.

[0156] In some embodiments, see Figure 7 , Figure 7 1 is a schematic flow chart of the steps of the model training method provided in the embodiment of the present application. Figure 7 As shown, the model training method specifically includes steps S201 to S204.

[0157] S201: Obtaining annotation data of video data.

[0158] It is understandable that each video feature expression vector corresponding to the video data may be annotated according to the degree of correlation between the video data and the text data, thereby obtaining annotated data of the video data.

[0159] For example, if the first video feature expression vector v1 to the fifth video feature expression vector v5 in the video encoding data correspond to the first video frame to the fifth video frame in the video data and are related to the sentence in the text data, then the first video feature expression vector v1 to the fifth video feature expression vector v5 are marked as the first video feature expression vector To the fifth video feature expression vector

[0160] For example, if the sixth video frame to the ninth video frame in the video data corresponding to the sixth video feature expression vector v6 to the ninth video feature expression vector v9 in the video encoding data are related to the sentence in the text data, then the sixth video feature expression vector v6 to the ninth video feature expression vector v9 are marked as the sixth video feature expression vector To the ninth video feature expression vector

[0161] It should be noted that by annotating the video encoding data or the video feature expression vector to obtain the annotated data, the video frames related to the text data can be determined, which makes the subsequent calculation of losses and training models more convenient and improves the accuracy of the model.

[0162] S202: Calculate cross entropy loss data based on the labeled data and the video clip related value data.

[0163] Exemplarily, the formula used for calculation may include:

[0164]

[0165] in, Can represent cross entropy loss data, L v It can represent the total number of video feature expression vectors corresponding to video frames or each video frame in the video coding data. It can represent the video clip related value data, and ai can represent the annotation data of the i-th frame video data or its corresponding video feature expression vector.

[0166] It should be noted that the labeled data here can be understood as a number with a value of 0 or 1.

[0167] Exemplarily, if the i-th frame of video data is related to the corresponding text data, then a i The value of is 1. If the i-th frame video data is not related to the corresponding text data, then a i The value of is 0.

[0168] S203. Calculate and obtain orthogonal loss data according to the feature text encoding.

[0169] Exemplarily, the formula used for calculation may include:

[0170]

[0171] in, It can represent the orthogonal loss data, L d It can represent the number of codes or virtual vectors included in the feature text encoding. It can represent the mth code in the feature text code or the corresponding mth virtual vector. It can represent the nth code in the feature text code or the corresponding nth virtual vector.

[0172] S204. Train the multimodal model using a back propagation algorithm based on the cross entropy loss data and the orthogonal loss data.

[0173] Exemplarily, the total loss data can be obtained by summing the cross entropy loss data and the orthogonal loss data; and the multimodal model can be trained using the back propagation algorithm based on the total loss data.

[0174] The summing method may include weighted summing.

[0175] It is understandable that the multimodal model may include a first text encoder, a second text encoder, or a video encoder. During the parameter update process, some parameters of the first text encoder, the second text encoder, or the video encoder may be frozen to improve training efficiency and model quality and reduce risks such as gradient explosion or gradient disappearance.

[0176] Exemplarily, during the process of training the multimodal model, the parameters of the video encoder, the first text encoder, and the second text encoder can be frozen, and then the multimodal model can be trained using the back propagation algorithm based on the total loss data obtained by weighted summation of the cross entropy loss data and the orthogonal loss data.

[0177] Furthermore, the multimodal model may be trained using one or more algorithms of Adam (Adaptive Moment Estimation), RMSprop (Root Mean Square Propagation) and Momentum.

[0178] It is understandable that the multimodal model may include the first text encoder and the video encoder, or may be distinguished from the first text encoder and the video encoder as an independent model.

[0179] like Figure 8 As shown, Figure 8 1 is a schematic diagram of the structure of a video retrieval device provided in an embodiment of the present application, wherein the video retrieval device is used to execute the aforementioned video retrieval method. The video retrieval device can be configured in a terminal or a server.

[0180] like Figure 8 As shown, the video retrieval device 100 includes a data information acquisition module 101, a data feature extraction module 102, a data feature interaction module 103, a significant mark calculation module 104, and a target data retrieval module 105.

[0181] The data information acquisition module 101 is used to acquire video data and text data related to the video data;

[0182] A data feature extraction module 102, configured to encode the video data using a video encoder to obtain video encoding data, and to encode the text data using a first text encoder and a second text encoder to obtain a first text encoding and a second text encoding;

[0183] A data feature interaction module 103, configured to calculate and obtain video segment related value data using a multimodal model according to the first text code, the second text code and the video code data;

[0184] A significant mark calculation module 104 is used to obtain a plurality of significant vectors, and calculate a significant mark according to the plurality of significant vectors, the video segment related value data and the video encoding data;

[0185] The target data retrieval module 105 is used to input the significant mark and the video data into a retrieval model to obtain target video data.

[0186] In some embodiments, the video retrieval device 100 may further include a model training module, which is used to calculate cross entropy loss data based on the annotation data and the video clip related value data after calculating the video clip related value data based on the first text encoding, the second text encoding and the video encoding data using a multimodal model; calculate orthogonal loss data based on the feature text encoding; and train the multimodal model based on the cross entropy loss data and the orthogonal loss data using a back propagation algorithm.

[0187] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0188] The above-mentioned device can be implemented in the form of a computer program. Fig. 9 Runs on the computer device shown.

[0189] See also Fig. 9 , Fig. 9 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server. Fig. 9 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0190] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any video retrieval method.

[0191] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0192] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any video retrieval method.

[0193] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Fig. 9The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0194] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0195] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0196] Acquire video data and text data related to the video data; encode the video data using a video encoder to obtain video encoding data, and encode the text data using a first text encoder and a second text encoder to obtain a first text encoding and a second text encoding; calculate video segment related value data using a multimodal model based on the first text encoding, the second text encoding and the video encoding data; acquire multiple salient vectors, and calculate salient markers based on the video segment related value data and the video encoding data; input the salient markers and the video data into a retrieval model to obtain target video data.

[0197] In some embodiments, when the processor encodes the text data using the first text encoder and the second text encoder to obtain the first text encoding and the second text encoding, it is specifically used to implement:

[0198] The text data is input into the first text encoder to obtain a first text code; the first text code is input into the second text encoder to obtain a feature text code; the first text code and the feature text code are concatenated to obtain a second text code.

[0199] In some embodiments, the multimodal model includes the first text encoder, the second text encoder, or the video encoder.

[0200] In some embodiments, after the processor is configured to calculate the video segment related value data using a multimodal model according to the first text code, the second text code and the video code data, it is configured to implement:

[0201] Acquire the annotated data of the video data; calculate the cross entropy loss data according to the annotated data and the video clip related value data; calculate the orthogonal loss data according to the feature text encoding; and train the multimodal model using a back propagation algorithm based on the cross entropy loss data and the orthogonal loss data.

[0202] In some embodiments, when the processor is used to calculate the significance mark according to the video segment related value data and the video encoding data, it is specifically used to implement:

[0203] A correlation matrix is ​​calculated based on the video coding data and the multiple salient vectors; a candidate weight vector is calculated based on the video clip correlation value data and the correlation matrix; an instantaneous description vector is determined and calculated based on the candidate weight vector and the multiple salient vectors; and one or more salient markers are calculated based on the instantaneous description vector.

[0204] In some embodiments, the retrieval model includes at least a multi-layer perception layer, a fully connected layer, an encoder module and a decoder module.

[0205] In some embodiments, when the processor is used to input the significant mark and the video data into the retrieval model to obtain the target video data, it is specifically used to implement:

[0206] The saliency mark is input into the multi-layer perception layer, and the output of the multi-layer perception layer is spliced ​​with the video data to obtain an encoder input vector; the encoder input vector is input into the encoder module, and encoded to obtain a feature encoding vector and a position encoding vector; a plurality of initialization vectors are obtained, and the position encoding vector is added to the plurality of initialization vectors respectively to obtain a plurality of auxiliary vectors; the encoder input data, the feature encoding vector and the plurality of auxiliary vectors are input into the decoder module to obtain a plurality of decoder output vectors; the plurality of decoder output vectors are input into the fully connected layer, and the target video data is matched by using the Hungarian matching algorithm.

[0207] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program. The computer program includes program instructions. The processor executes the program instructions to implement any video retrieval method provided in the embodiment of the present application.

[0208] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0209] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A video retrieval method, characterized in that: include: Acquire video data and text data related to the video data; Encoding the video data using a video encoder to obtain video encoded data, and encoding the text data using a first text encoder and a second text encoder to obtain a first text code and a second text code; Obtaining video segment related value data by calculating using a multimodal model according to the first text code, the second text code and the video code data; Acquire a plurality of saliency vectors, and calculate a saliency mark according to the plurality of saliency vectors, the video segment related value data and the video encoding data; The saliency mark and the video data are input into a retrieval model to obtain target video data.

2. The method according to claim 1, characterized in that The method of encoding the text data using the first text encoder and the second text encoder to obtain the first text encoding and the second text encoding includes: Inputting the text data into the first text encoder to obtain a first text code; Inputting the first text code into the second text encoder to obtain a feature text code; The first text code and the characteristic text code are concatenated to obtain a second text code.

3. The method according to claim 1 or 2, characterized in that: The multimodal model includes the first text encoder, the second text encoder, or the video encoder.

4. The method according to claim 3, characterized in that After obtaining the video segment related value data by calculating the first text code, the second text code and the video code data using a multimodal model, the method further includes: Acquire the annotation data of the video data; Calculating cross entropy loss data according to the labeled data and the video clip related value data; Calculate and obtain orthogonal loss data according to the feature text encoding; The multimodal model is trained using a back propagation algorithm according to the cross entropy loss data and the orthogonal loss data.

5. The method according to claim 1, characterized in that: The calculating and obtaining the saliency mark according to the plurality of saliency vectors, the video segment related value data and the video encoding data comprises: Calculating a correlation matrix based on the video coding data and the plurality of salient vectors; Calculating a candidate weight vector according to the video clip correlation value data and the correlation matrix; Determining and calculating a transient description vector according to the candidate weight vector and the plurality of salient vectors; One or more saliency tags are calculated based on the instantaneous description vector.

6. The method according to claim 1, characterized in that The retrieval model at least includes a multi-layer perception layer, a fully connected layer, an encoder module and a decoder module.

7. The method according to claim 6, characterized in that The step of inputting the saliency mark and the video data into a retrieval model to obtain target video data includes: Inputting the saliency mark into the multi-layer perception layer, and concatenating the output of the multi-layer perception layer with the video data to obtain an encoder input vector; Inputting the encoder input vector into the encoder module, encoding to obtain a feature encoding vector and a position encoding vector; Acquire multiple initialization vectors, and add the position encoding vector to the multiple initialization vectors respectively to obtain multiple auxiliary vectors; Inputting the encoder input data, the feature encoding vector and the plurality of auxiliary vectors into the decoder module to obtain a plurality of decoder output vectors; The multiple decoder output vectors are input into the fully connected layer, and the target video data is matched using the Hungarian matching algorithm.

8. A video retrieval device, characterized in that: include: A data information acquisition module, used to acquire video data and text data related to the video data; A data feature extraction module, used to encode the video data using a video encoder to obtain video encoding data, and to encode the text data using a first text encoder and a second text encoder to obtain a first text encoding and a second text encoding; A data feature interaction module, configured to calculate and obtain video segment related value data using a multimodal model according to the first text code, the second text code and the video code data; A significant mark calculation module is used to obtain a plurality of significant vectors, and calculate a significant mark according to the plurality of significant vectors, the video segment related value data and the video encoding data; The target data retrieval module is used to input the significant mark and the video data into a retrieval model to obtain target video data.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the video retrieval method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the video retrieval method according to any one of claims 1 to 7.

Images