Video text retrieval method, electronic device and medium

Abstract

The present invention relates to a video text retrieval method. The method inputs a text to be retrieved and a video set into a pre-built and trained video text retrieval model, obtains the similarity between the text and each video in the video set, and uses the video with the highest similarity as the text retrieval result. The video text retrieval model includes: an initial feature extraction module that extracts the input text and video to generate initial text features and initial video features; an event feature extraction module that processes the initial text features and initial video features to obtain event text features and event video features; and a video text feature alignment module that reconstructs the event text features and event video features using a cross-modal self-attention mechanism to obtain adaptively semantically aligned reconstructed video features and reconstructed text features, and outputs the similarity between the reconstructed video features and reconstructed text features. Compared with existing technologies, the present invention has the advantages of high retrieval performance and low operating overhead.

Description

Technical Field

[0001] The present invention relates to the technical field of information retrieval, and in particular to a video text retrieval method, electronic equipment and medium. Background Art

[0002] Recently, with the rapid development of network and multimedia technologies, the demand for streaming media has increased. With the development of network infrastructure, the exchange of multimedia information such as images, audio, and video has become increasingly convenient. However, with the ever-increasing amount of multimedia information, people are becoming overwhelmed by the video information and find it difficult to quickly filter out the information they need from this vast amount of information. This has made video text retrieval an important research direction. Retrieving the most appropriate results from a library containing a large number of videos based on a text query depends on a suitable video-text similarity metric. If a suitable similarity metric is available, only the result with the highest similarity needs to be used as the retrieval result. Therefore, the main challenge of video text retrieval is to find a suitable video-text similarity metric.

[0003] With the rapid development of deep learning technology, deep convolutional network (CNN) models have been proven to be effective in handling complex visual tasks, while recurrent neural networks (RNNs) have been proven to be effective in handling complex language understanding tasks. Based on the development of these two technologies, deep learning has been successfully applied to video text retrieval tasks. For example, in the most common framework, video information and text information are mapped into a common high-dimensional space using CNNs and RNNs, respectively, and the similarity between the two is then measured using Euclidean distance or cosine distance. For video text retrieval tasks, researchers have proposed a variety of frameworks and models based on different deep learning technologies, including video text retrieval algorithms using graph neural networks for relationship modeling, video text retrieval algorithms using Transformer-based hierarchical alignment, video text retrieval algorithms based on local feature alignment, and video text retrieval algorithms based on large-scale pre-trained models. Despite the rapid development of existing deep learning-based video text retrieval algorithms, they still suffer from unsatisfactory retrieval performance and high operating costs. Summary of the Invention

[0004] The purpose of the present invention is to provide a video text retrieval method, electronic device and medium in order to overcome the defects of the above-mentioned prior art. The present invention can effectively capture important event information in videos and texts, remove redundant information without discarding fine-grained information, and optimize the accuracy of model supervision signals. The test experimental results on the MSRVTT dataset show that compared with the prior art, the retrieval performance of the present invention is greatly improved under the condition of similar computational complexity.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] According to a first aspect of the present invention, a video text retrieval method is provided, wherein the text to be retrieved and a video set are input into a pre-built and trained video text retrieval model, the similarity between the text and each video in the video set is obtained, and the video with the highest similarity is used as the text retrieval result;

[0007] The video text retrieval model includes:

[0008] Initial feature extraction module: extracts input text and input video respectively to generate initial text features and initial video features;

[0009] Event feature extraction module: includes a nonlinear event excitation layer and a weighted average layer. The initial text features and initial video features are respectively passed through the corresponding nonlinear event excitation layer to obtain a first matching score and a second matching score. The first matching score is used to reflect the degree of matching between the initial text features and the events of interest to the model, and the second matching score is used to reflect the degree of matching between the initial video features and the events of interest to the model. After the first matching score and the initial text features are processed by the weighted average layer, the event text features are obtained. After the second matching score and the initial video features are processed by the weighted average layer, the event video features are obtained.

[0010] Video-text feature alignment module: The event text features and event video features are reconstructed separately through the cross-modal self-attention mechanism to obtain adaptively semantically aligned reconstructed video features and reconstructed text features, and the similarity of each reconstructed video feature and reconstructed text feature is output.

[0011] Preferably, the initial feature extraction module includes a pre-trained BERT network and a ViT network, and performs feature extraction on the input text through the pre-trained BERT network to generate initial text features; performs feature extraction on each video frame of the input video through the pre-trained ViT network, and then obtains the temporal information of the video frame features through a multi-layer Transformer model to obtain the initial video features.

[0012] Preferably, the formula describing the first matching score and the second matching score is:

[0013]

[0014]

[0015] Where, is the first matching score, is the second matching score, κ is the parameter that controls the sharpness of the distribution, is a nonlinear event excitation layer for video input, is the nonlinear event excitation layer for text input, t j is the jth text sample, N t is the number of text samples in the dataset, i and j are indexes, and v j is the jth video sample, N v is the number of video samples in the dataset.

[0016] Preferably, the formula describing the reconstruction process of the cross-modal self-attention mechanism is:

[0017]

[0018]

[0019] Where, and Reconstruct video features and reconstruct text features respectively, K v and K t are the number of key features of the video and the number of key features of the text, τ is the temperature coefficient, is the event video feature of the i-th video The transpose of is the text feature of the i-th text event The transpose of is the event video feature of the jth video, is the event text feature of the j-th text.

[0020] Preferably, the formula describing the similarity between the reconstructed video features and the reconstructed text features is:

[0021]

[0022] Preferably, during the training process of the video text retrieval model, the sample attributes of each input video text pair are optimized by using a cosine decay threshold, and according to the optimization results, the model L is optimized based on the bidirectional infoNCE as the loss function.

[0023] Preferably, the calculation formula describing the threshold value of cosine decay is:

[0024] λ=λ end -(λ end -λ start )·cos(l+pπ)

[0025] Where p is the percentage of training process, λ is the threshold, and λ end and λ start are the thresholds at the beginning and end of training, respectively.

[0026] Preferably, the loss function is calculated as:

[0027]

[0028]

[0029]

[0030] Among them, n is the number of sample pairs in the data set, L v2t is the infoNCE loss function with video samples as query samples, L t2v is the infoNCE loss function with text samples as query samples, N Vi represents the negative sample set obtained by taking the i-th video as the query, N Ti represents the negative sample set obtained by taking the i-th text as the query, T neg is the negative sample set N Vi The text negative samples contained in V neg is the negative sample set N Ti The video negative samples contained in .

[0031] According to a second aspect of the present invention, the present invention provides an electronic device, comprising:

[0032] One or more processors; a memory; and one or more programs stored in the memory, wherein the one or more programs include instructions for executing any of the above-described video text retrieval methods.

[0033] According to a third aspect of the present invention, the present invention provides a computer-readable storage medium comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs comprise instructions for executing any of the video text retrieval methods described above.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] (1) The present invention strengthens the extraction of key event features by setting a nonlinear event excitation layer, which can effectively capture key events in the original features, avoid the interference of irrelevant noise, and thus improve the quality of model extraction features.

[0036] (2) The present invention adopts feature reconstruction based on the cross-modal self-attention mechanism. By calculating the matching scores between video and text event features, the video and text features are reconstructed, thereby adaptively aligning cross-modal information, making the calculation of video-text similarity more reasonable.

[0037] (3) The present invention jointly judges the sample attributes through the cosine attenuation threshold and sample confidence, which can effectively eliminate pseudo-negative samples, thereby enabling the model to obtain more accurate supervision signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of the structure of a video text retrieval model of a video text retrieval method provided in this embodiment. DETAILED DESCRIPTION

[0039] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0040] This embodiment provides a video text retrieval method, comprising the following steps:

[0041] S1: Obtain n video-text pairs as a training set. Each video is semantically related to only one text. Semantically related video-text pairs are used as positive samples, and semantically unrelated video-text pairs are used as negative samples.

[0042] S2: Build a video text retrieval model, input the training set obtained in S1 into the built video text retrieval model for training, and obtain a trained video text retrieval model.

[0043] Specifically, refer to Figure 1 As shown in the figure, the video text retrieval model includes an initial feature extraction module, an event feature extraction module and a video text feature alignment module.

[0044] Among them, the initial feature extraction module extracts the input text and input video respectively to generate initial text features and initial video features.

[0045] As an optional implementation, the initial feature extraction module includes a pre-trained BERT network and a ViT network, and extracts features from the input text through the pre-trained BERT network to generate initial text features. The initial text features include N t The pre-trained ViT network is used to extract features from each video frame of the input video, and the temporal information of the video frame features is obtained through the multi-layer Transformer model, thereby obtaining the initial video features. The initial video features include N v Video frame features V.

[0046]

[0047]

[0048] Event feature extraction module: includes a nonlinear event excitation layer and a weighted average layer. The initial text features and the initial video features pass through the corresponding nonlinear event excitation layer respectively to obtain a first matching score and a second matching score. The first matching score is used to reflect the degree of matching between the initial text features and the events that the model focuses on, and the second matching score is used to reflect the degree of matching between the initial video features and the events that the model focuses on. After the first matching score and the initial text features are processed by the weighted average layer, the event text features are obtained. After the second matching score and the initial video features are processed by the weighted average layer, the event video features are obtained.

[0049] The formula describing the first match score and the second match score is:

[0050]

[0051]

[0052] Where, is the first matching score, is the second matching score, k is the parameter that controls the sharpness of the distribution, is a nonlinear event excitation layer for video input, is the nonlinear event excitation layer for text input, t j is the jth text sample, N t is the number of text samples in the dataset, i and j are indexes, and v j is the jth video sample, N v is the number of video samples in the dataset.

[0053] The formula for describing event text features and event video features is:

[0054]

[0055]

[0056] Where, is the event text feature of the i-th text, is the event video feature of the i-th video

[0057] By inputting the initial text features and event video features into the corresponding nonlinear event excitation layer respectively, the model can focus on the features of important events as much as possible without losing fine-grained information, and remove the background information, noise information and repeated information that are commonly present in videos and texts respectively. The matching score corresponding to the removed information is lower.

[0058] Video-text feature alignment module: The event text features and event video features are reconstructed separately through the cross-modal self-attention mechanism to obtain adaptively semantically aligned reconstructed video features and reconstructed text features, and the similarity of each reconstructed video feature and reconstructed text feature is output.

[0059] The formula describing the reconstruction process of the cross-modal self-attention mechanism is:

[0060]

[0061]

[0062] Where, and Reconstruct video features and reconstruct text features respectively, K v and K t are the number of key features of the video and the number of key features of the text, τ is the temperature coefficient, for The transpose of for The transpose of is the event video feature of the jth video, is the event text feature of the j-th text.

[0063] After reconstructing the video features and text features, due to the nature of cross-modal self-attention, the reconstructed features can be aligned with the original event features at the semantic level. Therefore, the dot product of the corresponding elements can be directly calculated and the average is taken to obtain the overall similarity of the video and text. The specific calculation method is as follows:

[0064]

[0065] During the training process, in order to prevent the model from treating semantically similar sample pairs as negative sample pairs for learning, this embodiment jointly judges the sample attributes through the cosine decay threshold and sample confidence, which can effectively eliminate pseudo-negative samples, thereby enabling the model to obtain more accurate supervision signals.

[0066] Specifically, the calculation formula describing the threshold of cosine attenuation is:

[0067] λ=λ end -(λ end -λ Stan )·cos(1+pπ)

[0068] Where p is the percentage of training process, λ is the threshold, and λ end and λ start are the thresholds at the beginning and end of training, respectively.

[0069] Based on the threshold of cosine decay, the optimized negative sample set Nq as follows:

[0070]

[0071] Among them, N is the original negative sample set, q is the query sample, and p is the sample in the sample library.

[0072] The threshold is set higher in the early stage of training to prevent instability in the early stage of model training, and is set lower in the later stage to avoid performance degradation of the model due to erroneous supervision signals as much as possible.

[0073] Based on the calculated video-text similarity and the optimized negative sample set, the bidirectional infoNCE is used as the loss function to optimize the model L. The calculation method is:

[0074]

[0075]

[0076]

[0077] Among them, n is the number of sample pairs in the data set, L v2t is the infoNCE loss function with video samples as query samples, L t2v is the infoNCE loss function with text samples as query samples, N Vi represents the negative sample set obtained by taking the i-th video as the query, N Ti represents the negative sample set obtained by taking the i-th text as the query, T neg is the negative sample set N Vi The text negative samples contained in V neg is the negative sample set N Ti The video negative samples contained in .

[0078] S3: The text and video set to be retrieved are input into a pre-built and trained video text retrieval model to obtain the similarity between the text and each video in the video set, and the video with the highest similarity is used as the text retrieval result.

[0079] In order to verify the performance of the present invention, the following experiments were designed.

[0080] The experiment compared the invented model (ATE) with other models on a video text retrieval benchmark dataset. Among them, CLIP2Video is one of the most effective methods currently, and the invented model still has a certain accuracy advantage over it, as shown in Table 1.

[0081] Table 1 Comparative experimental results of ATE model and other models on video text retrieval dataset

[0082]

[0083]

[0084] According to a second aspect of the present invention, this embodiment provides an electronic device, including:

[0085] One or more processors; a memory; and one or more programs stored in the memory, wherein the one or more programs include instructions for executing any of the above-described video text retrieval methods.

[0086] According to the third aspect of the present invention, this embodiment provides a computer-readable storage medium, including one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing any of the video text retrieval methods described above.

[0087] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0088] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A video text retrieval method, characterized in that: The text and video set to be retrieved are input into a pre-built and trained video text retrieval model to obtain the similarity between the text and each video in the video set, and the video with the highest similarity is used as the text retrieval result; The video text retrieval model includes: Initial feature extraction module: extracts input text and input video respectively to generate initial text features and initial video features; Event feature extraction module: includes a nonlinear event excitation layer and a weighted average layer. The initial text features and initial video features are respectively passed through the corresponding nonlinear event excitation layer to obtain a first matching score and a second matching score. The first matching score is used to reflect the degree of matching between the initial text features and the events of interest to the model, and the second matching score is used to reflect the degree of matching between the initial video features and the events of interest to the model. After the first matching score and the initial text features are processed by the weighted average layer, the event text features are obtained. After the second matching score and the initial video features are processed by the weighted average layer, the event video features are obtained. Video-text feature alignment module: This module reconstructs event text features and event video features through a cross-modal self-attention mechanism to obtain adaptively semantically aligned reconstructed video features and reconstructed text features, and outputs the similarity between each reconstructed video feature and reconstructed text feature. During the training process of the video text retrieval model, the sample attributes of each input video text pair are optimized through the threshold of cosine decay; The threshold value describing the cosine decay is calculated as: Where, p is the training process percentage, is the threshold, and are the thresholds at the beginning and end of training, respectively.

2. A video text retrieval method according to claim 1, characterized in that: The initial feature extraction module includes a pre-trained BERT network and a ViT network. The pre-trained BERT network is used to extract features from the input text to generate initial text features. The pre-trained ViT network is used to extract features from each video frame of the input video, and then the temporal information of the video frame features is obtained through a multi-layer Transformer model to obtain the initial video features.

3. A video text retrieval method according to claim 1, characterized in that: The formula describing the first match score and the second match score is: Where, is the first matching score, is the second matching score, is a parameter that controls the sharpness of the distribution, is a nonlinear event excitation layer for video input, A nonlinear event excitation layer for text input, For the j Text samples, is the number of text samples in the dataset, i and j is the index, For the j Video samples, is the number of video samples in the dataset.

4. A video text retrieval method according to claim 1, characterized in that: The formula describing the reconstruction process of the cross-modal self-attention mechanism is: Where, and Reconstruct video features and reconstruct text features respectively. and are the number of video key features and the number of text key features, respectively. is the temperature coefficient, For the i Event video features of videos The transpose of For the i Text event text features The transpose of For the j Event video features of videos, For the j Event text features of a text.

5. A video text retrieval method according to claim 4, characterized in that: The formula describing the similarity between the reconstructed video features and the reconstructed text features is: 。 6. A video text retrieval method according to claim 5, characterized in that: During the training process of the video text retrieval model, the sample attributes of each input video text pair are optimized by the threshold of cosine decay, and according to the optimization results, the model is optimized based on the bidirectional infoNCE as the loss function. .

7. A video text retrieval method according to claim 6, characterized in that: The loss function is calculated as: in, is the number of sample pairs in the dataset, is the infoNCE loss function with video samples as query samples, is the infoNCE loss function with text samples as query samples, Indicates the i The negative sample set obtained by using the video as the query is Indicates the i The negative sample set obtained by using text as query, is the negative sample set The text negative samples contained in is the negative sample set The video negative samples contained in .

8. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs stored in a memory, wherein the one or more programs include instructions for executing the video text retrieval method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The method comprises one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the video text retrieval method according to any one of claims 1 to 7.

Images