Video time retrieval method and system based on dual-stream Transformer
Through the video moment retrieval system based on dual-stream Transformer, the problems of insufficient understanding of the differential characteristics between modes and within modes and loss of long video processing information are solved, and higher video moment retrieval accuracy and stability are achieved.
Patent Information
- Application Number
- CN202210065751.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-20
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-01-20
AI Technical Summary
The existing video time search methods are not fully understood in terms of global local features between modes and within modes, and there is a problem of information loss in processing long videos.
Using a video moment retrieval system based on dual-stream Transformer, including training data module, feature embedding module, dual-stream Transformer feature interaction module, regression positioning module and loss module, the local and global information of video and text are captured through dual-stream Transformer, and self-attention and The cross-modal attention mechanism performs feature updates, and the continuous timing confidence score and start-end moment probability are calculated by multi-layer perceptrons.
It significantly improves the accuracy and performance of video moment retrieval, can better find video clips corresponding to text description, and improves the stability of the system and the processing ability of long videos.
Smart Images

Figure CN114417056B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of video time retrieval, and relates to a video time retrieval method and system based on dual-stream Transformer. Background Art
[0002] In recent years, multimedia information such as videos has come into the public eye, among which video moment retrieval is one of the basic tasks of computer vision. The goal of this task is to find a period of time in the video that corresponds to the query text. Unlike traditional image retrieval, videos contain temporal information. In addition, this task introduces natural language as a guide, making the task more direct and accurate. However, the existing detection methods do not fully understand the two modalities and cannot take good care of the global and local feature differences between and within the modalities. In addition, when facing long videos, processing methods such as pre-dividing video clips will cause information loss. How to better solve the above problems is one of the problems that need to be solved urgently. Summary of the invention
[0003] Aiming at the problems existing in the traditional video time retrieval process, the present invention proposes a novel video time retrieval method and system based on dual-stream Transformer.
[0004] In order to achieve the above object, the present invention is implemented by adopting the following technical solutions:
[0005] A video moment retrieval system based on dual-stream Transformer, including the following modules:
[0006] Training data module, which is used to train video and natural query description pairs;
[0007] Feature embedding module, used to extract features of videos and sentences;
[0008] The two-stream Transformer feature interaction module is used to obtain local and global information of video and text modalities, which can effectively capture the interaction information between language and long and short videos;
[0009] The regression positioning module is used to obtain continuous temporal confidence scores and start and end time probabilities to determine the final target segment;
[0010] The loss module is used to calculate the loss function of the training process. First, the binary cross entropy loss is used to calculate the matching degree of the video-query language description pair; then the cross entropy loss function is used to calculate the start and end time.
[0011] Reasoning module: It is used to select the best prediction segment moment based on the video timing score sequence and probability sequence.
[0012] An electronic device comprising the above system comprises a memory and a processor and computer instructions stored in the memory and executed on the processor, wherein the computer instructions complete the functions of each module in the above system when executed by the processor.
[0013] The present invention provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the functions of each module in the above system are completed.
[0014] Compared with the prior art, the advantages and positive effects of the present invention are:
[0015] The algorithm of the present invention is complete and the structure is reasonably designed. With the help of dual-stream Transformer, visual features and text features can be better captured, significantly improving the accuracy of video moment retrieval. At the same time, for long videos, Transformer uses its strong temporal modeling capabilities to better find video clips corresponding to text descriptions, improving the performance and stability of the present invention.
[0016] Compared with traditional network models, such as simple multiplication and addition fusion modes, this invention introduces the interactive mechanism of Transformer, that is, paying attention to global and local information between visual and textual modalities. On the one hand, the self-attention model is used to learn local information; on the other hand, the cross-modal attention mechanism is used to focus on global differences, thereby updating the features of the two modalities, and then combining the two to obtain the relationship between the video and the query text. The fused features are passed through a multi-layer perceptron to obtain continuous temporal confidence scores and start and end time probabilities to determine the final target segment. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 A flowchart is provided for Example 1 of the video time retrieval method based on dual-stream Transformer of the present invention.
[0018] Figure 2 Schematic diagram of the network of the video moment retrieval method based on dual-stream Transformer of the present invention. DETAILED DESCRIPTION
[0019] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described below in conjunction with specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0020] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.
[0021] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0022] Example 1
[0023] Unless otherwise specified, all technical and scientific terms used in this embodiment have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs. The video moment retrieval method based on the dual-stream Transformer provided in this embodiment is only for a specific task and is not universal. Therefore, in this embodiment, Figure 1 As shown in the figure, a new and universal dynamic cross-modal fusion framework is proposed. Based on this framework, a video moment retrieval method based on a two-stream Transformer is provided. The specific steps are as follows:
[0024] S1: Feature extraction, used to extract features of videos and query statements;
[0025] Specifically, the video is firstly extracted using an expanded 3D convolutional neural network (I3D), which is similar to the C3D network, except that the RGB and optical flow are trained separately. During the test, their predicted results are averaged and merged. Taking the commonly used dataset Charades-STA as an example, the dimension of the extracted video features is (N, 1024), where N is the total number of video frames. For the query statement, GloVe is used to extract the embedded 300-dimensional word features. The output size of LSTM is set to 256, and the size of the hidden layer in LSTM2 is also 256. This bidirectional LSTM is used for natural language feature embedding.
[0026] S2: Modality fusion based on dual-stream Transformer is used to obtain local and global features of video and text modalities, which can effectively enhance the interactive information between language and vision fields;
[0027] Specifically, S2-1: Video-level Transformer branch flow. This branch contains a self-attention module and a dual-attention module. First, the query sentence features are linearly transformed, and the key vector K is obtained according to the self-attention mechanism. q and a numerical vector V q; Then linearly transform the query text, and similarly, obtain the query vector Q of the video v , in order to guide the bimodal interaction and focus on the local information in the video. Through the guidance vector, the relevant matching segments in the video can get more weight, and the irrelevant segments can get less weight. The above process can be expressed as:
[0028] K q =fc(F Q θ qk );V q =fc(F Q θ qv );Q v =fc(F V θ vq ).
[0029] Among them, fc is a multi-layer perceptron, F v is the video feature, F Q is the text feature, θ qk ,θ qv ,θ vq They are the parameters of the corresponding network to be learned.
[0030] Then refer to the dual attention module, that is, the multi-head attention model, which can be expressed as:
[0031] F V '=MA(K q ,V q ,Q v )·F V .
[0032] Where MA stands for the general multi-head attention model.
[0033] S2-2: Text-level Transformer branch flow. Similar to S2-1, this branch contains a self-attention module and a dual-attention module. First, the video features are linearly transformed, and the key vector K is obtained according to the self-attention mechanism. v and a numerical vector V v ; Then linearly transform the query text, and similarly, obtain the query vector Q of the video q , in order to guide the bimodal interaction and focus on the local information in the video. Through the guidance vector, the relevant matching segments in the video can get more weight, and the irrelevant segments can get less weight. The above process can be expressed as:
[0034] K v =fc(F Q θ vk );V v =fc(F Q θ vv );Qq =fc(F V θ qv ).
[0035] Among them, fc is a multi-layer perceptron, F v is the video feature, F Q is the text feature, θ vk ,θ vv ,θ qq They are the parameters of the corresponding network to be learned.
[0036] Then refer to the dual attention module, that is, the multi-head attention model, which can be expressed as:
[0037] F Q '=MA(K v ,V v ,Q q )·F Q .
[0038] Where MA stands for the general multi-head attention model.
[0039] At this point, the video and text after the interaction are updated to F V ' and F Q '. In order to be simple and feasible and reduce the computational cost, this method directly concatenates the two features, that is, F mul =concat(F V ',F Q ').
[0040] S3: Positioning regression module. The regression positioning module is used to obtain continuous temporal confidence scores and start and end time probabilities to determine the final target segment. The fusion features obtained in the above process are passed through a multi-layer perceptron to obtain a matching score. In order to solve the problems of information loss caused by the need to slide a window to divide the pre-selected video segments and poor effect of long videos, this method passes the above matching scores through a multi-layer perceptron and normalizes them to obtain the start and end probabilities of each frame of the video, which can be expressed as:
[0041] start,end=softmax(fc(score)).
[0042] S4: Loss calculation. The loss function used to calculate the training process consists of two parts: one is the matching score loss L sc and position loss L lo First, we use the binary cross entropy loss to calculate the matching degree of the video-query language description pair; then we use the cross entropy loss function to calculate the start and end time. The specific expression is:
[0043] L sc =BE(score,scogt )
[0044] L lo =CE(start,start gt )+CE(end,end gt ).
[0045] Where BE stands for general binary cross entropy loss and CE stands for cross entropy loss. gt represents the true match score label of each frame, score represents the predicted score, start and end represent the predicted start and end probabilities of the target segment, respectively. gt and end gt Represent the actual start and end time of the target segment respectively.
[0046] S5: Reasoning stage. In the test stage, for the video time series score sequence, when the matching score goes from high to low, it means that the frame is the end time of the candidate moment; when the matching score goes from low to high, it means that the frame is the start time of the candidate moment. From the above candidate moments, the one with the highest joint probability of the predicted start and end is selected.
[0047] Example 2
[0048] This embodiment provides a video moment retrieval system based on a dual-stream Transformer, including: a video and text to be processed, extracting features of the video and the sentence, and also including the following modules:
[0049] Training data module, which is used to train video and natural query description pairs;
[0050] Feature embedding module, used to extract features of videos and sentences;
[0051] The two-stream Transformer feature interaction module is used to obtain local and global information of video and text modalities, which can effectively capture the interaction information between language and long and short videos;
[0052] The regression positioning module is used to obtain continuous temporal confidence scores and start and end time probabilities to determine the final target segment;
[0053] The loss module is used to calculate the loss function of the training process. First, the binary cross entropy loss is used to calculate the matching degree of the video-query language description pair; then the cross entropy loss function is used to calculate the start and end time.
[0054] Reasoning module: It is used to select the best prediction segment moment based on the video timing score sequence and probability sequence.
[0055] Example 3
[0056] This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, the method described in Embodiment 1 or the functions of each module in the system described in Embodiment 2 are completed.
[0057] In this embodiment, the processor may be a central processing unit CPU, or may be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
[0058] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.
[0059] Example 4
[0060] This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method described in Embodiment 1 or the functions of each module in the system described in Embodiment 2 are completed.
[0061] The method in Example 1 can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0062] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in conjunction with this embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0063] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.
[0064] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.
Claims
1. A video time retrieval method based on dual-stream Transformer, characterized in that: The steps include: Get the video to be processed; Use I3D network to extract feature information of all video clips; Obtaining a natural language query statement to be processed; Use Glove and bidirectional LSTM to extract query features; Two-stream Transformer feature interaction obtains local and global information of video and text modalities, and captures the interactive information between language and long and short videos; Sliding multi-layer perceptron, capturing the complete temporal information of long videos with an end-to-end model, and inferring the best prediction segment; The specific steps are as follows: S1: Feature extraction, using dilated 3D convolutional neural network to extract features of videos and query statements; S2: Modal fusion based on two-stream Transformer obtains local and global features of video and query modalities, and strengthens the interactive information between language and vision fields; S2-1: Video-level Transformer branch flow, including a self-attention module and a dual-attention module; first, the query statement features are linearly transformed, and the key vector Kq and the numerical vector Vq are obtained according to the self-attention mechanism; then the query statement is linearly transformed to obtain the query vector Qv of the video, so as to guide the dual-modal interaction and focus on the local information in the video. Through the guidance vector, the relevant matching fragments in the video are given more weight, and the irrelevant fragments are given less weight. The formula is as follows: , in, fc is a multi-layer perceptron, F v is the video feature, F Q It is a text feature , , , are the parameters to be learned by the corresponding network; Referring to the dual attention module again, the formula is as follows: , Where MA stands for the general multi-head attention model; S2-2: Text-level Transformer branch flow, including a self-attention module and a dual-attention module; first, the video features are linearly transformed, and the key vector Kv and the numerical vector Vv are obtained according to the self-attention mechanism; then the query text is linearly transformed to obtain the query vector Qq of the video, guiding the dual-modal interaction and focusing on the local information in the video. Through the guidance vector, the relevant matching fragments in the video are given more weight, and the irrelevant fragments are given less weight. The formula is as follows: , in, fc is a multi-layer perceptron, F v is the video feature, F Q It is a text feature , , , are the parameters to be learned by the corresponding network; Referring to the dual attention module again, the formula is as follows: , Where MA stands for the general multi-head attention model; The video and text after the interaction are updated as follows: and ,Will and Fusion: ; S3: Positioning regression module, the fusion features obtained in step S2 are passed through a multi-layer perceptron to obtain a matching score; the matching score is passed through a multi-layer perceptron and normalized to obtain the start and end probabilities of each frame of the video, which is expressed as follows: , S4: Loss calculation, matching score loss and position loss , first use the binary cross entropy loss to calculate the matching degree of the video-query language description pair; then use the cross entropy loss function to calculate the start and end time, as shown below: , Where BE stands for general binary cross entropy loss, CE stands for cross entropy loss, sco gt represents the true matching score label of each frame, score represents the predicted score, start and end represent the predicted start and end probabilities of the target segment respectively, and Respectively represent the real start and end time of the target segment; S5: Inference stage, when the matching score goes from high to low, it means that the frame is the end time of the candidate moment; when the matching score goes from low to high, it means that the frame is the start time of the candidate moment; select the video moment with the highest joint probability of multiplying the start time and the end time, and output the result.
2. A system for implementing the dual-stream Transformer video time retrieval method according to claim 1, characterized in that: Includes the following modules: Training data module, Video and natural query description pairs for training; Feature Embedding Module, Used to extract features of videos and sentences; Two-stream Transformer feature interaction module, It is used to obtain local and global information of video and text modalities, and can effectively capture the interactive information between language and long and short videos; Regression positioning module, Used to obtain continuous timing confidence scores and start and end time probabilities to determine the final target segment; Loss module, The loss function used to calculate the training process first uses the binary cross entropy loss to calculate the matching degree of the video-query language description pair; then the cross entropy loss function is used to calculate the start and end time; Reasoning module, It is used to select the best prediction segment moment according to the video timing score sequence and probability sequence.
3. An electronic device comprising a memory and a processor and computer instructions stored in the memory and executed on the processor, characterized in that: When the computer instructions are executed by the processor, the video moment retrieval method based on dual-stream Transformer according to claim 1 is executed.
4. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by the processor, the video moment retrieval method based on dual-stream Transformer according to claim 1 is executed.
