Video-linguistic understanding method and system

Abstract

The application provides a video-language understanding method and system, relates to the technical field of video understanding, and obtains a video and text to be understood; based on a trained understanding model, the input video and text are processed to generate a final understanding result; wherein the understanding model comprises a video encoder and a text query component, the video encoder comprises a clustering component and a timing component; the image blocks of all video frames in the video are filtered by using the clustering component to obtain video frame embeddings after eliminating redundant information; the timing dependency between the video frame embeddings is reconstructed by using the timing component to obtain a video feature matrix; based on the video feature matrix, the embedding of the text is performed by using the text query component to obtain the final understanding result; the application comprehensively considers three challenges of information redundancy, time dependency and scene complexity, proposes an understanding model composed of three key components, each component is aimed at a specific challenge, and the video-language understanding ability of an intelligent agent is significantly improved.

Description

Technical Field

[0001] This invention belongs to the field of video understanding technology, and in particular relates to a video-language understanding method and system. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Over the past few years, both artificial intelligence technology and video streaming have experienced significant growth. Video-language understanding reflects an agent's ability to perceive and interpret visual and textual content in the real world. This ability can be applied to a range of tasks, such as cross-modal text-to-video retrieval, video description generation, and video question answering. However, this technology faces three main challenges: information redundancy, time dependence, and scene complexity. Since these challenges are complementary, such as reducing redundant information in videos, which can significantly reduce the complexity of video scenes, it is particularly important to consider all three challenges simultaneously.

[0004] Current methods primarily address the information redundancy problem by selecting meaningful markers or keyframes. However, this selection disrupts the spatial consistency between information, thus increasing the difficulty of temporal modeling. On the other hand, placing the feature selection module last to address the temporal dependency challenge may introduce excessive redundant information into the preceding modules, which is not the optimal solution. In contrast, some methods choose to ignore the redundancy of video information and focus only on solving the problems of temporal dependency and scene complexity.

[0005] Therefore, existing methods mainly focus on solving one or two of the challenges, failing to take into account other factors that affect video understanding, resulting in low accuracy and performance. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, this invention provides a video-language understanding method and system. Taking into account the three challenges of information redundancy, time dependence, and scene complexity, it proposes an understanding model composed of three key components, each targeting a specific challenge, which significantly improves the video-language understanding ability of the agent.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0008] The first aspect of this invention provides a video-language understanding method.

[0009] A video-language understanding method that acquires the video and text to be understood;

[0010] Based on the trained understanding model, the input video and text are processed to generate the final understanding result;

[0011] The understanding model includes a video encoder and a text query component. The video encoder includes a clustering component and a temporal component. The clustering component is used to filter image blocks of all video frames in the video to obtain video frame embeddings after eliminating redundant information. The temporal dependency relationship between video frame embeddings is reconstructed through the temporal component to obtain a video feature matrix. Based on the video feature matrix, the text query component is used to embed text to obtain the final understanding result.

[0012] The understanding model can handle at least text-video cross-modal retrieval tasks, video description generation tasks, or video question answering tasks.

[0013] Furthermore, the video encoder also includes a ViT layer;

[0014] The ViT layer is used to encode each video frame in blocks to obtain semantically meaningful image block embeddings. Image block embeddings belonging to the same video frame constitute video frame embeddings.

[0015] Furthermore, the step of using a clustering component to filter image blocks across all video frames specifically involves:

[0016] The video frames output from the ViT layer are embedded into groups of S segments, each segment containing F / S frames;

[0017] Clustering algorithm is used to analyze the F / S(1+P) values ​​contained in each segment. in Clustering of image blocks produces (1+P) in ( ) clusters, selecting blocks located at the centroid of each cluster to form a video embedding V after removing redundant information. k ;

[0018] Where S represents the number of segments, F represents the number of video frames, and P... in This indicates the number of blocks in each video frame.

[0019] Furthermore, in the temporal component, an information tagging mechanism is used to reconstruct the temporal dependencies between video frame embeddings, specifically as follows:

[0020] Embed the video frames after removing redundant information into V k Inputting the ViT-ATM layer yields a video feature matrix with temporal dependencies.

[0021] Furthermore, the text query component integrates an L-layer MoED, which consists of four modules: Bidirectional Self-Attention Mechanism (BiSA), Causal SA, Channel Attention (CA), and Feedforward Neural Network (FFA). These four modules form three variants to complete the corresponding task-specific text embedding: a text encoder, a video-based text encoder, and a video-based text decoder.

[0022] For text-video cross-modal retrieval tasks, the similarity between the video feature matrix and the text feature matrix obtained by the text encoder is calculated to obtain a set of videos that meet the similarity conditions.

[0023] For the video description generation task, the video feature matrix is ​​input into the video-based text decoder to generate text, which serves as the video description text.

[0024] For video question answering tasks, the video feature matrix and the question text are input into a video-based text encoder to obtain a multimodal feature matrix. Then, the multimodal feature matrix is ​​input into a video-based text decoder to generate text, which serves as the answer to the question.

[0025] Furthermore, the text encoder encodes the input text using BiSA and FFN at each layer, appends a [CLS] token to the beginning of the text input, and outputs a text feature matrix;

[0026] The video-based text encoder collects task-related visual information by adding CA between BiSA and FFN in each layer of the text encoder; in CA, the input text is used as a query, and the embedded video feature matrix is ​​used as a key and value to generate a multimodal feature matrix.

[0027] The video-based text decoder replaces the BiSA layer of the video-based text encoder with a Causal SA layer to decode the input multimodal feature matrix into text.

[0028] Furthermore, the text-video cross-modal retrieval task is divided into two stages: recall and reordering.

[0029] In the recall phase, the Top Q video segments are recalled by calculating the cosine similarity between the [cls] tokens in the video feature matrix and the text feature matrix obtained by the text encoder.

[0030] In the reordering stage, the video-text query is input into the video-based text encoder, and the output [Encode] is embedded into the input fully connected layer and the sigmoid function to obtain the final score. The Q segments of video are reordered according to the score, and a preset number of videos are selected to form a video set.

[0031] Furthermore, the video question answering task also includes a multiple-choice question answering task, which is performed as a classification task. The question and answer are concatenated into a complete sentence, and then input together with the video feature matrix into a video-based text encoder to encode the video and question-answer text pairs into a multimodal feature matrix. Finally, the multimodal feature matrix is ​​input into a linear layer and a softmax layer to obtain the score of the best answer.

[0032] Furthermore, the training of the understanding model specifically involves:

[0033] For the text encoder, video-text contrastive loss is used to encourage matching video-text pairs to have more similar [cls] tags than mismatched video-text pairs to align the feature spaces of video and text.

[0034] For video-based text encoders, video-text matching loss is used to learn a multimodal representation of video and text, capturing fine-grained alignment between video and text;

[0035] For a video-based text decoder, a language modeling loss is used, and the cross-entropy loss is optimized to train the model to maximize the probability of the text in an autoregressive manner.

[0036] A second aspect of the present invention provides a video-language understanding system.

[0037] A video-language understanding system includes an acquisition module and a understanding module:

[0038] The acquisition module is configured to acquire the video and text to be understood.

[0039] The understanding module is configured to process the input video and text based on a trained understanding model and generate the final understanding result.

[0040] The understanding model includes a video encoder and a text query component. The video encoder includes a clustering component and a temporal component. The clustering component is used to filter image blocks of all video frames in the video to obtain video frame embeddings after eliminating redundant information. The temporal dependency relationship between video frame embeddings is reconstructed through the temporal component to obtain a video feature matrix. Based on the video feature matrix, the text query component is used to embed text to obtain the final understanding result.

[0041] The understanding model can handle at least text-video cross-modal retrieval tasks, video description generation tasks, or video question answering tasks.

[0042] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of a video-language understanding method as described in the first aspect of the present invention.

[0043] A fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a video-language understanding method as described in the first aspect of the present invention.

[0044] The above one or more technical solutions have the following beneficial effects:

[0045] (1) In order to improve the video-language understanding ability of intelligent agents and overcome the limitations and defects of existing technologies, this invention aims to eliminate redundant information in videos and reconstruct the temporal dependence of features based on image-text cross-modal embedding. Different architectures are used for different tasks to cope with the complexity of video scenes, so as to help intelligent agents effectively complete video-language understanding tasks. Specifically, this invention designs two independent neural network modules (video encoder and text query component), which contain three special components that work together to generate the final task target. The three components can be implemented by any appropriate method.

[0046] (2) This invention integrates multimodal information to achieve natural language understanding and inference of video content. Specifically, this invention obtains the effective encoding of the video through a video encoder and processes the text and video encoding in different ways through a text query component in order to cope with three different video-language understanding tasks: cross-modal retrieval, video description generation, and video question answering, thereby obtaining the corresponding results. Its advantage is that it effectively reduces the impact of complex video scenes on the results and improves the understanding ability of the model.

[0047] (3) This invention utilizes the clustering component and temporal component in the video encoder module to extract key blocks embedded in video frames and reconstruct time dependencies; it not only effectively reduces the redundancy of video information, but also maintains the time dependency of embedded features, further improving the robustness of the model.

[0048] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0049] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0050] Figure 1 This is a flowchart of the method in the first embodiment.

[0051] Figure 2 This is a flowchart of the overall processing in the first embodiment.

[0052] Figure 3 This is a flowchart of the text-video cross-modal retrieval task for the first embodiment.

[0053] Figure 4 The first embodiment is a flowchart describing the task generation process.

[0054] Figure 5 This is a flowchart of an open-ended question-answering task in the first embodiment of a video question-answering task.

[0055] Figure 6 This is a flowchart of a multiple-choice question-answering task in the video question-answering task of the first embodiment. Detailed Implementation

[0056] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0057] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0058] Terminology Explanation:

[0059] Self-Attention (SA): Also known as the self-attention mechanism, it is an attention mechanism used for processing sequence data. It is a core component of the Transformer model. The self-attention mechanism learns the relationships between different positions in the input sequence by mapping queries, keys, and values ​​to a high-dimensional space and calculating the similarity between them; this allows the model to encode, model, and capture long-distance dependencies based on the dependencies within the sequence.

[0060] Transformer: The Transformer model is built on the basis of the self-attention mechanism. It uses multi-head self-attention to learn the relationship between different positions in parallel, which improves the model's representation ability and generalization ability.

[0061] ViT: Uses the Transformer model to extract features and learn representations from image sequences, capturing the relationships between different regions in an image and effectively learning global contextual information. A ViT layer consists of multiple Transformer modules, each including multiple self-attention heads and feedforward neural networks; these modules are stacked to construct the entire ViT layer. Each module receives input from the previous layer and generates richer feature representations in its output.

[0062] ViT-ATM is a further improvement on the Vision Transformer (ViT) model, adding a temporal module to process video data sequences. Traditional ViT models can only process single static images and cannot model video sequences; while ViT-ATM extends the ViT model by introducing a temporal module, capturing the relationships between different frames in a video sequence and learning their temporal characteristics, thus providing a better understanding of video data.

[0063] k-medoids++ is an improved k-medoids clustering algorithm used to divide a dataset into k distinct clusters. It increases representativeness and diversity by using a probability distribution to select initial medoids, uses actual sample points as cluster centroids, is more robust to outliers, and can handle non-Euclidean distance metrics.

[0064] [cls]token: This is a special token used in Natural Language Processing (NLP). It is often used as a start token for input sequences and plays an important role in Transformer models and attention-based models.

[0065] Bi-directional Self Attention (BiSA) is a variant of an attention mechanism in machine learning that models dependencies between words in a sentence by simultaneously considering contextual information. It uses two separate attention mechanisms: forward attention and backward attention. In forward attention, each word is correlated with all the words preceding it to obtain forward contextual information. Similarly, in backward attention, each word is correlated with all the words following it to obtain backward contextual information. Ultimately, by merging the outputs of the forward and backward attention, a more comprehensive representation is obtained to better understand the relationships between words in a sentence.

[0066] Causal Self Attention (Causal SA) is a variant of the attention mechanism in machine learning, primarily used in sequence data processing tasks. It introduces the concept of causality to ensure that the model can only rely on previous information when making predictions. Causality is achieved by modifying the attention matrix so that each word can only be correlated with the words preceding it. In other words, the model can only predict the output at the current position based on the observed context, and cannot use future information.

[0067] Cross Attention (CA) is an attention mechanism that introduces associations between multiple input sequences. It achieves cross-sequence information transfer by calculating the relevance scores between the query sequence and the key sequence, and then using these scores as weights to perform a weighted summation of the value sequences.

[0068] Feed Forward Network (FFN): A common feedforward neural network structure in machine learning, which transforms input data through multiple fully connected layers and non-linear activation functions to extract features and enhance the representational power of the model.

[0069] BERT is a pre-trained language model that learns general sentence representations through self-supervised training on large-scale unlabeled text. Unlike traditional language models, BERT uses a Transformer architecture, which includes multiple self-attention layers.

[0070] Cosine similarity is a measure of the similarity between two vectors. It calculates the cosine of the angle between the two vectors, ranging from -1 to 1. When the cosine similarity is close to 1, it means that the two vectors are very similar in direction; when the cosine similarity is close to -1, it means that the two vectors are pointing in opposite directions; when the cosine similarity is close to 0, it means that there is no significant similarity between the two vectors.

[0071] This invention comprehensively considers the aforementioned three challenges and proposes an understanding model called RTQ, which consists of three key components, each addressing a specific challenge. First, the first component (clustering component) uses clustering methods to eliminate redundant information in adjacent video frames and select representative blocks. Next, the second component (temporal component) uses temporal modeling to perceive and interpret the temporal relationships between blocks, thereby avoiding processing for spatial consistency between representative blocks. Finally, the third component (text query component), comprising a text encoder, a video-based text encoder, and a video-based text decoder, progressively retrieves task-related information through language queries. These three components effectively address the aforementioned three challenges and significantly improve the agent's video-language understanding capabilities. These three components can be implemented using any appropriate method.

[0072] Example 1

[0073] In one or more embodiments, a video-language understanding method is disclosed, such as Figure 1 As shown, it includes:

[0074] Step 1: Obtain the video and text to be understood;

[0075] Step 2: Based on the trained understanding model, process the input video and text to generate the final understanding result;

[0076] The understanding model includes a video encoder and a text query component. The video encoder includes a clustering component and a temporal component. The clustering component is used to filter image blocks of all video frames in the video to obtain video frame embeddings after eliminating redundant information. The temporal dependency relationship between video frame embeddings is reconstructed through the temporal component to obtain a video feature matrix. Based on the video feature matrix, the text query component is used to embed text to obtain the final understanding result.

[0077] The understanding model can handle at least text-video cross-modal retrieval tasks, video description generation tasks, or video question answering tasks.

[0078] The implementation process of a video-language understanding method in this embodiment will be described in detail below.

[0079] This embodiment aims to design a novel deep learning model to learn video and text features, and improve the video-language understanding ability of an agent based on the task. It effectively addresses the three major challenges of information redundancy, time dependence, and scene complexity in the video-language understanding process, with the goal of completing three types of tasks: cross-modal text-to-video retrieval, video description generation, and video question answering. The overall processing flow is as follows: Figure 2 As shown, specifically:

[0080] First, given a video segment, each video frame is encoded through the Visual Transformer (ViT) layer in the video encoder to obtain semantically meaningful image block embeddings; then, a clustering component is used to retain representative blocks to eliminate redundant blocks.

[0081] The remaining blocks are then fed into the ViT-ATM component to capture the temporal dependencies between video frames and generate a video feature matrix as the output of the video encoder.

[0082] Finally, the text query component extracts task-related information layer by layer from the video feature matrix and text, and outputs the results corresponding to the task.

[0083] The above three components can be implemented by any appropriate method. This embodiment only describes one implementation method, which specifically includes the following steps:

[0084] S1: Establish a process for embedding video frames containing temporal relationships and eliminating redundant information using clustering components, and obtain the video features after eliminating redundant information, specifically:

[0085] S11: Utilize a K-layer ViT layer to generate semantically meaningful image patch embeddings from the video input to the video encoder. Where F is the number of video frames input to the model, 1 represents the [cis]token, and P in This represents the number of blocks into which each video frame is divided by the ViT layer, and d represents the hidden dimension.

[0086] S12: Before the Kth layer ViT, link the [cls] token to generate information tag embedding before video frame embedding. Then, this information is used to mark and execute SA to obtain the temporal dependencies between video frames:

[0087]

[0088] Embed the timing dependency into m k Along with other frame embeddings, they are fed into the Kth ViT layer to obtain image patch embeddings.

[0089] S13: Embed the video frames output from the Kth layer ViT layer into S segments, each segment containing F / S frames.

[0090] S14: Use the k-medoids++ clustering algorithm to analyze the F / S(1+P) clusters contained in each segment. in Clustering of image blocks produces (1+P) in ( ) clusters, selecting blocks located at the centroid of each cluster to form a video embedding after removing redundant information.

[0091] Clustering algorithms are not limited to the k-medoids++ clustering algorithm; other clustering methods can also be used.

[0092] S2: The video features after eliminating redundant information are reconstructed using a temporal component to maintain the temporal dependence of the video features.

[0093] Specifically, in the temporal component, an information tagging mechanism is used to reconstruct the temporal dependencies between video embeddings. Specifically, the temporal component uses an (LK) layer ViT-ATM layer, where L represents the total number of layers in the video encoder; the video embeddings obtained in S14 after removing redundant information, V... k Inputting temporal components results in a video embedding with temporal dependencies. It is used as the video feature matrix output by the video encoder.

[0094] S3: For three tasks—cross-modal retrieval, video description generation, and video question answering—a text query component is used to perform task-specific text embedding and obtain the final understanding results, including:

[0095] S31: After the previous two components, the video has been encoded into a time-aware representation with high information density, namely the video feature matrix; however, due to the complexity of the video scene, there is still a lot of information that is irrelevant to the task; in order to solve this problem, a text query component is introduced, which uses queries corresponding to specific tasks to gradually collect relevant details and generate the final understanding result.

[0096] The text query component integrates the L-layer MoED, which consists of four modules: BiSA, Causal SA, CA, and FFA. These four modules form three variants to complete the corresponding tasks.

[0097] The three variants of MoED specifically include:

[0098] (1) The text encoder, similar to BERT, encodes the text using BiSA and FFN at each layer and summarizes it by appending a [CLS] token to the beginning of the text input.

[0099] (2) A video-based text encoder collects task-relevant visual information through channel attention (CA) between a bidirectional self-attention mechanism (BiSA) and a feedforward neural network (FFN) in each layer of the text encoder. In channel attention, the text input is used as the query, and the video feature matrix is ​​used as the key and value. For task-specific purposes, an [Encode] tag is added to the text input, so that the resulting embedding contains a multimodal representation of the video-text pair.

[0100] (3) A video-based text decoder is responsible for collecting task-specific visual information to generate the required text output; the BiSA layer of the video-based text encoder is replaced with a Causal SA layer. A [Decode] marker is used to identify the start of a sequence, and a sequence end marker is used to identify the end of a sequence.

[0101] Using the three variants mentioned above, we demonstrate the video-language understanding capabilities of the agent for three tasks: text-video cross-modal retrieval, video description generation, and video question answering. Specifically, this includes:

[0102] (1) For the text-video cross-modal retrieval task, it is divided into two stages: recall and re-ranking.

[0103] like Figure 3 As shown, in the recall phase, the top Q video segments are recalled by calculating the cosine similarity between the video feature matrix and the [cls] tokens in the text embedding obtained from the text encoder. Then, in the reordering phase, the video-text query is input into the video-based text encoder, and the output [Encode] embedding is input into the fully connected layer and the sigmoid function to obtain the final score. The Q video segments are then reordered based on the score. The SA and FFD layers in the text encoder and the video-based text encoder share parameters.

[0104] (2) For video description generation tasks, such as Figure 4 As shown, the video feature matrix is ​​input into a video-based text decoder to generate descriptive text for the video.

[0105] (3) For video question answering tasks, including open-ended question answering tasks and multiple-choice question answering tasks.

[0106] For open-ended question-answering tasks, such as Figure 5 As shown, the video feature matrix and the question text are first input into a video-based text encoder to obtain a multimodal feature matrix. Then, the multimodal feature matrix is ​​input into a video-based text decoder to generate the answer. The encoder and decoder share parameters.

[0107] For multiple-choice question-answering tasks, treat them as classification tasks, such as... Figure 6 As shown, the question and answer are first concatenated into a complete sentence, and then input together with the video feature matrix into a video-based text encoder to encode the video and question-answer text pairs into a multimodal feature matrix; finally, the multimodal feature matrix is ​​input into a linear layer and a softmax layer to obtain the score of the best answer.

[0108] S4: Reasoning and Training, specifically including:

[0109] (1) For the text-video cross-modal retrieval task, jointly train the text encoder and the video-based text encoder.

[0110] For the text encoder, the Video-Text Contrast Loss (VTC) is used to encourage matching video-text pairs to have more similar [cls] labels than mismatched video-text pairs to align the feature spaces of the video and text.

[0111] First, for the i-th video-text pair, given their [CLS] embeddings, the video hidden vector is obtained by using a linear projection layer and an L2 normalization layer according to CLIP[5]. and text hidden vectors

[0112] To maximize the benefits of large-batch contrastive learning, three memory libraries are maintained to store the latest M video vectors. Text vector and the corresponding video

[0113] Then, the text-to-video contrast loss is calculated. Contrast loss between video and text

[0114]

[0115] in, It is a positive sample set, and τ is a learnable temperature parameter.

[0116] Finally, the two losses above are combined to obtain the VTC loss.

[0117]

[0118] To compensate for potential false negative samples in the momentum encoder, the dynamic gravity distillation strategy in ALBEF[6] is used to generate soft labels.

[0119] For video-based text encoders, a video-text matching loss (VTM) is used to learn a multimodal representation of video and text, capturing fine-grained alignment between the video and text. VTM corresponds to a binary classification task, where the model uses a VTM head (linear layer) to predict whether a video-text pair is positive (match) or negative (mismatch) given the multimodal features of the [Encode] label.

[0120] For the i-th video-text pair, first calculate their positive matching score. Then, a video / text sample is randomly selected to replace the video / text, resulting in a negative matching score. Finally, the video-text matching loss is obtained.

[0121]

[0122] To make the VTM loss provide more information, a hard negative sample mining strategy is used to sample negative samples.

[0123] Will and The sum of these results in the final loss.

[0124] (2) For the video description text generation task, the language modeling loss LM is used on the decoder, and the cross-entropy loss is optimized. The model is trained to maximize the probability of text in an autoregressive manner; for each video-text pair (v, t):

[0125]

[0126] Where L is the total length of the sentence; label smoothing of 0.1 is used when calculating the loss; compared with masked language modeling loss, which is widely used for video language pre-training, LM enables the model to generalize and can convert visual information into coherent captions.

[0127] (3) For video question answering tasks, open-ended question answering uses LM loss, while multiple-choice question answering uses VTM loss. Unlike text-to-video retrieval, negative samples come from incorrect question-answer pairs, rather than being generated using sampling.

[0128] This embodiment aims to improve the video-language understanding capabilities of intelligent agents. System analysis shows that current video-language understanding methods focus on limited aspects of the task, and methods for different challenges can complement each other. Therefore, a framework integrating redundancy removal, temporal modeling, and query components is proposed to jointly address information redundancy, temporal dependence, and scene complexity, respectively. Extensive experimental evaluation demonstrates the effectiveness and superiority of the proposed method. Future work aims to pre-train the model to help it acquire more knowledge and develop more effective redundancy removal, temporal modeling, and query components to improve the overall performance of the proposed method. By exploring the potential of this proposed method, it is hoped that it will contribute to the development of intelligent agent video-language understanding technology.

[0129] Example 2

[0130] In one or more embodiments, a video-language understanding system is disclosed, including an acquisition module and a understanding module:

[0131] The acquisition module is configured to acquire the video and text to be understood.

[0132] An understanding module, configured to: process the input video and text based on a trained understanding model to generate a final understanding result;

[0133] Wherein, the understanding model includes a video encoder and a text query component, and the video encoder includes a clustering component and a temporal component; the clustering component is used to screen image blocks of all video frames in the video to obtain video frame embeddings after eliminating redundant information; the temporal component is used to reconstruct the temporal dependence relationship between the video frame embeddings to obtain a video feature matrix; based on the video feature matrix, text embedding is performed through the text query component to obtain a final understanding result;

[0134] The understanding model can at least handle text-video cross-modal retrieval tasks, video description generation tasks or video question answering tasks.

[0135] Embodiment III

[0136] The purpose of this embodiment is to provide a computer-readable storage medium.

[0137] A computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the steps in a video-language understanding method as described in Embodiment I of the present disclosure.

[0138] Embodiment IV

[0139] The purpose of this embodiment is to provide an electronic device.

[0140] An electronic device, including a memory, a processor, and a program stored on the memory and executable on the processor, and when the processor executes the program, it implements the steps in a video-language understanding method as described in Embodiment I of the present disclosure.

[0141] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention may have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A video-language understanding method, characterized in that, include: Obtain the video and text to be understood; Based on the trained understanding model, the input video and text are processed to generate the final understanding result; The understanding model includes a video encoder and a text query component. The video encoder includes a clustering component and a temporal component. The clustering component is used to filter image blocks of all video frames in the video to obtain video frame embeddings after eliminating redundant information. The temporal dependency relationship between video frame embeddings is reconstructed through the temporal component to obtain a video feature matrix. Based on the video feature matrix, the text query component is used to embed text to obtain the final understanding result. The understanding model can handle at least text-video cross-modal retrieval tasks, video description generation tasks, or video question answering tasks; In the temporal component, an information tagging mechanism is used to reconstruct the temporal dependencies between video frame embeddings, specifically as follows: Embed the video frames after removing redundant information Inputting the ViT-ATM layer yields a video feature matrix with temporal dependencies. The ViT-ATM layer is a further improvement on the Vision Transformer model, adding a temporal module to process video data sequences. The timing module is used to capture the relationships between different frames in a video sequence and learn their temporal features, specifically: In the Before layer ViT, link the [cls] token to generate information tag embedding before embedding the video frame. Then, this information is used to mark and perform SA to obtain the temporal dependencies between video frames: Embed timing dependencies Send along with other frame embeddings to the first Image patch embeddings are obtained from the ViT layer. .

2. The video-language understanding method as described in claim 1, characterized in that, The video encoder also includes a ViT layer; The video encoder includes a ViT layer, a clustering component, and a timing component connected in sequence; The ViT layer is used to encode each video frame in blocks to obtain semantically meaningful image block embeddings. Image block embeddings belonging to the same video frame constitute video frame embeddings.

3. The video-language understanding method as described in claim 2, characterized in that, The process of using a clustering component to filter image blocks across all video frames specifically involves: The video frames output from the ViT layer are embedded into groups. Each paragraph contains frame; Use clustering algorithms to analyze the data contained in each segment. Clustering of block image blocks produces For each cluster, a block located at the centroid of each cluster is selected to form a video embedding after redundancy removal. ; in, Indicates the number of segments. F Indicates the number of video frames. This indicates the number of blocks in each video frame.

4. The video-language understanding method as described in claim 1, characterized in that, The text query component integrates... layer , It consists of four modules: Bidirectional Self-Attention Mechanism (BiSA), Causal SA, Channel Attention (CA), and Feedforward Neural Network (FFA). These four modules form three variants to complete the corresponding task-specific text embedding: text encoder, video-based text encoder, and video-based text decoder. Causal SA introduces the concept of causality to ensure that the model can only rely on previous information when making predictions. By modifying the attention matrix, each word can only be correlated with the words before it, thus achieving causality. For text-video cross-modal retrieval tasks, the similarity between the video feature matrix and the text feature matrix obtained by the text encoder is calculated to obtain a set of videos that meet the similarity conditions. For the video description generation task, the video feature matrix is ​​input into the video-based text decoder to generate text, which serves as the video description text. For video question answering tasks, the video feature matrix and the question text are input into a video-based text encoder to obtain a multimodal feature matrix. Then, the multimodal feature matrix is ​​input into a video-based text decoder to generate text, which serves as the answer to the question.

5. The video-language understanding method as described in claim 4, characterized in that, The text encoder encodes the input text using BiSA and FFN at each layer, appends a [CLS] token to the beginning of the text input, and outputs a text feature matrix. The video-based text encoder collects task-related visual information by adding CA between BiSA and FFN in each layer of the text encoder; in CA, the input text is used as a query, and the embedded video feature matrix is ​​used as a key and value to generate a multimodal feature matrix. The video-based text decoder replaces the BiSA layer of the video-based text encoder with a Causal SA layer to decode the input multimodal feature matrix into text.

6. The video-language understanding method as described in claim 1, characterized in that, The text-video cross-modal retrieval task is divided into two stages: recall and reordering. In the recall phase, the Top Q video segments are recalled by calculating the cosine similarity between the [cls] tokens in the video feature matrix and the text feature matrix obtained by the text encoder. In the reordering stage, the video-text query is input into the video-based text encoder, and the output [Encode] is embedded into the input fully connected layer and the sigmoid function to obtain the final score. The Q segments of video are reordered according to the score, and a preset number of videos are selected to form a video set.

7. The video-language understanding method as described in claim 1, characterized in that, The video question answering task also includes a multiple-choice question answering task, which is treated as a classification task. The question and answer are concatenated into a complete sentence, and then input together with the video feature matrix into a video-based text encoder to encode the video and question-answer text pairs into a multimodal feature matrix. Finally, the multimodal feature matrix is ​​input into a linear layer and a softmax layer to obtain the score of the best answer.

8. The video-language understanding method as described in claim 1, characterized in that, The training of the understanding model specifically involves: For the text encoder, video-text contrastive loss is used to encourage matching video-text pairs to have more similar [cls] tags than mismatched video-text pairs to align the feature spaces of video and text. For video-based text encoders, video-text matching loss is used to learn a multimodal representation of video and text, capturing fine-grained alignment between video and text; For a video-based text decoder, a language modeling loss is used, and the cross-entropy loss is optimized to train the model to maximize the probability of the text in an autoregressive manner.

9. A video-language understanding system, characterized in that, Includes an acquisition module and an understanding module: The acquisition module is configured to acquire the video and text to be understood. The understanding module is configured to process the input video and text based on a trained understanding model and generate the final understanding result. The understanding model includes a video encoder and a text query component. The video encoder includes a clustering component and a temporal component. The clustering component is used to filter image blocks of all video frames in the video to obtain video frame embeddings after eliminating redundant information. The temporal dependency relationship between video frame embeddings is reconstructed through the temporal component to obtain the video feature matrix. Based on the video feature matrix, text is embedded through a text query component to obtain the final understanding result; The understanding model can handle at least text-video cross-modal retrieval tasks, video description generation tasks, or video question answering tasks; In the temporal component, an information tagging mechanism is used to reconstruct the temporal dependencies between video frame embeddings, specifically as follows: Embed the video frames after removing redundant information Inputting the ViT-ATM layer yields a video feature matrix with temporal dependencies. The ViT-ATM layer is a further improvement on the Vision Transformer model, adding a temporal module to process video data sequences. The timing module is used to capture the relationships between different frames in a video sequence and learn their temporal features, specifically: In the Before layer ViT, link the [cls] token to generate information tag embedding before embedding the video frame. Then, this information is used to mark and perform SA to obtain the temporal dependencies between video frames: Embed timing dependencies Send along with other frame embeddings to the first Image patch embeddings are obtained from the ViT layer. .

Images