Video text processing method, model training method and related equipment

Abstract

The embodiment of the invention discloses a video text processing method, a model training method and related equipment. The method comprises the following steps: acquiring a to-be-processed video and a query text; processing the to-be-processed video and the query text through the language model to obtain video frame features of video frames in the to-be-processed video and prediction reply text features; extracting time identification features contained in the predicted reply text features; matching the time identification feature of the predicted reply text feature with the video frame features to determine a target video frame feature from the video frame features; and determining time information in a predicted reply text corresponding to the predicted reply text feature according to the target video frame feature. By adopting the method and the device, the prediction accuracy of predicting the time information in the reply text can be improved. The scheme provided by the embodiment of the invention relates to artificial intelligence and large model technologies.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a video text processing method, a model training method, a video text processing apparatus, a model training apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Pre-trained models (PTMs), also known as foundational models or large models, refer to deep neural networks (DNNs) with a large number of parameters. These DNNs are trained on massive amounts of unlabeled data. Leveraging the function approximation capabilities of large-parameter DNNs, PTMs extract common features from the data. Through fine-tuning, efficient parameter fine-tuning (PEFT), and prompt-tuning techniques, they are suitable for downstream tasks. Therefore, pre-trained models can achieve ideal results in scenarios with few or zero samples.

[0003] Based on the data modalities they process, PTMs can be categorized into language models (e.g., ELMO, BERT, GPT), visual models (e.g., Swin-transformer, ViT, V-MOE), speech models (e.g., VALL-E), and multimodal models (e.g., ViBERT, CLIP, Flamingo, Gato). Multimodal models refer to models that establish feature representations for two or more data modalities. Pre-trained models are important tools for outputting Artificial Intelligence Generated Content (AIGC) and can also serve as a general interface connecting multiple task-specific models.

[0004] However, the language models in related technologies have flaws in their methods for locating time information, which limits the models' understanding and generation of timestamps. Summary of the Invention

[0005] This disclosure provides a video text processing method, a model training method, a video text processing apparatus, a model training apparatus, a computer device, a computer-readable storage medium, and a computer program product, which can improve the prediction accuracy of time information in predicted response text, thereby improving the language model's understanding and generation of timestamps.

[0006] This disclosure provides a video text processing method, which includes: acquiring a video to be processed and query text; processing the video to be processed and the query text using a language model to obtain video frame features and predicted response text features of video frames in the video to be processed; extracting time identifier features contained in the predicted response text features; matching the time identifier features of the predicted response text features with the video frame features to determine target video frame features from the video frame features; and determining time information in the predicted response text corresponding to the predicted response text features based on the target video frame features.

[0007] This disclosure provides a model training method, comprising: acquiring sample video, sample query text, labeled response text, and labeled time information in the labeled response text; processing the sample video and sample query text using a language model to obtain sample frame features and predicted sample response text features of the sample video; extracting time identifier features contained in the predicted sample response text features; matching the time identifier features of the predicted sample response text features with the sample frame features to determine target sample frame features from the sample frame features; obtaining prediction time information in the predicted sample response text corresponding to the predicted sample response text features based on the target sample frame features; obtaining a first loss function based on the labeled response text and the predicted sample response text, and obtaining a second loss function based on the labeled time information and the predicted time information, so as to train the language model based on the first loss function and the second loss function.

[0008] This disclosure provides a video text processing apparatus, comprising: a receiving unit for acquiring a video to be processed and query text; a processing unit for processing the video to be processed and the query text using a language model to obtain video frame features of video frames in the video to be processed and predicted response text features of the query text; the processing unit is further configured to extract time signature features contained in the predicted response text features; the processing unit is further configured to match the time signature features of the predicted response text features with the video frame features to determine target video frame features from the video frame features; the processing unit is further configured to determine time information in the predicted response text corresponding to the predicted response text features based on the target video frame features.

[0009] This disclosure provides a model training apparatus, comprising: a receiving unit for acquiring a video to be processed and query text; a processing unit for processing the video to be processed and the query text using a language model to obtain video frame features of video frames in the video to be processed and predicted response text features of the query text; the processing unit is further configured to extract time signature features contained in the predicted response text features; the processing unit is further configured to match the time signature features of the predicted response text features with the video frame features to determine target video frame features from the video frame features, and to train the language model based on a first loss function and a second loss function.

[0010] This disclosure provides a computer device including a processor, a memory, and an input / output interface. The processor is connected to both the memory and the input / output interface. The input / output interface is used to receive and output data. The memory is used to store a computer program. The processor is used to invoke the computer program to cause the computer device containing the processor to execute the method in any embodiment of this disclosure.

[0011] This disclosure provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, such that a computer device having the processor performs the methods of any embodiment of this disclosure.

[0012] This disclosure provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various alternative embodiments of this disclosure. Attached Figure Description

[0013] Figure 1 This is a flowchart of a video text processing method provided in an embodiment of this disclosure.

[0014] Figure 2 This is a schematic diagram of a method for locating time in related technologies.

[0015] Figure 3 This is a schematic diagram of the structure of a model provided in an embodiment of this disclosure.

[0016] Figure 4 This is a flowchart of a model training method provided in an embodiment of this disclosure.

[0017] Figure 5This is a schematic diagram of a video under a RAR (Referred Action Recognition) task provided in an embodiment of this disclosure.

[0018] Figure 6 This is a schematic diagram of a video under an ECA (Event-Caption Alignment) task provided in an embodiment of this disclosure.

[0019] Figure 7 This is a schematic diagram of a video under an RVQ (Referred Video Question-Answering) task provided in an embodiment of this disclosure.

[0020] Figure 8 This is a schematic diagram of a video under a TVG (Temporal Video Grounding) task provided in an embodiment of this disclosure.

[0021] Figure 9 This is a schematic diagram of a video under an EPM (Episodic Memory) task provided in an embodiment of this disclosure.

[0022] Figure 10 This is a schematic diagram of a video under a TAL (Temporal Action Localization) task provided in an embodiment of this disclosure.

[0023] Figure 11 This is a schematic diagram of a video under an EVS (Extractive Video Summarization) task provided in an embodiment of this disclosure.

[0024] Figure 12 This is a schematic diagram of a video under a VHD (Video Highlight Detection) task provided in an embodiment of this disclosure.

[0025] Figure 13 This is a schematic diagram of a video under a DVC (Dense Video Captioning) task provided in an embodiment of this disclosure.

[0026] Figure 14 This is a schematic diagram of a video under an SLC (Step Localization and Captioning) task provided in an embodiment of this disclosure.

[0027] Figure 15 This is a schematic diagram of a video under a TEM (Temporal Event Matching) task provided in an embodiment of this disclosure.

[0028] Figure 16 This is a schematic diagram of a video under a GVQ (Grounded Video Question-Answering) task provided in an embodiment of this disclosure.

[0029] Figure 17 This is a schematic diagram of a task classification method and sample distribution provided in an embodiment of this disclosure.

[0030] Figure 18 This is a schematic diagram showing the distribution of the average video duration (in seconds) of the 12 tasks provided in this embodiment of the disclosure.

[0031] Figure 19 This is a schematic diagram of a video text processing device provided in an embodiment of the present disclosure.

[0032] Figure 20 This is a schematic diagram of a model training device provided in an embodiment of this disclosure.

[0033] Figure 21 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0034] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0035] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0036] When applying the data collection and processing disclosed herein, it is essential to strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the data subject, and conduct subsequent data use and processing within the scope authorized by laws and regulations and the data subject.

[0037] First, some of the terms used in the embodiments of this disclosure will be explained.

[0038] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0039] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0040] Computer Vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and further processes images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision technology. Pre-trained models in the vision field, such as Swin-transformer, ViT, V-MOE, and MAE, can be quickly and widely applied to downstream tasks after fine-tuning. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0041] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP deals with natural language, the language people use in daily life, and is closely related to linguistics; it also involves computer science and mathematics. Pre-trained models, a crucial technique for model training in artificial intelligence, evolved from Large Language Models (LLMs) in NLP. After fine-tuning, LLMs can be widely applied to downstream tasks. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0042] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and pre-trained learning. Pre-trained models are the latest development in deep learning, integrating all of these techniques.

[0043] Transformer: A network structure for parallel processing of sequential tasks.

[0044] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, digital twins, virtual humans, robots, AI-generated content (AIGC), conversational interaction, smart healthcare, smart customer service, and game AI. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0045] The solutions provided in this disclosure involve technologies such as computer vision, natural language processing, and machine learning in artificial intelligence, and are specifically illustrated through the following embodiments.

[0046] Figure 1 This is a flowchart of a video text processing method provided in an embodiment of this disclosure. Figure 1The steps in the method provided in the embodiments can be executed by any computer device. This example illustrates execution by any server / terminal, but this disclosure is not limited thereto. Figure 1 As shown, the method provided in this disclosure embodiment may include the following steps.

[0047] In S110, the video to be processed and the query text are obtained.

[0048] In this disclosure, the video to be processed refers to a video that needs to be processed by a language model to understand the events, actions, characters, or objects contained within it. The query text refers to the instruction or question that needs to be answered regarding the video to be processed. Since the query text is input into the language model along with the video to be processed, it is sometimes referred to as "input" in the following text. When the input query text is a question, it can also be called a question; when the input query text tells the language model the purpose it needs to achieve by understanding the video to be processed, it can also be called an instruction. This disclosure does not limit the source of the video to be processed and the query text. For example, the video to be processed could be a video being played on a user's terminal, and the query text could be a question posed by the user regarding the playing video.

[0049] In this disclosure, an event refers to a change in a state or an event that occurs at a specific point in time or within a specific geographical area and consists of one or more actions performed by one or more actors.

[0050] In S120, the language model is used to process the video to be processed and the query text to obtain the video frame features of the video frames in the video to be processed and the predicted response text features.

[0051] The language model in this embodiment can be a type of large model, and therefore can also be called a large language model, or LLM.

[0052] The video frame features in this embodiment refer to obtaining video frames in the video to be processed by sampling the video to be processed, and then obtaining the video frame features of the corresponding video frames by performing feature extraction or feature encoding on the sampled video frames.

[0053] In this embodiment of the disclosure, the predicted response text refers to the answer, response, or output extracted by the language model from the video to be processed, which is used to respond to the question or instruction posed in the query text. The predicted response text refers to the text features corresponding to the predicted response text output by the language model. Based on these features, the corresponding predicted response text can be obtained. This predicted response text can then be displayed to the user / terminal that submitted the query text.

[0054] In an exemplary embodiment, a language model is used to process the video to be processed and the query text to obtain video frame features and predicted response text features of the video frames in the video to be processed. This includes: sampling video frames of the video to be processed at time t, where t is a positive integer greater than or equal to 1 and less than or equal to T, and T is a positive integer greater than or equal to 1; processing the video frames at time t through a visual encoder to obtain the block vector of the video frames at time t; processing the block vector of the video frames at time t and the query text through a frame compressor to obtain the compressed frame features of the video frames at time t; processing the query text through a text encoder to obtain the text features of the query text; and processing the concatenated compressed frame features and text features from time 1 to T through a language model to obtain video frame features and predicted response text features.

[0055] Here, it is assumed that after sampling the video to be processed, a total of T frames / video frames were sampled, and t in the following representation represents the t-th frame or the t-th video frame among these T frames. In this embodiment, a frame compressor compresses the block vectors of the video frames at each time step into a compressed frame feature guided by the query text. On the one hand, this reduces the redundant computation of each video frame; on the other hand, it ensures that the compressed frame feature retains as many features as possible that are most relevant to the query text. Simultaneously, by representing each video frame with a compressed frame feature, more video frames can be obtained when using the LLM model for prediction, thereby improving the temporal resolution and accuracy of the prediction. An exemplary structure and processing procedure for a video LLM including a visual encoder, a text encoder, a frame compressor, and an LLM can be specifically referred to below. Figure 2 Description of the embodiments.

[0056] In an exemplary embodiment, the concatenated compressed frame features and text features from times 1 to T are processed by a language model to obtain video frame features and predicted response text features, including: extracting time text from the query text; replacing the part of the text features corresponding to the time text with a time identifier; obtaining target text features based on the time identifier in the text features and the corresponding compressed frame features; and processing the concatenated compressed frame features and target text features from times 1 to T by a language model to obtain video frame features and predicted response text features of the video to be processed.

[0057] In some embodiments, the query text input by the user may contain time information (here referred to as time text for distinction). For example, the input time information is used to instruct the LLM model to extract content for answering questions within the time point or time period indicated by the time information. In this case, the position in the text feature corresponding to the time text can be replaced with a preset time identifier. The time identifier in this embodiment can be any identifier that can be used to indicate that the feature at the corresponding position is time information; hereinafter, it will be referred to as time information. <vid>This example uses a time identifier, but this disclosure is not limited to this. For specific handling of cases where the input query text contains time text, please refer to the following. Figure 2 Description in the embodiments.

[0058] In an exemplary embodiment, the compressed frame features of the video frame at time t are obtained by processing the block vector and query text of the video frame at time t through a frame compressor, including: processing the block vector and query text of the video frame at time t through a transformer in the frame compressor to obtain the learned query vector at time t; processing the block vector and the learned query vector of the video frame at time t through a context attention module (CAM) in the frame compressor to obtain the compressed frame vector of the video frame at time t; and projecting the compressed frame vector of the video frame at time t onto the same vector space as the language model to obtain the compressed frame features of the video frame at time t.

[0059] In some embodiments, the transformer in the frame compressor can be a lightweight transformer that uses a learnable query vector set to extract learned query vectors from a frozen visual encoder. These learned query vectors are visual features extracted from the video frame at time t. The lightweight transformer can learn the most relevant visual representation / visual features to the query text from the video frame at time t, and this representation can also be interpreted by an LLM. The following examples use the Q-Former as an example, but this disclosure is not limited thereto.

[0060] The context attention module (CAM) in this embodiment is an attention mechanism for processing image or sequence data (including block vectors of video frames at time t and learned query vectors at time t). It allows the model to assign different attention weights based on the contextual information of the input data when generating output. On the one hand, the CAM enhances the model's perception of contextual information; it captures contextual information in the input data, enabling the model to consider more background knowledge when generating output. This is particularly important for tasks requiring a deep understanding of the input data (such as natural language processing). On the other hand, by assigning different attention weights to different input elements, the CAM helps the model focus more accurately on important information in the input, thereby improving the model's prediction accuracy. Furthermore, because the CAM considers the contextual information of the input data, it is more robust to noise and outliers in the input data. Even if there are errors or noise in the input data, the CAM can still guide the model to make correct predictions. The CAM calculates the correlation or similarity between different elements in the input data and then assigns attention weights based on these correlations or similarities.

[0061] In an exemplary embodiment, the cascaded compressed frame features and text features from time 1 to T are processed by a language model to obtain video frame features and predicted response text features of the video to be processed. This includes: processing the compressed frame features from time 1 to T through a bi-directional self-attention module in the language model to obtain video frames of the video to be processed; and processing the text features through a causal self-attention module in the language model to obtain predicted response text features.

[0062] Bidirectional self-attention is an attention mechanism in deep learning, primarily used for processing sequential data, especially in Natural Language Processing (NLP) tasks. When processing each element in a sequence (e.g., compressed frame features from time 1 to T), it simultaneously considers both the forward and backward contextual information of that element. This mechanism significantly improves the model's understanding of the overall structure of the sequence and helps capture long-distance dependencies. The bidirectional self-attention module simultaneously computes forward and backward attention weights for each element in the sequence. Forward attention weights are based on the current element (e.g., compressed frame features at time t) and the sequence information preceding it (e.g., compressed frame features from time 1 to t-1), while backward attention weights are based on the current element and the sequence information following it (e.g., compressed frame features from time t+1 to T). In the bidirectional self-attention module, some form of similarity function (e.g., dot product, cosine similarity, etc.) can be used to calculate the correlation between the current element and other elements in the sequence. Based on these correlation scores, the model can generate an attention weight vector for each element in the sequence, reflecting the relative importance of that element compared to other elements in the sequence. Bidirectional self-attention modules, by directly calculating the correlation between any two elements in a sequence, can more easily capture long-distance dependencies, thereby improving the model's understanding of the overall sequence structure. Simultaneously, bidirectional self-attention modules can process all elements in the sequence in parallel, significantly improving computational efficiency. This makes bidirectional self-attention modules more scalable when handling large-scale datasets.

[0063] A causal self-attention module (CSI) is an attention mechanism used in deep learning models, particularly in Natural Language Processing (NLP), to process sequential data (such as text features). It is designed primarily for tasks that require consideration of previous information within the sequence (e.g., assuming the current element is a text feature at time t, its previous information includes text features from time 1 to t-1). CSI is a unidirectional attention mechanism; the model can only focus on the element and its preceding elements, not future elements. This design aligns with the needs of many sequence generation tasks, where the model's output is typically generated based on the current and previous inputs. CSI effectively avoids this information leakage through its unidirectional design. When processing sequences, CSI can compute the attention weights of all elements in the sequence on other elements in parallel, making it highly efficient for long sequences. The CSI module consists of three steps: calculating similarity, calculating weights, and weighted summation. For each element in the sequence, the model calculates its similarity to other elements (including only previous elements), then calculates weights based on these similarities, and sums these elements according to their weights to obtain a representation that takes into account the information of all previous elements.

[0064] In S130, the time signature features contained in the predicted response text features are extracted.

[0065] In an exemplary embodiment, extracting time signature features contained in the predicted response text features includes: extracting the hidden states of the time signatures in the predicted response text at layer L-1 in the language model, and using these as the time signature features of the predicted response text features. Here, L is the layer number of the language model, and L is a positive integer greater than or equal to 1.

[0066] In some embodiments, the hidden state of the time stamp in layer L-1 of the LLM model can be extracted as the time stamp feature. That is, assuming the number of layers in the LLM model is L, the hidden state of the second-to-last layer is extracted as the time stamp feature for matching with video frame features. This retains more features and improves the accuracy of matching. However, this disclosure is not limited to this. In other embodiments, the hidden state of the time stamp in layer L (the last layer) of the LLM model can also be extracted as the time stamp feature.

[0067] In S140, the time signature features of the predicted response text features are matched with the video frame features to determine the target video frame features from the video frame features.

[0068] In an exemplary embodiment, matching the time signature feature of the predicted response text feature with the video frame feature to determine the target video frame feature from the video frame feature includes: aligning the time signature feature of the predicted response text feature and the video frame feature to obtain aligned time signature features of the predicted response text feature and video frame feature; calculating the similarity between the aligned time signature feature of the predicted response text feature and the video frame feature; and using the video frame feature that meets the similarity condition as the target video frame feature.

[0069] In this embodiment of the disclosure, the target video frame feature refers to the video frame feature corresponding to the time identifier feature among the T-frame video frames used in the video to be processed, whose similarity to the time identifier feature meets the condition (e.g., the highest similarity). After the aforementioned processing, the time identifier feature predicted and output by the LLM model includes the video feature corresponding to the time information most relevant to the query text in the video to be processed. Therefore, after the matching process, the target frame index or time position of the target video frame most similar to the time identifier feature in the video to be processed is the time information in the predicted response text.

[0070] In S150, the time information in the predicted response text corresponding to the predicted response text features is determined based on the target video frame features.

[0071] In an exemplary embodiment, determining the time information in the predicted response text corresponding to the predicted response text features based on the target video frame features includes: obtaining the target frame index of the target video frame features; and determining the time information in the predicted response text based on the target frame index and the frame rate of the video to be processed.

[0072] For example, multiplying the target frame index by the frame rate yields the time information of the target video frame in the video to be processed, in seconds (s).

[0073] The video text processing method provided in some embodiments of this disclosure, when processing the video to be processed and the corresponding query text through a language model to obtain video frame features and predicted response text features, extracts time stamp features contained in the predicted response text features, and uses these time stamp features to match the video frame features of the video to be processed, so as to determine the target video frame features that match the time stamp features from the video frame features. Based on the frame number or time of the target video frame corresponding to the target video frame features in the video to be processed, the time information in the predicted response text predicted by the language model can be accurately located, thereby improving the prediction accuracy of the time information in the predicted response text, and thus improving the language model's understanding and generation of timestamps.

[0074] Figure 2 This is a schematic diagram of a method for locating time in related technologies.

[0075] like Figure 2 As shown, assuming the input video and instructions are identical, the instruction is to find the moment in the input video when "a dog is eating cake". In related technologies, LLM processes timestamps in videos or coordinates in images by (a) directly representing them in text form, (b) introducing special tokens to specifically represent time, or (c) using external modules.

[0076] Representing numbers directly as text loses the continuity between them. Because text cannot express the continuity of time, the model cannot understand the relationships such as the size of the individual numbers.

[0077] The special token approach quantizes time / location (or coordinates) into a fixed number (e.g., 100 to 300) of special tokens by defining a set of special tokens. This approach inevitably incurs significant quantization loss and is inflexible for videos of variable length. The special token format inevitably quantizes time; for example, using only 100 special tokens would divide each video into 100 equal parts, with each token corresponding to a short segment of time, resulting in coarse temporal granularity for long videos. Furthermore, an incorrect prediction can be far from reality. For instance, in this paradigm, all numbers are divided into individual tokens from 0 to 9. For example, 103.4 is represented in the model as "1", "0", "3", ".", and "4", a total of five tokens. There is no distinction in size between the tokens corresponding to different numbers; the model cannot differentiate between "1" and "0," only recognizing them as two different characters. Therefore, to output 103.4, the model would need to output these tokens five times consecutively and correctly. This approach cannot guarantee the continuity of time and is prone to errors during output. For example, if "1" is mistakenly output as "2" in the first step, the entire predicted time will deviate significantly from the correct time. Furthermore, introducing too many new tokens into the vocabulary will disrupt the pre-training distribution of LLM, making it difficult to optimize without pre-training. In other words, discretely predicting the next token has inherent flaws in numerical computation tasks, limiting the understanding and generation of timestamps.

[0078] External module approaches utilize pre-trained external models (such as SAM) for localization. This introduces additional parameters and latency into LLM. It is also not directly compatible with video because the temporal basis models in related techniques are domain-specific and difficult to generalize to all scenarios such as SAM.

[0079] This disclosure reformulates timestamp prediction as embedding matching, employing a new paradigm to avoid directly predicting time. Whenever time needs to be output, the model directly outputs the feature corresponding to a certain video frame. This feature is then used to compare with each frame of the video (e.g., the video to be processed) (i.e., each video frame feature). The frame with the highest similarity (corresponding to the time) is the time that the model wants to express.

[0080] Figure 3 The model provided in this disclosure embodiment is referred to as the ETChat model. It is understood that... Figure 3 The given model structure is for illustrative purposes only, and this disclosure is not limited thereto. Figure 3 This is the overall architecture of the ETChat model proposed in this disclosure. This disclosure uses the prediction or location of timestamps (i.e., time information) as the prediction for embedding matching.

[0081] like Figure 3 As shown, the ETChat model includes a visual encoder (Visual Encoder E). v 30. Frame Compressor c The system consists of a tokenizer (40), an LLM (Low-Low Mechanism) module (50), a first projector (70), and a second projector (80). The LLM module (60) may include a bidirectional self-attention module (61) and a causal self-attention module (62). Assuming the input query text (20) or question is "Where did the puppy's cake come from between 35s and 42s?", where "35s to 42s" represents the time information in the query text, the tokenizer (50) processes the query text (20) and outputs text tokens. In other words, N in the query text is a sequence of text features, where N is a positive integer greater than or equal to 1, and N represents the number of text features. This represents the nth text feature, where n is a positive integer greater than or equal to 1 and less than or equal to N. The structures of the visual encoder and text encoder are not limited in this embodiment.

[0082] Figure 3 The term "frozen" in this context refers to the visual encoder E during the fine-tuning training process of the model. v The model parameters for 30 and the text encoder (Tokenizer) 50 are frozen and do not change. "Learnable" refers to the frame compressor (E... c 40. The model parameters of the first projector 70 and the second projector 80 are adjustable during training. LLM is fine-tuned, for example, using the LoRA technique.

[0083] refer to Figure 3 The input video (in the inference phase, it is the video to be processed; in the training phase, it is the sample video; this example uses the inference phase, and the training phase can be similarly referenced) 10. At time t, a video frame is obtained by sampling this video. Where t∈T, W and H are the height and width of the video frame, respectively, where W and H are both positive integers greater than or equal to 1, and 3 represents three channel components, such as RGB. First, the frozen visual encoder E is used... v 30. Convert the video frame into patch embeddings. Here, K and C represent the number of blocks and feature dimensions of the video frame, respectively, and both K and C are positive integers greater than or equal to 1.

[0084] In this embodiment, the input video frame can first be uniformly scaled and cropped to 224*224*3 pixels, i.e., assuming H=W=224 (this is only for illustrative purposes and is not limited to this). Then, the video frame is converted into blocks of size 14*14 pixels, and the size of each block is E of the visual encoder. v 30. In the training settings, the pre-trained visual encoder E was used directly. v 30, alignment can be maintained. However, this disclosure is not limited to this, when the visual encoder E v When the size of the block set during training changes, a corresponding change will occur.

[0085] To maintain high temporal resolution while reducing redundant computations, embodiments of this disclosure employ a frame compressor E. c 40. Merge the block vectors and then project them onto a single token. Where D is the embedding dimension of the LLM, and D is a positive integer greater than or equal to 1, representing a single token. This refers to the compressed frame features of the video frame at time t. By compressing each video frame into a single token using a frame compressor, compared to the schemes of most related technologies (256 tokens per frame), the computational load can be significantly reduced, and more frames can be sampled to ensure temporal resolution.

[0086] Then, compress the frame tokens. (i.e., the sequence of compressed frame features from time 1 to time T) and text tokens The data is concatenated and sent to LLM 60 to generate response 90, which is the predicted response text, such as "It was provided by the owner at 26s". Here, "26s" refers to the time information or timestamp in the predicted response text.

[0087] Within the LLM 60, frame tokens (video tokens, i.e., compressed frame features from time 1 to T) 41 and text tokens (text tokens, i.e., 1 to N text features) 42 are distinguished in the concatenated input sequence. A bidirectional self-attention module 61 is used to model the relationships between frame tokens 41, while a unidirectional causal self-attention module 62 is used to model the text tokens 42. The output is a new sequence of frame tokens, identical in form to the original frame tokens, but each frame token has been updated. Bidirectional means that each frame token 41 can see all other frame tokens 41, while unidirectional means that each text token 42 can only fuse itself with the previous text tokens 42 (corresponding to...). Figure 3 The output of the text token 42 in the upper middle is a new text token sequence, which is the same as the original text token, but each text token has been updated.

[0088] Figure 3 In this example, the question "where did the puppy's cake come from at 35s-42s?" is directly input into the frame compressor E. c 40, instead of inputting the text token output by Tokenizer 50 into the frame compressor E. c 40, because of the frame compressor E c 40 has its own internal Tokenizer, which is incompatible with the external Tokenizer 50 and needs to be handled separately.

[0089] The following section introduces frame compression. For example, similar to the context generation process, the frame compressor E... c 40 includes Q-FormerE q Contextual attention module E att And MLP (Multi-Layer Perceptron) projector E p Among them, Q-FormerE q There are M learnable query vectors, where M is a positive integer greater than or equal to 1. For each time step, Q-FormerE... q Receive block vector P t and text prompts (e.g.) Figure 3 The query vector ("where did the puppy's cake come from at 35s-42s?") is taken as input and resampled into a learnable query vector (i.e., the learned query vector at time t).

[0090] Then, E att In Q t Under the condition of P t Compressed into a single token, E p Project it into the same vector space as the LLM:

[0091]

[0092] in, It is the compressed frame token at time t (i.e., the compressed frame vector of the video frame at time t), which includes text-conditioned visual information. mean means to calculate the average, and the softmax function is a mathematical function used in deep learning and machine learning. Indicates that for P t Find the transpose of the matrix. Equations (1) and (2) can be performed in parallel across all video frames. Then, the projected video token sequence e is used. v With text token e q Cascaded to form an LLM with an input of (T+N)×D.

[0093] Video LLMs represent a class of intelligent chatbots capable of understanding video and performing various open-ended tasks. A video LLM comprises a visual encoder for perception, a projector for feature alignment, and an LLM for reasoning and response generation. Better solutions are achieved by adding audio modalities, jointly training images and videos, or performing alignment before projection. A recent trend involves integrating video LLMs with time-sensitive understanding capabilities, but solutions in related technologies remain suboptimal. Therefore, in embodiments of this disclosure, timestamp generation is redefined as embedding matching, i.e., the way temporal information is processed.

[0094] like Figure 2 As described in the embodiments, directly generating a continuous signal (e.g., a timestamp in this case, since time is continuous and can be infinitely subdivided) by discretizing the next token prediction is suboptimal. The LLM model in this disclosure is based on transformers. While transformers excel at selective copying rather than numerical computation, this disclosure reformulates timestamp prediction as embedding matching. That is, the model is trained to generate / copy the vectors / features of the video frames that the query text wants to reference, and the timestamp is obtained by matching these vectors / features back to the video frame features output by the LLM model. Specifically, the LLM model is first trained to generate or copy the vectors / features of the video frame pointed to by the reference information (e.g., where did the puppy's cake at 35s-42s come from? above), and then the vectors / features of that video frame are used to match the vectors / features of every video frame in the video to determine which frame is the most similar. The time of the most similar video frame in the video is used as the predicted time information.

[0095] For example, a special token can be defined. <vid>As a time marker, it is used in the stimulus matching process. When generated during inference... <vid>For example, the LLM model outputs "the event occurred..." <vid>The event happens around <vid>)",Should <vid>The token is used to match video frame tokens (i.e., individual video frame features output by the LLM model), so that the desired timestamp can be easily obtained from the matched frame index (i.e., the target frame index).

[0096] The LLM model receives both frame tokens 41 and text tokens 42 as input, and outputs "the event happens around" according to the normal LLM process. <vid>Then, add a post-processing step whenever the LLM model outputs... <vid>At that time, a match is performed with the features of each video frame, and the token (i.e., the event happens around) is used. <vid>In <vid>Replace ) with the time obtained from the match.

[0097] For example, for a video sampled at 1 FPS (Frames Per Second), if <vid>If it matches the i-th frame, then <vid>Let represent the i-th second of the video. The matching process is designed to be simple and efficient.

[0098] Output of the LLM model <vid>Token 65 (64 represents) <vid>Other text features besides token 65 and the l-th hidden state of video tokens 63 are represented as follows: and l is a positive integer greater than or equal to 1 and less than or equal to L. In this embodiment of the disclosure, the video frame features and text features used for matching are tokens output after processing by the LLM model, which facilitates feature alignment.

[0099] During the matching process, two MLP E are first used. vid and E frame (i.e., the first projector 70 and the second projector 80) project the hidden state onto the alignment space g:

[0100]

[0101] L refers to the total number of layers in the LLM model. In this embodiment, extraction is performed from the penultimate layer. <vid>The hidden states are used to preserve a larger feature range. This is because the features of the last layer of the LLM model primarily serve token prediction and contain less information that can be used for frame matching.

[0102] Then, calculate g respectively vid 81 and each g v The cosine similarity between 71 and 1 (i.e., it is assumed here that cosine similarity is used to measure the similarity between time stamp features and video frame features, but this disclosure is not limited thereto) is used to obtain the matching frame index (i.e., the target frame index) t. match :

[0103]

[0104] t match =argmax(s)(6)

[0105] argmax represents the frame index of the video frame that maximizes s, as t. match Then, the frame index t match Multiplying the time stamp by the frame rate (also known as the frame sampling rate) r yields the true timestamp (which refers to the time information predicted by the LLM model, and may not necessarily be consistent with the labeled time information), in seconds (s).

[0106] Through this operation, direct prediction of timestamps is replaced by embedding matching, which is more easily learned by transformer-based models as a selective copying problem.

[0107] for <vid>The situation that appears in the input, <vid>The input features are added to the corresponding frame features. That is, when the model's input (e.g., query text) contains time-related text, regular expression matching can be used to extract the time from the query text, and then the corresponding video frame (features) is used to replace that part of the text. For example, if the input is "What did the girl do around the 12th second?", the video frame features at the 12th second will be extracted and used to replace "around the 12th second" in the text. This refers to the actual operation where the corresponding video frame features are extracted and then added to the input. <vid>The characteristics of the token itself are added together, and then used for replacement. <vid>token.

[0108] For example, first, the sequence of video frames and the sequence of text composed of query text are processed using a visual encoder E. v Frame compressor E c The text encoder and tokenizer perform their respective processing, and after cascading, they obtain: <v1> <v2> ... <v100> <where> ... <cake> <at> <3><5> <s><-><4>... <from>At this time <v1> <v2> ... <v100>Represents frame compressor E c The output video frame features / compressed frame features (assuming there are 100 frames here), and the remaining tokens. <where> ... <cake> <at> <3><5> <s><-><4>... <from>"This represents the text token derived from the Tokenizer, i.e., the text feature. Then, the time-related parts are replaced with..." <vid> ( <vid>The number depends on the number of corresponding frames. For example, if it's 35-42 seconds and there's one frame per second, then there are 8. <vid> ):

[0109] <v1> <v2> ... <v100> <where> ... <cake> <at> <vid> <vid> <vid> ... <vid> <vid> <from>

[0110] Then, put each <vid>The corresponding video frame features / compressed frame features are from the left. <v1> <v2> ... <v100>Copy it over and match the corresponding <vid>Adding them together, we get the final sequence that is input into the LLM model:

[0111] <v1> <v2> ... <v100> <where> ... <cake> <at><vid+v35><vid+v36><vid+v37>...<vid+v42> <from>

[0112] It should be noted that the output of the LLM model does not involve frame features and <vid>The addition process only involves the output of the LLM model each time. <vid>When that happens, extract its second-to-last layer of features and match it with all the frame features output by the LLM (similarity calculation).

[0113] During training, an additional binary matching loss (i.e., a second loss function) is used:

[0114]

[0115] Where T represents the number of sampled frames in the sample video, and T is a positive integer greater than or equal to 1; t is a positive integer greater than or equal to 1 and less than or equal to T; y t This represents a binary annotation used to indicate whether the t-th frame in the sample video corresponds to the labeled time information. For example, if the time corresponding to the 1st frame in the sample video is the labeled time information, then y1 = 1; otherwise, y1 = 0. t This indicates the similarity between the time signature features of the corresponding sample frame features and the time signature features of the predicted sample response text features, for example, the calculation method of the above formula (5) can be referred to.

[0116] The above matching process still cannot maintain numerical continuity between them because the hidden states of adjacent frames may be far apart. That is, the results of frame matching lack continuity; frames with high matching similarity are not necessarily clustered together and may be scattered. Normally, the results of matching adjacent frames should be similar, and the matching results should exhibit a similar normal distribution. This disclosure effectively alleviates this problem by introducing two modifications. First, it is observed that causal self-attention in LLM blocks bidirectional information flow. This is reasonable for text but limits the capabilities of video understanding. Therefore, bidirectional self-attention is allowed between video tokens. This is because text understanding can proceed from left to right, possessing a left-to-right causal relationship. Video understanding requires bidirectional modeling to guarantee the aforementioned continuity; each frame needs to fuse adjacent frames on both the left and right sides to maintain continuity.

[0117] Secondly, a smooth annotation is introduced to replace the binary annotation y in formula (7). t The second loss function L matching This can be expressed by the following formula:

[0118]

[0119] in, Δ represents the smoothing annotation, used to indicate whether the t-th frame in the sample video corresponds to the annotation time information; Δ represents the parameter / hyperparameter controlling the smoothing degree, which is a real number greater than 0; t gt This indicates the annotation frame index corresponding to the annotation time information, that is, the frame number corresponding to the annotation time information t; s t This indicates the similarity between the time signature features of the corresponding sample frame features and the time signature features of the predicted sample response text features.

[0120] L matching Compared with the original language model loss L langage (The first loss function) is summed or weighted and summed together to optimize the model. L langage For example, it could be the cross-entropy loss function, used to classify each output token. During the instruction tuning phase, only the text token output by the LLM model is used to calculate the loss; the video token and the text token input to the LLM model are not used.

[0121] Table 1 shows the effect of different Δ values ​​on annotation smoothing. Smaller Δ values ​​make the optimization objective of the matching score smoother. Setting Δ in the range of 2.0 to 3.0 yields considerable results.

[0122] Table 1: The effect of Δ on annotation smoothing

[0123]

[0124]

[0125] Among them, Acc ref This indicates that accuracy is used to measure the model's reffering ability; F1 gnd This indicates that the F1 score is used to measure the model's grounding ability; F1 cap This indicates that the F1 score is used to measure the model's dense captioning ability; Sim cap This indicates that sentence similarity is used to measure the model's Dense Captioning ability; Rec com This indicates that recall is used to measure the model's Complex Understanding capability.

[0126] In this embodiment, the ViT-G / 14 pre-trained in EVA-CLIP can be used as the visual encoder. The frame compressor and LLM architecture are based on Q-Former and Vicuna-1.5-7B, and their weights are initialized from the second stage checkpoint of LLaMA VID. MLP projectors E vid and E frame It is randomly initialized, and <vid>The token embeddings are initialized from the average vector of all existing tokens. During training, the visual encoder and the FFN layer in the Q-Former can be frozen, and a LoRA adapter is introduced on the LLM. Therefore, only the frame compressor Ec, the LoRA adapter, and the projector (Ec) for matching are needed. vid &E frame The attention layer and projector in the model are learnable. The model can be trained at mixed precision (FP16) on a compute node with an 8×V100 GPU. The training process takes approximately 20 hours. Table 2 lists the hyperparameters in more detail.

[0127] Table 2: Hyperparameters used for instruction fine-tuning

[0128]

[0129]

[0130] The learning rate scheduler (LR Scheduler) and the optimizer are two components in the LLM training process. The optimizer primarily updates the model parameters based on the gradient of the loss function, minimizing the loss function by adjusting the model parameters. Optimizers include SGD, Adam, and RMSprop. The optimizer's behavior is typically controlled by a set of hyperparameters, one of which is the learning rate. The learning rate scheduler dynamically adjusts the learning rate to optimize model performance and convergence speed during training.

[0131] The embodiments disclosed herein also verify that <vid>The effectiveness of bidirectional attention between tokens and video tokens, as well as annotation smoothing during training, is discussed. Table 3 compares the results. As can be seen from Table 3, without introducing... <vid>In the case of tokens (first line), the model's performance reverts to that of using numerical expressions, making timestamp prediction difficult even with instruction tuning on the ET164K. By reformulating timestamp prediction as embedding matching (second line), the method provided in this disclosure works better on all tasks on ETBench. Additional modifications, namely bidirectional attention (third line) and annotation smoothing (fourth line), further enhance the model to achieve better localization and F1 accuracy. gnd The significant increase proves this point.

[0132] Table 3: Comparison of Structural Designs

[0133] <vid>Token < / vid> Bi-directional Smoothing <![CDATA[Acc ref ]]> <![CDATA[F1 gnd ]]> <![CDATA[F1 cap ]]> <![CDATA[Sim cap ]]> <![CDATA[Rec com ]]> 25.2 17.8 21.4 11.6 9.6 √ 34.1 24.0 26.3 12.7 12.2 √ √ 33.6 29.1 26.7 13.5 12.8 √ √ 34.5 25.3 25.9 10.4 13.1 √ √ √ 36.7 30.5 27.5 13.9 13.3

[0134] Table 4 compares frame compressor E c The performance of two design options was compared: simple spatial pooling between block vectors Pt and query-guided compression based on Q-Former. The results confirmed that using Q-Former for frame compression proved to be a superior choice.

[0135] Table 4: Selection of Frame Compressor

[0136] method <![CDATA[Acc ref ]]> <![CDATA[F1 gnd ]]> <![CDATA[F1 cap ]]> <![CDATA[Sim cap ]]> <![CDATA[Rec com ]]> Pooling 29.1 23.9 19.5 12.4 12.1 Q-Former 36.7 30.5 27.5 13.9 13.3

[0137] Table 5: Selecting the layer to match

[0138] <![CDATA[Layer vid ]]> <![CDATA[Layer frame ]]> <![CDATA[Acc ref ]]> <![CDATA[F1 gnd ]]> <![CDATA[F1 cap ]]> <![CDATA[Sim cap ]]> <![CDATA[Rec com ]]> -1 -1 36.2 30.1 27.3 14.0 12.9 -1 -2 35.5 29.6 25.8 11.4 12.2 -2 -1 36.7 30.5 27.5 13.9 13.3 -2 -2 35.8 30.8 26.4 12.5 12.7

[0139] Layer vid and Layer frame This is the index of the LLM layer used for matching. For example, "-1" indicates using the hidden state of the last layer, and "-2" indicates using the hidden state of the second-to-last layer. During the matching process, <vid>The token uses the penultimate hidden state, while the video frame uses the last hidden state. This strategy takes into account... <vid>The final hidden state of the token has a small feature range. Its effectiveness is further verified in Table 5. Table 5 shows that the penultimate layer was chosen as... <vid>The token matching demonstrates superior overall performance.

[0140] Table 6: Comparison of Training Strategies

[0141]

[0142]

[0143] ATTN and FFN (attention and feed forward layers) represent the attention layer and feed forward layer in the Q-Former, respectively. Table 6 demonstrates the rationale for the training strategy of fine-tuning for instructions. Updating only the LoRA parameters of the projector Ep (first row) or LLM (second row) makes training difficult to converge with new tokens; therefore, jointly updating them (third row) yields better performance. Due to the constraints of the pre-trained data, this limitation, acting as a frame compressor, hinders the model's performance. Rows 4–6 confirm this hypothesis, as updating the Q-Former on ET164K results in a significant performance improvement. Furthermore, it is observed that freezing the FFN layer of the Q-Former may strike a balance between adapting to new data and preserving the ability to pre-train.

[0144] Figure 4 This is a flowchart of a model training method provided in an embodiment of this disclosure. Figure 4 The steps in the method provided in the embodiments can be executed by any computer device. Here, execution by any server / terminal is used as an example for illustration, but this disclosure is not limited thereto.

[0145] like Figure 4 As shown, the method provided in this disclosure embodiment may include the following steps.

[0146] In S410, the sample video, sample query text, labeled response text, and labeled time information in the labeled response text are obtained.

[0147] In an exemplary embodiment, the sample video contains multiple events. For example, the sample video, sample query text, labeled response text, and labeled time information in the labeled response text in this embodiment can all come from a novel instruction-tuning dataset, ET164K, proposed in this embodiment. The samples in the instruction-tuning dataset are presented in the form of "instruction-question-answer," where "instruction" tells the model the goal of the task and the format of the answer, "question" asks a specific question, and "answer" refers to the correct answer to the question (i.e., the model's expected output, i.e., the labeled response text).

[0148] The proposed ET164K dataset contains multi-event understanding samples generated from 14 datasets, as shown in Table 7. It covers a wide range of event-level understanding tasks.

[0149] Table 7: Task and Sample Distribution in ET164K

[0150]

[0151] To fill the gap in multi-event and time-sensitive training data for LLMs, this disclosure introduces ET164K, a large-scale instruction fine-tuning dataset with fine-grained timestamp annotations. Statistics for the dataset are shown in Table 7. ET164K covers nine event-level understanding tasks, including [RVC], [TVG], [TAL], [EVS], [VHD], [DVC], [TVC], [SLC], and [GVQ]. Most tasks are consistent with ETBench, but the source datasets differ. The only exceptions are [RVC] and [TVC], the former requiring the model to generate textual descriptions for given time boundaries, and the latter similar to [DVC] but only with start timestamps. Note that [RAR], [ECA], [RVQ], [EPM], and [TEM] are only present in ETBench and can be considered as shelved tasks during evaluation.

[0152] Videos and annotations were carefully sampled from 14 datasets. During sampling, it was ensured that the videos did not overlap with ETBench. A filtering and rule-based cleaning process was performed on the videos and annotations. Note that videos from EgoTimeQA were randomly cropped to 150 seconds in length to reduce ambiguity during training.

[0153] The original annotations were then converted into instruction formats. For each task, a carefully crafted instruction was written, and GPT-4 was prompted to expand it into several different expressions. Six expressions were selected and refined as instruction templates for each task. To obtain basic factual responses, the original annotations were converted into a natural language style using the designed templates. The generated instruction templates and response formats are shown in Table 8.

[0154] Table 8: Command and Response Templates in ET164K

[0155]

[0156]

[0157]

[0158]

[0159] In Table 8, italicized text indicates placeholders to be filled for each sample, and bold text indicates timestamps.

[0160] In S420, the sample video and the sample query text are processed by a language model to obtain the sample frame features of the sample video and the predicted sample response text features.

[0161] In S430, the time signature features contained in the predicted sample response text features are extracted.

[0162] In S440, the time signature feature of the predicted sample response text feature is matched with the sample frame feature to determine the target sample frame feature from the sample frame feature.

[0163] In S450, based on the target sample frame features, the prediction time information in the prediction sample response text corresponding to the prediction sample response text features is obtained.

[0164] In S460, a first loss function is obtained based on the labeled response text and the predicted sample response text, and a second loss function is obtained based on the labeled time information and the predicted time information, so as to train the language model based on the first loss function and the second loss function.

[0165] In an exemplary embodiment, the second loss function L matching You can refer to formulas (7) or (8) above.

[0166] In an exemplary embodiment, Δ takes the value of a real number between [2, 3].

[0167] Table 9 compares the joint effects of model design and instruction tuning dataset collection. Two representative models (LLaMA VID and TimeChat) were selected as baselines and trained on ET164K. ETChat, provided in this embodiment, was also trained on the TimeIT dataset for in-depth comparison. The comparison results between rows 1 and 2, rows 3 and 4, and rows 5 and 6 demonstrate the effectiveness of ET164K. The results in rows 2, 4, and 6 validate the significance of the model design provided in this embodiment.

[0168] Table 9: Joint effect of model and instruction fine-tuning dataset (IT)

[0169] Model IT Dataset <![CDATA[Acc ref ]]> <![CDATA[F1 gnd ]]> <![CDATA[F1 cap ]]> <![CDATA[Sim cap ]]> <![CDATA[Rec com ]]> LLaMA-VID 723K Corpus 32.5 8.9 16.4 11.9 4.0 LLaMA-VID ET164K 31.3 16.1 20.3 15.0 7.8 TimeChat TimeIT 27.7 22.0 21.6 10.9 9.8 TimeChat ET164K 29.5 24.6 22.4 12.1 11.7 ETChat TimeIT 34.9 22.4 20.2 13.5 6.9 ETChat ET164K 36.7 30.5 27.5 13.9 13.3

[0170] It can be observed that the models in the relevant technologies have the following drawbacks: 1) weak instruction following capability, 2) low time resolution, 3) lack of event-level and time-sensitive design, and 4) lack of multi-event instruction adjustment data.

[0171] Recent advances in Video Large Language Models (MLMs) have demonstrated their immense potential in general-purpose video understanding. For example, the recent emergence of Multi-modal Large Language Models (MLLMs) has led to a substantial paradigm shift in visual language understanding, moving from designing task-specific models and collecting domain-specific data to developing general-purpose task solvers for open-ended scenarios. By integrating LLMs with visual encoders, these models collectively benefit from perceptual capabilities and powerful reasoning skills, demonstrating the significant potential of this approach even in unnamed applications. To validate / effectively evaluate the importance of these models, numerous benchmarks have been proposed to diagnose their capabilities across different scenarios and investigate their usability in various contexts. However, benchmarks in this field currently evaluate models solely through video-level question-answering, lacking fine-grained event-level assessment and task diversity. Furthermore, most benchmarks focus on image or short (second-long) video understanding, which requires strong static scene understanding capabilities but ignores fine-grained temporal information. Recent works have attempted to evaluate MLLMs on longer videos, but they still primarily use multiple-choice question-answering (MCQ) as the main task, lacking the flexibility of open-ended tasks. To fill this gap, embodiments of this disclosure introduce ETBench (Event-Level & Time-Sensitive Video Understanding Benchmark), a large-scale, high-quality benchmark for open-ended, event-level video (or video language) understanding.

[0172] In this disclosure, "open-ended" means that the questions in the evaluation benchmark can include both multiple-choice questions (including multiple-selection questions) and open-ended questions. "Video-level" means that the evaluation benchmark in related technologies only targets short videos (e.g., around 10 seconds) containing a single event, and the questions only address that event itself. "Event-level" means that the evaluation benchmark proposed in the embodiments of this disclosure can target medium-to-long videos (e.g., around 30 seconds to 600 seconds) containing multiple events, and different questions may point to different parts / different events in the same video. "Time-sensitive" includes two parts: 1) The question may contain time information, such as the question "What happened between 10 seconds and 14 seconds in the video?", where "10 seconds to 14 seconds" is the time information. 2) The model needs to use time information to answer the question, such as the model predicting the answer as "The event described by the user occurred between 20 seconds and 25 seconds", where "20 seconds to 25 seconds" is the time information. That is, the embodiments of this disclosure provide the following... Figures 5 to 16 All 12 evaluation tasks listed include time information in their input (the question posed in the video) and / or output (the answer predicted by the model from the video for the question).

[0173] ETBench categorizes data according to a 3-level task taxonomy, including 7.8K samples across 12 evaluation tasks and 7.7K videos (total length 266.3 hours) across 8 domains, providing a comprehensive evaluation.

[0174] This disclosure extensively evaluated 9 image LLMs and 10 video LLMs in benchmark tests. The results show that models in related techniques for coarse-level (i.e., video-level) understanding struggle with the fine-grained tasks proposed in this disclosure, such as grounding events of interest within videos (e.g., temporal localization of events of interest within videos). This is primarily because the training data for these models in related techniques consists of short videos (lacking contextual length) or training data lacking multi-event representations; that is, the training data in related techniques is mostly single-event videos, leading to models that are only adept at processing single-event videos, or models that use inappropriate temporal representations (see above). Figure 2 ).

[0175] To address these issues, this disclosure proposes a powerful baseline model, ETChat, and a fine-grained instruction-tuning dataset, ET164K, for event-level understanding. This simple yet effective solution demonstrates superior performance in multiple scenarios. Specifically, this disclosure addresses the problem from two perspectives: training data and the model. Firstly, it constructs the instruction-tuning dataset ET164K, which contains videos with multiple events, to train the ETChat model provided in this disclosure, thereby enabling the trained ETChat to recognize multiple events.

[0176] However, existing benchmarks are not designed for multi-event or time-sensitive scenarios, and therefore suffer from severe single-frame bias (meaning that most videos only require sampling and viewing one frame to answer questions, without involving temporal understanding), as can be seen from the comparable performance between image and video LLMs in these benchmarks. To address these issues and better understand the open-ended capabilities of these models, embodiments of this disclosure propose ETBench, a comprehensive benchmark for event-level and time-sensitive video understanding. The benchmark proposed in this disclosure differs significantly from benchmarks in related technologies, focusing on the temporal-sensitive understanding of long-duration and multi-event videos.

[0177] A well-performing video LLM should possess the ability to accurately refer to and localize any event that aligns with the user's interests. Based on this assumption, the task taxonomy provided in this disclosure is constructed by summarizing four fundamental capabilities required for time-sensitive video understanding: referring, grounding, dense captioning, and complex understanding. To verify whether the model possesses these capabilities, evaluation tasks were carefully designed for each capability. Scene diversity was ensured by carefully collecting videos from 16 datasets covering 8 domains. 7.8K high-quality annotations were generated using a comprehensive data cleaning, annotation repurposing (rewriting the original video annotations according to the purpose of the tasks designed in this disclosure; the rewriting process is rule-based, with rules designed for batch processing), instruction design, validation, and sampling (sampling videos and questions from the datasets).

[0178] The following is through Figures 5 to 16 Examples of tasks defined in ETBench are provided. These 12 tasks stem from four fundamental capabilities of time-sensitive video understanding: referring, grounding, dense captioning, and complex understanding. The questions and / or answers for each of the 12 tasks designed in this disclosure embodiment necessarily include time information. The videos are all multi-event videos, but the questions may only address one or more of the events.

[0179] like Figure 5 As shown, assuming the input question for this video is "What type of action is the camera wearer performing at around 23 seconds?", then the "refer" in the RAR task refers to the time information given in the question, such as "around 23 seconds". The following options are provided:

[0180] (A) Taking something out of something

[0181] (B) Opening something

[0182] (C) Putting something on top of something

[0183] (D) Closing something

[0184] Therefore, the correct answer should be Correct Answer: (B)

[0185] like Figure 6 As shown, suppose the input question for this video is "Which segment can be best described by 'a person sitting by the table while eating chocolate'?" In the ECA task, "caption" refers to a textual description of the event or visual information, generally a declarative sentence, such as "a person sits by the table while eating chocolate." The following options are provided:

[0186] (A)From 1.2to 7.6seconds(B)From 10.2to 15.3seconds

[0187] (C)From 18.6to 24.1seconds(D)From 12.5to 28.4seconds

[0188] Correct Answer: (C)

[0189] like Figure 7 As shown, suppose the input question for this video is "Watch the video from 1.4s to 5.2s and answer 'Which object was taken by the person?'", and the following options are provided:

[0190] (A)The dish(B)The kettle(C)The plate

[0191] (D) The refrigerator (E) Unable to answer

[0192] Correct Answer: (E)

[0193] like Figure 8 As shown, assuming the input for this video is "Localize the event about "buffalo is running", then the correct answer is: The event happens in 44.1-50.1 seconds.

[0194] like Figure 9 As shown, assuming the input for this video is "Where did I find the blue T-shirt in this shop?", then "Correct Answer: The place can be viewed in 330.7-350.2 seconds." means that this place can be seen within 330.7-350.2 seconds.

[0195] like Figure 10 As shown, assuming the input for this video is "Detect all the moments containing "cricket shot", then the correct answer is: It happens in 27.5-29.2 seconds and 48.1-50.3 seconds.

[0196] like Figure 11 As shown, assuming the input for this video is "Summarize this video to 15% of the total length.", then the correct answer is: The video summary locates in 4-12, 28-32, 56-58, 64-66, and 88-100 seconds.

[0197] like Figure 12 As shown, assuming the input for this video is "Identify the highlights of 'girl making chocolate juice'", then the Correct Answer is: The highlights of the given query happen at 18, 76, 84, and 105 seconds.

[0198] like Figure 13 As shown, assuming the input for this video is "describe all the events in the video," then the Correct Answer is: 0-56 seconds, a woman is talking to the camera. 58-94 seconds, she introduces the ingredients. 105-170 seconds... That is, from 0-56 seconds, a woman speaks to the camera. From 58-94 seconds, she introduces the ingredients.

[0199] like Figure 14 As shown, assuming the input for this video is "Identify and summarize all the key steps in the video," then the Correct Answer is: 0-4 seconds, shake and spray it. 42-68 seconds, rub it around. 78-90 seconds, wash with water. 98-112 seconds... That is, 0-4 seconds, shake and spray. 42-68 seconds, rub it around. 78-90 seconds, rinse with water.

[0200] like Figure 15 As shown, assuming the input for this video is "Find a similar moment to the one in 56-64 seconds," then the Correct Answer is: A similar event (a person with a mask pushing a shopping cart) can be found in 18-30 seconds.

[0201] like Figure 16 As shown, suppose the input for this video is "How many screws did I pick just now?", and the following options are provided:

[0202] (A) Five (B) Two (C) Three (D) One

[0203] The correct answer is (C), and the answer is: The relevant event can be found in 235.5-241.6 seconds.

[0204] Table 10: Quantitative Comparison of Video LLM Evaluation Benchmarks in ETBench and Related Technologies

[0205]

[0206] In Table 10 above, "Open" in the "Domain" column means that the video source used for evaluation is not restricted to any particular domain. "Open" in the "Answer Type" column means that the model's predicted output is an open-ended answer, i.e., not one or more options in a multiple-choice question. Avg. / Max Duration refers to the average / maximum duration of the videos used in the evaluation.

[0207] Nineteen models were extensively evaluated on ETBench, including seven open-source image LLMs, eight open-source video LLMs, and four commercial LLMs. Results show that models from related technologies, as proposed on the VideoQA benchmark, struggle on ETBench, particularly in grounding, dense captioning, and complex understanding tasks. First, the discrete next-token prediction paradigm has inherent limitations in numerical calculations, restricting timestamp understanding and generation. Second, most instruction fine-tuning datasets in related technologies primarily consist of short videos with coarse annotations, creating a significant gap between training and real-world applications. To address these issues, this disclosure proposes ETChat, a novel time-sensitive video LLM that redefines timestamp prediction as an embedding matching problem, serving as a strong foundation on ETBench. Regarding data, ET164K, an instruction fine-tuning dataset tailored for multi-event and time-sensitive scenarios, was also curated. Extensive comparisons demonstrate the effectiveness of the proposed model and dataset.

[0208] The following describes the detailed process used to develop ETBench. A thorough pre-filtering, annotation repurposing, instruction writing (first obtaining an instruction template, then filling in the processed annotations according to rules to form a complete instruction), checking, and sampling process were performed to obtain high-quality, fine-grained annotations. All samples for each subtask were checked, and 500 samples were randomly selected from the qualified samples. Annotations were cleaned and integrated based on rules. This step mainly removed incorrect annotations and then converted some task annotations into a format more suitable for LLM evaluation, without directly rewriting the content.

[0209] The process begins by defining four fundamental capabilities for event-level and time-sensitive video understanding: referring, grounding, dense captioning, and complex understanding, arranged in ascending order of difficulty. For each capability, a series of tasks are specifically designed to effectively evaluate its respective function. For each task, a carefully selected dataset with timestamps provided by annotators is used, and rewritten to the required format to ensure high quality and realism. ETBench's diversity is guaranteed through the careful selection of variable-length videos from different domains. Finally, a thorough inspection, filtering, and sampling process is conducted to eliminate unqualified samples. Details of each step are described below.

[0210] Figure 17 This is a schematic diagram of a task classification method (hierarchical task taxonomy) and sample distribution provided in an embodiment of this disclosure. To evaluate open video LLM from different perspectives, a three-level task classification method is designed, such as... Figure 17 As shown. The functions and tasks are defined as follows. Figure 17 The distribution of tasks, subtasks, and samples in ETBench is shown. Here, a subtask is defined as a task-source combination; for example, [TVG] contains three subtasks from three source datasets. Due to the emphasis on the moment localization capabilities of video LLM, a large portion of the samples belong to the grounding class.

[0211] Referring refers to the ability to understand temporal information from user input. For example, given a question like "What is this person doing from 23 seconds to 35 seconds?", the model must understand which part of the video the user is referring to and consider that segment more in its answer. To better quantify model performance, all referring tasks are designed as multiple-choice question answering (MCQ), including:

[0212] 1) [RAR]: Identifies the action given a rough timestamp prompt (e.g., "around 12 seconds"). The model must determine the actual reference based on the video and the options. That is, the model needs to determine which event "around 12s" specifically refers to based on the video itself and the content of the options.

[0213] 2) [ECA]: Select the correct time boundary for a given textual description. The model needs to understand and distinguish multiple timestamps in the options. Assuming the task has four options containing multiple times, the model needs to understand and distinguish them to answer correctly.

[0214] 3) [RVQ]: Answer questions based on a given passage. Each question provides four candidates and an "Unable to answer" option, indicating a situation where an answer cannot be given given the passage.

[0215] Grounding represents the ability to locate events or moments of interest using precise timestamps. This differs from previous methods that only considered coarse-level localization, such as "at the beginning / middle / end of a video." The output of the localization task is open-ended and processed by rule-based parsers, then evaluated using continuous metrics. Rule-based parsers extract time and description from the model output using regular expression matching, which is then used to calculate the evaluation metrics.

[0216] The definition of a grounding task is:

[0217] 1) [TVG]: Determine the time boundaries of a single event based on the text description.

[0218] 2) [EPM]: Locating an event in an egocentric scenario that can answer a given question, such as "Where is my backpack?".

[0219] 3) [TAL]: Detect and locate a series of segments containing a given action, for example, find all "golf swing" segments in a long video.

[0220] 4) [EVS]: Provides a list of segments that can be merged to form a compact video summary (approximately 15% of the total duration).

[0221] 5) [VHD]: Select the single timestamp (e.g., "15s") that best reflects the highlight moment corresponding to the query. [TAL], [EVS], and [VHD] have been modified to suit the nature of LLM. For example, the original TAL required detecting all known actions in the video (simultaneously locating multiple actions), while in the scenario of this embodiment, only one given action needs to be located, reducing the difficulty.

[0222] Dense Captioning is a more complex function that requires jointly locating key events and generating descriptions / summaries for each segment. Compared to video-level text descriptions on clips, this is more practical for storytelling or extracting key information from long videos. Two dense captioning tasks are defined based on different objectives: 1) [DVC]: Comprehensively describe all events occurring in the video. This is a general case, aiming to report as many events as possible. 2) [SLC]: Identify and describe only the key steps in the instructional video. In this case, compared to [DVC], the segments are shorter and non-overlapping, and the step descriptions are more precise.

[0223] Complex Understanding refers to the multi-functional integration of temporal understanding and event localization mentioned above, requiring the model to demonstrate proficient event-level and time-sensitive understanding. The two tasks are: 1) [TEM]: Find and localize similar events within a given segment under identical video conditions. This involves a two-stage reasoning process: first, identifying events at a given timestamp, then localizing another segment with the most similar content. 2) [GVQ]: Answer a given multiple-choice question by selecting an option and localizing the segment supporting the answer. This is also a complex scenario requiring both understanding and localization capabilities. Validating the localization results can help diagnose the reasoning process of video LLM.

[0224] A key challenge in data collection is obtaining videos with accurate temporal boundary annotations. Work in related technologies either generates annotations directly via LLM or uses predefined rules to convert human-annotated time stamps (e.g., 5.2s) into boundaries (e.g., 3.2s to 7.2s). These solutions can only generate temporal boundaries with significant noise, making them unsuitable for accurate model evaluation. Therefore, embodiments of this disclosure carefully select multi-event videos from a dataset with high-quality annotated timestamps and transform the annotations according to the required task format, thereby repurposing the annotations for specific uses.

[0225] To ensure diversity of scenarios, 16 datasets were carefully selected from eight domains: indoor activities, desktop, sports, wildlife, egocentricity, cooking, news and video logs, and operational methods. All videos were collected from either the evaluation or test splits to prevent potential data leakage; that is, the evaluation videos were not used during training.

[0226] For annotation generation, a complete process was developed for each task, including pre-filtering, manual annotation repurposing, and instruction template design. After annotation generation, the samples underwent careful review, focusing on content suitability, annotation accuracy, instruction clarity, scene diversity, video and event length, and task difficulty. Feedback from this review helped to actively optimize the generation process. Finally, to balance the quality and efficiency of the evaluation, 500 sample subtasks (task-source combinations) were randomly selected. In most cases, each video was sampled only once.

[0227] The word cloud of the text queries (ETBench Text Queries word cloud) shows a considerable degree of diversity. Due to the broad scope of the video domain, the queries also vary considerably in terms of nouns and verbs. While most queries are human-centered, describing human activities or human-computer interactions, in some tasks, queries may contain unusual words, such as the scientific names of wild animals.

[0228] Figure 18 This is a schematic diagram showing the distribution of the average video duration (in seconds) for the 12 tasks provided in this embodiment. The video durations provided in this embodiment are quite varied; the videos for the referring task are relatively short, while the videos for grounding and dense captioning are longer. Experimental results show that the video duration has a significant impact on model performance.

[0229] Overall, the proposed ETBench contains 7789 samples, categorized into 4 abilities, 12 tasks, and 21 sub-tasks. A total of 7737 unique videos come from 16 datasets, covering 8 domains. The average video duration is 124 seconds, with a minimum of 5 seconds and a maximum of 795 seconds. This differs from most existing benchmarks, which have an average duration of only 10–20 seconds. Different answer types are also provided for different tasks, including MCQs and open-ended styles. The evaluation process is entirely rule-based, without human intervention or LLM integration, ensuring satisfactory objectivity.

[0230] Extensive evaluations have shown that video LLMs in related technologies do not perform well on ETBench, especially on more complex tasks such as grounding, dense captioning, and complex understanding. As previously mentioned, this is due to limitations in model design and data in related technologies. The ETChat and ET164K provided in this disclosure address these limitations accordingly.

[0231] Table 11: Performance of representative MLLMs on ETBench

[0232]

[0233]

[0234] In Table 11, the LLM size of the open-source model is 7B by default. The best and second-best results are marked with bold and underline, respectively. Since different tasks in ETBench are under different settings and have different output formats, a single metric (e.g., accuracy) like existing benchmarks is insufficient. To balance the number of metrics and ease of ranking, the metrics for each capability are unified, utilizing accuracy (Acc) for the referring task, F1 score for the grounding task, F1 score and sentence similarity (sim) for the dense captioning task, and recall (Rec) for the complex understanding task.

[0235] Seven open-source image LLMs and eight open-source video LLMs were extensively evaluated on ETBench. For image LLMs, eight frames were uniformly sampled, with an additional cue indicating the video duration as a timestamp. For video LLMs, their default frame count was used as input. The evaluation results are shown in Table 11. Due to space constraints, average metrics across subtasks are reported. "Random" in the first row refers to random guessing. The main findings from the results are summarized below. Performance gap between image and video LLMs. It was observed that most image and video LLMs performed at the same level in the referring task. Some image LLMs, such as XComposer and Qwen VL Chat, even outperformed most video LLMs. This is because the videos used for referring are typically short, so the sampled eight frames cover most of the information.

[0236] The gap widens on tasks with longer videos, such as [TVG] and [TAL], highlighting the importance of temporal modeling on ETBench compared to other benchmarks.

[0237] Some state-of-the-art video LLMs based on existing benchmarks, such as Video-LLaMA-2 and PLAVA, perform poorly on ETBench, especially in grounding and dense captioning evaluation tasks. This is attributed to single-frame bias caused by model architecture and training data. It also motivates considering a balance between spatial and temporal modeling in video LLMs.

[0238] Some video LLMs were not operated according to the instructions. During the evaluation, it was also noted that some models, such as Video LLaVA and Video-LLaMA-2, failed to generate the required output format for certain tasks, even with well-designed instructions and examples. For example, Video LLaVA only generated repetitive text output, [EVS] had no timestamps, and Video-LLaMA-2 faced similar issues in almost all grounding tasks. This is because their instructions adjust the data and do not include any timestamped output.

[0239] Thanks to its novel timestamp handling design and multi-event instruction fine-tuning of data, ETChat achieves state-of-the-art performance among open-source MLLMs on most tasks. Notably, significant improvements can be seen on [EPM], [VHD], [TEM], and [GVQ].

[0240] This work introduces ETBench, a large-scale comprehensive evaluation benchmark for multi-event and time-sensitive video language understanding. The benchmarks provided in this disclosure cover a wide range of tasks across different video domains, evaluating various capabilities of video LLM. Experimental results demonstrate the limitations of current video LLM model design and instruction fine-tuning data in terms of timestamp representation and multi-event modeling. To address these challenges, this disclosure develops a novel ETChat model and incorporates the multi-event instruction fine-tuning dataset ET164K as a robust foundational solution for ETBench.

[0241] like Figure 19 As shown, the video text processing apparatus 100 provided in this embodiment includes a receiving unit 110 and a processing unit 120. The receiving unit 110 is used to acquire a video to be processed and a query text. The processing unit 120 is used to process the video to be processed and the query text using a language model to obtain video frame features of video frames in the video to be processed and predicted response text features of the query text. The processing unit 120 is further used to extract time signature features contained in the predicted response text features. The processing unit 120 is further used to match the time signature features of the predicted response text features with the video frame features to determine target video frame features from the video frame features. The processing unit 120 is further used to determine the time information in the predicted response text corresponding to the predicted response text features based on the target video frame features.

[0242] In an exemplary embodiment, the processing unit 120 is further configured to: extract the hidden state of the time identifier in the predicted response text at layer L-1 in the language model, as the time identifier feature of the predicted response text feature; wherein, L is the layer number of the language model, and L is a positive integer greater than or equal to 1.

[0243] In an exemplary embodiment, the processing unit 120 is further configured to: perform alignment processing on the time identifier feature of the predicted response text feature and the video frame feature to obtain aligned time identifier features and video frame features of the predicted response text feature; calculate the similarity between the aligned time identifier feature of the predicted response text feature and the video frame feature; and use the video frame feature that satisfies the similarity condition as the target video frame feature.

[0244] In an exemplary embodiment, the processing unit 120 is further configured to: obtain the target frame index of the target video frame features; and determine the time information in the predicted response text based on the target frame index and the frame rate of the video to be processed.

[0245] In an exemplary embodiment, the processing unit 120 is further configured to: sample video frames of the video to be processed at time t, where t is a positive integer greater than or equal to 1 and less than or equal to T, and T is a positive integer greater than or equal to 1; process the video frames at time t using a visual encoder to obtain block vectors of the video frames at time t; process the block vectors of the video frames at time t and the query text using a frame compressor to obtain compressed frame features of the video frames at time t; process the query text using a text encoder to obtain text features of the query text; and process the concatenated compressed frame features from time 1 to time T and the text features using the language model to obtain the video frame features and the predicted response text features.

[0246] In an exemplary embodiment, the processing unit 120 is further configured to: extract time text from the query text; replace the part of the text feature corresponding to the time text with a time identifier; obtain target text features based on the time identifier in the text feature and the corresponding compressed frame feature; and process the concatenated compressed frame features from time 1 to time T and the target text feature through the language model to obtain the video frame features of the video to be processed and the predicted response text features.

[0247] In an exemplary embodiment, the processing unit 120 is further configured to: process the block vector of the video frame at time t and the query text through the transformer in the frame compressor to obtain the learning query vector at time t; process the block vector of the video frame at time t and the learning query vector at time t through the context attention module in the frame compressor to obtain the compressed frame vector of the video frame at time t; and project the compressed frame vector of the video frame at time t onto the same vector space as the language model to obtain the compressed frame features of the video frame at time t.

[0248] In an exemplary embodiment, the processing unit 120 is further configured to: process the compressed frame features from time 1 to T through the bidirectional self-attention module in the language model to obtain the video frames of the video to be processed; and process the text features through the causal self-attention module in the language model to obtain the predicted response text features.

[0249] like Figure 20 As shown, the model training apparatus 200 provided in this embodiment includes a receiving unit 210 and a processing unit 220. The receiving unit 210 is used to acquire sample videos, sample query text, labeled response text, and labeled time information in the labeled response text. The processing unit 220 is used to process the sample videos and sample query text using a language model to obtain sample frame features and predicted sample response text features of the sample videos. The processing unit 220 is also used to extract time signature features contained in the predicted sample response text features. The processing unit 220 is also used to match the time signature features of the predicted sample response text features with the sample frame features to determine target sample frame features from the sample frame features. The processing unit 220 is also used to obtain prediction time information in the predicted sample response text corresponding to the predicted sample response text features based on the target sample frame features. The processing unit 220 is also used to obtain a first loss function based on the labeled response text and the predicted sample response text, and a second loss function based on the labeled time information and the predicted time information, to train the language model based on the first loss function and the second loss function.

[0250] In an exemplary embodiment, Δ takes the value of a real number between [2, 3].

[0251] In an exemplary embodiment, the sample video contains multiple events.

[0252] It is understood that the terminal mentioned in the embodiments of this disclosure can be a computer device, including but not limited to a terminal or a server. In other words, the computer device can be a server or a terminal, or a system composed of a server and a terminal. The terminal mentioned above can be an electronic device, including but not limited to mobile phones, tablets, desktop computers, laptops, handheld computers, in-vehicle devices, augmented reality / virtual reality (AR / VR) devices, head-mounted displays, smart TVs, wearable devices, smart speakers, digital cameras, webcams, and other mobile internet devices (MIDs) with network access capabilities, or terminals in scenarios such as trains, ships, and flights.

[0253] The servers mentioned above can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, vehicle-road cooperation, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0254] Optionally, the data involved in the embodiments of this disclosure may be stored in a computer device or may be stored based on cloud storage technology, without limitation.

[0255] See Figure 21 , Figure 21 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. Figure 21 As shown, the computer device in this embodiment may include one or more processors 1701, a memory 1702, and an input / output interface 1703. The processor 1701, memory 1702, and input / output interface 1703 are connected via a bus 1704. The memory 1702 stores a computer program, which includes program instructions. The input / output interface 1703 receives and outputs data, such as for data interaction between the host machine and the computer device, or for data interaction between various virtual machines within the host machine. The processor 1701 executes the program instructions stored in the memory 1702.

[0256] The processor 1701 can perform the following operations: acquire a video to be processed and a query text; process the video to be processed and the query text using a language model to obtain video frame features and predicted response text features of video frames in the video to be processed; extract time identifier features contained in the predicted response text features; match the time identifier features of the predicted response text features with the video frame features to determine the target video frame features from the video frame features; and determine the time information in the predicted response text corresponding to the predicted response text features based on the target video frame features.

[0257] Alternatively, the processor 1701 may perform the following operations: acquire a sample video, sample query text, labeled response text, and labeled time information in the labeled response text; process the sample video and sample query text using a language model to obtain sample frame features and predicted sample response text features of the sample video; extract time signature features contained in the predicted sample response text features; match the time signature features of the predicted sample response text features with the sample frame features to determine target sample frame features from the sample frame features; obtain the predicted time information in the predicted sample response text corresponding to the predicted sample response text features based on the target sample frame features; obtain a first loss function based on the labeled response text and the predicted sample response text, and obtain a second loss function based on the labeled time information and the predicted time information, so as to train the language model based on the first loss function and the second loss function.

[0258] In some feasible implementations, the processor 1701 may be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0259] The memory 1702 may include read-only memory and random access memory, and provides instructions and data to the processor 1701 and input / output interface 1703. A portion of the memory 1702 may also include non-volatile random access memory. For example, the memory 1702 may also store device type information.

[0260] In practice, the computer device can execute the implementation methods provided by each step in any of the above method embodiments through its built-in functional modules. For details, please refer to the implementation methods provided by each step in the figure shown in the above method embodiments, which will not be repeated here.

[0261] This disclosure provides a computer device including a processor, an input / output interface, and a memory. The processor retrieves a computer program from the memory and executes the steps of the method shown in any of the above embodiments.

[0262] This disclosure also provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the video text processing method provided in each step of any of the above embodiments. Specific implementations of each step in each of the above embodiments can be found therein and will not be repeated here. Furthermore, the beneficial effects of using the same method will not be repeated here either. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this disclosure, please refer to the description of the method embodiments of this disclosure. As an example, the computer program can be deployed to execute on a single computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed across multiple locations and interconnected via a communication network.

[0263] The computer-readable storage medium can be the video text processing apparatus or model training apparatus provided in any of the foregoing embodiments, or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0264] This disclosure also provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various alternative embodiments described above.

[0265] The terms "first," "second," etc., in the specification, claims, and drawings of this disclosure are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0266] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0267] The methods and related apparatus provided in this disclosure are described with reference to the method flowcharts and / or structural diagrams provided in this disclosure. Specifically, each block of the method flowchart and / or structural diagram, as well as combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable application display device to create a machine, such that the instructions, which execute via the processor of the computer or other programmable application display device, generate instructions for implementing the process... Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable application display device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable application display device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or structures illustrate the steps of the functions specified in one or more boxes.

[0268] The above-disclosed embodiments are merely preferred embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Therefore, any equivalent variations made in accordance with the claims of this disclosure shall still fall within the scope of this disclosure.< / vid> < / vid> < / vid> < / vid> < / vid> < / vid> < / vid> < / vid> < / from> < / at> < / cake> < / where> < / v100> < / v2> < / v1> < / vid> < / v2> < / v1> < / vid> < / from> < / vid> < / vid> < / vid> < / vid> < / vid> < / at> < / cake> < / where> < / v100> < / v2> < / v1> < / vid> < / vid> < / vid> < / from> < / s> < / at> < / cake> < / where> < / v2> < / v1> < / from> < / s> < / at> < / cake> < / where> < / v100> < / v2> < / v1> < / vid> < / vid> < / vid> < / vid> < / vid> < / vid> < / vid> < / vid> < / vid> < / vid> < / vid> < / vid> < / vid> < / vid> < / vid> < / vid> < / vid> < / vid> < / vid>

Claims

1. A method of video text processing, characterized by, The method comprises: acquiring a to-be-processed video and a query text; processing the to-be-processed video and the query text through a language model to obtain video frame features of video frames in the to-be-processed video and predicted reply text features; extracting time identifier features contained in the predicted reply text features; matching the time identifier features of the predicted reply text features with the video frame features to determine target video frame features from the video frame features; determining time information in a predicted reply text corresponding to the predicted reply text features according to the target video frame features.

2. The method of claim 1, wherein, extracting the time identifier features contained in the predicted reply text features comprises: extracting hidden states of the time identifier in the L-1 layer of the language model in the predicted reply text as the time identifier features of the predicted reply text features; wherein L is the number of layers of the language model, and L is a positive integer greater than or equal to 1.

3. The method of claim 1 or 2, wherein, matching the time identifier features of the predicted reply text features with the video frame features to determine target video frame features from the video frame features comprises: aligning the time identifier features of the predicted reply text features and the video frame features to obtain aligned time identifier features of the predicted reply text features and video frame features; calculating the similarity between the aligned time identifier features of the predicted reply text features and the video frame features; taking the video frame features with a similarity satisfying a condition as the target video frame features.

4. The method of claim 1, wherein, determining the time information in the predicted reply text corresponding to the predicted reply text features according to the target video frame features comprises: obtaining a target frame index of the target video frame features; determining the time information in the predicted reply text according to the target frame index and the frame rate of the to-be-processed video.

5. The method of claim 1, wherein, processing the to-be-processed video and the query text through a language model to obtain video frame features of video frames in the to-be-processed video and predicted reply text features comprises: sampling a video frame of the to-be-processed video at time t, t being a positive integer greater than or equal to 1 and less than or equal to T, and T being a positive integer greater than or equal to 1; processing the video frame at time t through a visual encoder to obtain a block vector of the video frame at time t; processing the block vector of the video frame at time t and the query text through a frame compressor to obtain a compressed frame feature of the video frame at time t; processing the query text through a text encoder to obtain a text feature of the query text; processing the concatenated compressed frame features at times 1 to T and the text feature through the language model to obtain the video frame features and the predicted reply text features.

6. The method of claim 5, wherein, processing the concatenated compressed frame features at times 1 to T and the text feature through the language model to obtain the video frame features and the predicted reply text features comprises: extracting time text in the query text; replacing a part of the text feature corresponding to the time text with a time identifier; obtaining a target text feature according to the time identifier in the text feature and the corresponding compressed frame feature; The compressed frame features of the 1st to Tth moments are concatenated with the target text features, and processed through the language model to obtain the video frame features of the to-be-processed video and the predicted reply text features.

7. The method of claim 5, wherein, The block vector of the video frame at the tth moment and the query text are processed through the frame compressor to obtain the compressed frame features of the video frame at the tth moment, including: The block vector of the video frame at the tth moment and the query text are processed through the transformer in the frame compressor to obtain the learning query vector at the tth moment. The block vector of the video frame at the tth moment and the learning query vector at the tth moment are processed through the context attention module in the frame compressor to obtain the compressed frame vector of the video frame at the tth moment. The compressed frame vector of the video frame at the tth moment is projected into the same vector space as the language model to obtain the compressed frame features of the video frame at the tth moment.

8. The method of claim 5, wherein, The compressed frame features of the 1st to Tth moments are concatenated with the text features, and processed through the language model to obtain the video frame features of the to-be-processed video and the predicted reply text features, including: The compressed frame features of the 1st to Tth moments are processed through the bidirectional self-attention module in the language model to obtain the video frames of the to-be-processed video. The text features are processed through the causal self-attention module in the language model to obtain the predicted reply text features.

9. A model training method, comprising: Including: Obtaining a sample video, a sample query text, a labeled reply text, and labeled time information in the labeled reply text; Processing the sample video and the sample query text through a language model to obtain sample frame features of the sample video and predicted sample reply text features; Extracting time identification features contained in the predicted sample reply text features; Matching the time identification features of the predicted sample reply text features with the sample frame features to determine target sample frame features from the sample frame features; According to the target sample frame features, obtaining predicted time information in a predicted sample reply text corresponding to the predicted sample reply text features; According to the labeled reply text and the predicted sample reply text, obtaining a first loss function, and according to the labeled time information and the predicted time information, obtaining a second loss function, to train the language model based on the first loss function and the second loss function.

10. The method of claim 9, wherein, The second loss function L matching is expressed by the following equation: wherein T represents a sampling frame number of the sample video, T is a positive integer greater than or equal to 1; t is a positive integer greater than or equal to 1 and less than or equal to T; y t represents a binary label, used to indicate whether the t-th frame in the sample video corresponds to the label time information; s t represents a similarity between the corresponding sample frame feature and the time identification feature of the predicted sample reply text feature.

11. The method of claim 9, wherein, The second loss function L matching is expressed by the following equation: Wherein, T represents the sampling frame number of the sample video, T is a positive integer greater than or equal to 1; t is a positive integer greater than or equal to 1 and less than or equal to T; Indicates a smooth label, used to indicate whether the t-th frame in the sample video corresponds to the label time information; alpha represents a parameter for controlling the smoothness, which is a real number greater than 0; t gt Indicates the label frame index corresponding to the label time information; s t Indicates the similarity between the corresponding sample frame feature and the time identification feature of the predicted sample reply text feature.

12. The method of claim 11, wherein, The sample video contains multiple events.

13. A video text processing apparatus, characterized by comprising: Including: A receiving unit configured to obtain a to-be-processed video and a query text; A processing unit configured to process the to-be-processed video and the query text through a language model to obtain video frame features of video frames in the to-be-processed video and predicted reply text features of the query text; The processing unit is further configured to extract time identification features contained in the predicted reply text features; The processing unit is further configured to match the time identification features of the predicted reply text features with the video frame features to determine target video frame features from the video frame features, to train the language model based on the first loss function and the second loss function.

14. A model training apparatus, comprising: Including: The receiving unit is configured to acquire a sample video, a sample query text, a labeled reply text, and labeled time information in the labeled reply text; The processing unit is configured to process the sample video and the sample query text thereof by using a language model to obtain sample frame features of the sample video and predicted sample reply text features; The processing unit is further configured to extract time identifier features contained in the predicted sample reply text features; The processing unit is further configured to match the time identifier features of the predicted sample reply text features with the sample frame features to determine target sample frame features from the sample frame features; The processing unit is further configured to obtain predicted time information in a predicted sample reply text corresponding to the predicted sample reply text features according to the target sample frame features; The processing unit is further configured to obtain a first loss function according to the labeled reply text and the predicted sample reply text, and obtain a second loss function according to the labeled time information and the predicted time information, so as to train the language model based on the first loss function and the second loss function.

15. A computer device, comprising: The computer device comprises a processor, a memory, and an input / output interface; The processor is connected with the memory and the input / output interface respectively, wherein the input / output interface is configured to receive and output data, the memory is configured to store a computer program, and the processor is configured to call the computer program to enable the computer device to execute the method of any one of claims 1-8 or the method of any one of claims 9-12.

16. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is adapted to be loaded and executed by the processor to enable the computer device with the processor to execute the method of any one of claims 1-8 or the method of any one of claims 9-12.

17. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-8 or the method of any one of claims 9-12.

Images