Training method of video understanding model, processing method and device of video data, equipment, storage medium and program product

Abstract

The application provides a video understanding model training method, a video data processing method, an apparatus, a device, a storage medium and a program product. The method comprises: obtaining a first visual feature sample and a first audio feature sample of a video sample, and obtaining a text feature sample of a prompt word sample; generating a first thinking process sample based on the first visual feature sample, the first audio feature sample and the text feature sample through a first video understanding model; generating a first predicted text sample based on the first thinking process sample through the first video understanding model; dividing the first predicted text sample into a plurality of predicted text segment samples, determining a first reward value based on the plurality of predicted text segment samples and at least one real text segment, and updating parameters of the first video understanding model based on the first reward value. Through the application, the accuracy of the video understanding model in generating structured transcription content corresponding to a video can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a training method for a video understanding model, a method for processing video data, an apparatus, a device, a storage medium, and a program product. Background Technology

[0002] With the development of artificial intelligence technology, video understanding models have been widely used in scenarios such as video content parsing and narrative transcription. In related technologies, end-to-end model architectures generate text descriptions corresponding to video data. However, due to the tendency of video data features to undergo information dilution and forgetting during network forward propagation, the accuracy of structured transcription of video content is affected. Summary of the Invention

[0003] This application provides a training method for a video understanding model, a method for processing video data, an apparatus, a device, a storage medium, and a program product, which can improve the accuracy of the structured transcription content corresponding to the video generated by the video understanding model.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a method for training a video understanding model, the method comprising: The system acquires a first visual feature sample and a first audio feature sample of a video sample, as well as a text feature sample of a preset prompt word sample. The prompt word sample is used to instruct the first video understanding model to perform a video content transcription task for the video sample. The sequence length of the first visual feature sample is the same as the sequence length of the first audio feature sample. The first video understanding model generates a first thought process sample based on the first visual feature sample, the first audio feature sample, and the text feature sample. The first visual feature sample and the first audio feature sample are injected into network layers of different depths in the first video understanding model. The first video understanding model generates a first predicted text sample based on the first thought process sample. The first predicted text sample includes structured transcription content of multiple events, and the first thought process sample includes a content summary and role association information of the video sample. The first predicted text sample is divided into multiple predicted text segment samples. Based on the multiple predicted text segment samples and at least one real text segment corresponding to the multiple predicted text segment samples, a first reward value is determined, and the parameters of the first video understanding model are updated based on the first reward value.

[0005] This application provides a method for processing video data, including: The system acquires first visual features and first audio features of the video data to be processed, and acquires text features of preset prompt words, wherein the prompt words are used to instruct the video understanding model to perform a video content transcription task for the video data, the sequence length of the first visual features is the same as the sequence length of the first audio features, and the video understanding model is trained by the training method of the video understanding model provided in the embodiments of this application. The video understanding model generates a thought process based on the first visual feature, the first audio feature, and the text feature, and generates predicted text based on the thought process. The first visual feature and the first audio feature are injected into network layers of different depths in the video understanding model. The thought process includes a summary of the video data and role association information. The predicted text includes structured transcribed content of multiple events.

[0006] This application provides a training apparatus for a video understanding model, comprising: The data processing module is used to acquire first visual feature samples and first audio feature samples of video samples, and to acquire text feature samples of preset prompt words, wherein the prompt word samples are used to instruct the first video understanding model to perform a video content transcription task for the video samples, and the sequence length of the first visual feature samples is the same as the sequence length of the first audio feature samples. The training module is used to generate a first thought process sample based on the first visual feature sample, the first audio feature sample and the text feature sample through the first video understanding model, wherein the first visual feature sample and the first audio feature sample are injected into network layers of different depths of the first video understanding model. The training module is further configured to generate a first predicted text sample based on the first thought process sample using the first video understanding model, wherein the first predicted text sample includes structured transcribed content of multiple events, and the first thought process sample includes a content summary and role association information of the video sample; The training module is further configured to divide the first predicted text sample into multiple predicted text segment samples, determine a first reward value based on the multiple predicted text segment samples and at least one real text segment corresponding to the multiple predicted text segment samples, and update the parameters of the first video understanding model based on the first reward value.

[0007] This application provides a video data processing apparatus, including: The data processing module is used to acquire first visual features and first audio features of the video data to be processed, and to acquire text features of preset prompt words, wherein the prompt words are used to instruct the video understanding model to perform a video content transcription task for the video data, the sequence length of the first visual features is the same as the sequence length of the first audio features, and the video understanding model is trained by the training method of the video understanding model provided in the embodiments of this application. The text generation module is used to generate a thought process based on the first visual feature, the first audio feature, and the text feature through the video understanding model, and to generate predicted text based on the thought process. The first visual feature and the first audio feature are injected into network layers of different depths in the video understanding model. The thought process includes a content summary and role association information of the video data. The predicted text includes structured transcription content of multiple events.

[0008] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the training method for the video understanding model provided in the embodiments of this application, or implements the video data processing method provided in the embodiments of this application.

[0009] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implement the training method for the video understanding model provided in this application, or the video data processing method provided in this application.

[0010] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the training method for the video understanding model provided in this application, or the video data processing method provided in this application.

[0011] The embodiments of this application have the following beneficial effects: By injecting first visual feature samples and first audio feature samples with the same sequence length into network layers of different depths in the first video understanding model, it is ensured that the video understanding model can continuously acquire original visual and audio information at multiple levels of forward propagation. By first generating a first thought process sample including a summary of the video content and information on the relationships between characters, and then generating predicted text containing structured transcribed content including multiple events based on the first thought process sample, the video understanding model can be guided to prioritize the construction of a clear macro-level content logic and a framework of character relationships. This provides coherent semantic support for the generation of structured transcribed content for multiple subsequent events, improving the logical rationality of complex video content transcription. By dividing the first predicted text sample into multiple predicted text fragment samples and combining them with the corresponding real text fragments to determine the first reward value, the video understanding model can be provided with highly discriminative and fine-grained supervision signals when processing long structured text. This makes the optimization direction of the video understanding model parameters more targeted, thereby improving the accuracy of the video understanding model in generating structured transcribed content corresponding to the video. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the training system architecture of the video understanding model provided in this application embodiment; Figure 2A This is a first structural diagram of the server provided in an embodiment of this application; Figure 2B This is a schematic diagram of the second structure of the server provided in an embodiment of this application; Figure 3A This is a schematic diagram of the first process of the training method for the video understanding model provided in the embodiments of this application; Figure 3B This is a schematic diagram of the second process of the training method for the video understanding model provided in the embodiments of this application; Figure 3C This is a schematic diagram of the third process of the training method for the video understanding model provided in the embodiments of this application; Figure 4 This is a flowchart illustrating the video data processing method provided in an embodiment of this application; Figure 5 This is a schematic diagram of the video transcription results provided in the embodiments of this application; Figure 6 This is a schematic diagram of the architecture of the video understanding model provided in the embodiments of this application; Figure 7 This is a schematic diagram illustrating the actual text construction principle provided in the embodiments of this application; Figure 8 This is a schematic diagram of the evaluation process provided in the embodiments of this application; Figure 9This is a comparison chart of model performance changes under different video durations provided in the embodiments of this application.

[0013] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0016] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0017] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0018] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0019] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0020] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0021] 1) Video understanding model refers to a computer model that is based on deep learning algorithms and processes and analyzes video data to realize the cognition, analysis and reasoning of video content. It can extract objects, scenes, human behavior and complex event logic from videos.

[0022] 2) Large Language Models (LLMs), also known as large models, are large-scale language models designed to understand and generate human language. They are trained on massive amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more. The defining characteristic of large language models is their sheer size, containing billions of parameters that help them learn complex patterns in language data. They are typically based on deep learning architectures. Large language models refer to deep learning models trained on vast amounts of text data, containing billions or even more parameters. They can be used to generate natural language text and understand its meaning. Through training, the models learn the statistical regularities and semantic relationships of language to build a vast language knowledge base, thereby simulating human language comprehension and generation capabilities.

[0023] 3) Multimodal Large Language Model (MLLM) refers to a large-scale artificial intelligence model that builds upon the single-modal large language model by incorporating the perception and understanding capabilities of multiple non-textual data modalities, such as visual (images, video frames) and audio. By mapping or aligning features from different modalities to a unified semantic space, the MLLM not only retains the advanced logical reasoning and text generation capabilities brought by its large number of parameters, but also achieves cross-modal information fusion and joint modeling, enabling it to perform comprehensive tasks such as visual question answering, generating text descriptions from videos, cross-modal retrieval, and complex audio and video content transcription.

[0024] With the development of artificial intelligence technology, video understanding models have been widely used in scenarios such as video content parsing and narrative transcription. In related technologies, end-to-end model architectures generate text descriptions corresponding to video data. However, due to the tendency of video data features to undergo information dilution and forgetting during network forward propagation, the accuracy of structured transcription of video content is affected.

[0025] This application provides a training method for a video understanding model, a method for processing video data, an apparatus, a device, a computer-readable storage medium, and a computer program product, which can improve the accuracy of the structured transcription content corresponding to the video generated by the video understanding model. The following describes exemplary applications of the electronic devices provided in this application. The electronic devices provided in this application can be implemented as various types of terminal devices such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and vehicle terminals, or they can be implemented as servers.

[0026] See Figure 1 , Figure 1 This is a schematic diagram of the training system architecture for the video understanding model provided in this application embodiment. Figure 1 The system involves server 100, terminal device 200, and network 300. Terminal device 200 is connected to server 100 through network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0027] In some embodiments, the training method for the video understanding model provided in this application can be implemented collaboratively by a server and a terminal device. For example, the terminal device 200 sends video samples and prompt word samples to the server 100. The server 100 receives the video samples and prompt word samples, trains a first video understanding model using the training method for the video understanding model provided in this application, and sends the first video understanding model to the terminal device 200. The terminal device 200 receives the first video understanding model and implements the video data processing method provided in this application based on the first video understanding model.

[0028] The first video understanding model trained through the embodiments of this application can be used in scenarios requiring video understanding, as shown in the following example: 1) In film and television content creation and automated narration scenarios, for example, content editors on video platforms or self-media creators can use the first video understanding model trained through the embodiments of this application to input long-form film and television dramas, variety shows, or documentary video materials. The first video understanding model can first generate a preliminary text including an overall story summary and a network of relationships between core characters, and then generate structured plot analysis content listed chronologically (e.g., "05:00-15:20, Character A and Character B confront each other in an abandoned factory; 15:20-18:00, Character C collects relevant evidence at the scene"). This can help creators quickly generate basic text or video timestamp guides for "film and television narration," greatly saving the time cost of manual summarization and organization, and improving the efficiency of secondary creation and content distribution.

[0029] 2) Intelligent office and meeting minutes generation scenarios, for example, the enterprise intelligent office platform, through the first video understanding model trained by the embodiment of this application, takes as input a long meeting video recording (integrating meeting screen and on-site audio) including multiple speakers, and generates a structured meeting minutes including a summary of the core meeting topics, the interaction relationship of the participants, and the minutes divided by time sequence or topic (such as "10:05-10:20, Li reported the data of the previous quarter; 10:20-10:45, the R&D and sales teams reached two consensuses on the progress plan for the next quarter"), thereby improving the accuracy of internal information flow and the efficiency of digital office.

[0030] 3) Intelligent protection scenarios, for example, the protection center uses the first video understanding model trained by the embodiment of this application to input long-term video data of key points. The first video understanding model can combine the on-screen actions with environmental sounds (such as collision sounds, shouting sounds) to generate a structured event development tracing report (such as "22:15:30, target object A appeared around the south gate warehouse; 22:17:10, accompanied by abnormal sounds, target object A came into contact with person B and left two minutes later"), thereby quickly extracting a logically clear event chain.

[0031] 4) In online education scenarios, for example, an online education platform can use the first video understanding model trained through the embodiments of this application to input long teaching videos delivered by teachers. The first video understanding model can comprehensively analyze the visual features of the teacher's audio and the blackboard / courseware to generate a macro-level knowledge outline for the course and output structured text transcription content including multiple teaching segments (e.g., "00:00-15:00 Knowledge Module 1: Explanation of the basic concepts of XX Law; 15:00-35:00 Knowledge Module 2: Derivation of typical examples"). This can automatically generate accurate accompanying lecture notes, timestamp directories, and key knowledge segment search tags for various course videos, providing students with fine-grained structured review materials, thereby improving the intelligence level of the online learning platform and the user experience.

[0032] In other embodiments, the training method for the video understanding model provided in this application can be implemented independently by a terminal device or a server. The terminal device 200 or the server 100 calls video samples and prompt word samples from its local database, obtains a trained first video understanding model using the training method provided in this application, and implements the video data processing method provided in this application based on the trained first video understanding model.

[0033] In other embodiments, the video data processing method provided in this application can be implemented by a terminal device or a server alone. The terminal device 200 or the server 100 trains a first video understanding model using the training method for the video understanding model provided in this application, and calls the first video understanding model for processing using the video data processing method provided in this application.

[0034] Here, server 100 can be a single server. In this case, the training method of the video understanding model and the video data processing method provided in this application embodiment can be implemented by the same server. Server 100 can also be a cluster of servers. In the case that server 100 is a server cluster, the training method of the video understanding model and the video data processing method provided in this application embodiment can be implemented by different servers. This application embodiment does not impose any limitations on this.

[0035] Taking the server used for training video understanding models as an example, see [link to relevant documentation]. Figure 2A , Figure 2A This is a first structural diagram of the server provided in an embodiment of this application. Figure 2A The server 100-1 shown includes at least one processor 110-1, memory 130-1, and at least one network interface 120-1. The various components in server 100-1 are coupled together via a bus system 140-1. It is understood that the bus system 140-1 is used to implement communication between these components. In addition to a data bus, the bus system 140-1 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2A The general designated all buses as Bus System 140-1.

[0036] The processor 110-1 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0037] The memory 130-1 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 130-1 may optionally include one or more storage devices physically located away from the processor 110-1.

[0038] The memory 130-1 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 130-1 described in this application embodiment is intended to include any suitable type of memory.

[0039] In some embodiments, memory 130-1 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0040] Operating system 131-1 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 132-1 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 120-1, such as Bluetooth, WiFi, and Universal Serial Bus (USB). In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2A A training device 133 for a video understanding model stored in memory 130-1 is shown. This device can be software in the form of programs and plugins, and includes the following software modules: a data processing module 1331 and a training module 1332. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0041] Taking the server's use for video data processing as an example, see [link to relevant documentation]. Figure 2B , Figure 2B This is a schematic diagram of the second structure of the server provided in an embodiment of this application. Figure 2B The server 100-2 shown includes at least one processor 110-2, memory 130-2, and at least one network interface 120-2. The various components in server 100-2 are coupled together via a bus system 140-2. It is understood that the bus system 140-2 is used to implement communication between these components. In addition to a data bus, the bus system 140-2 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2B All buses are labeled as Bus System 140-2. For detailed explanations of Processor 110-2 and Memory 130-2, please refer to the above text; they will not be repeated here.

[0042] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2B A processing apparatus 134 for video data stored in memory 130-2 is shown. This apparatus can be software in the form of programs and plugins, and includes the following software modules: a data processing module 1341 and a text generation module 1342. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0043] In some embodiments, the terminal device or server can implement the training method for the video understanding model and the video data processing method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0044] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the training method of the video understanding model and the video data processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0045] The following will describe the training method of the video understanding model provided in this application embodiment, with the server as the execution subject, using exemplary applications and implementations of the server provided in the embodiments of this application. See also Figure 3A , Figure 3AThis is a schematic diagram of the first process of the training method for the video understanding model provided in this application embodiment, which will be combined with Figure 3A The steps shown are explained.

[0046] In step 101, a first visual feature sample and a first audio feature sample of the video sample are obtained, as well as a text feature sample of a preset prompt word sample. The prompt word sample is used to instruct the first video understanding model to perform a video content transcription task for the video sample. The sequence length of the first visual feature sample is the same as the sequence length of the first audio feature sample.

[0047] Here, the prompt word sample refers to the pre-defined natural language constraint instructions and task guidance text, which are used as a query benchmark in the multimodal understanding framework to give the first video understanding model prior guidance semantics about "what video information needs to be extracted" and "what format to output".

[0048] Here, the video content transcription task refers to the task of parsing and mapping raw video multimodal data, which includes continuous dynamic images and complex acoustic mixed information, into a structured representation that follows a hierarchical structure (such as JSON format).

[0049] For example, the transcribed content output by the first video understanding model after performing a video content transcription task (such as the first predicted text sample) includes, but is not limited to: Metadata: Global attributes at the video level, such as title, list of characters (e.g., "Zhang XX"), video content summary, etc.

[0050] Scene information: location (e.g., "outdoor street"), environmental descriptions, time attributes (e.g., "daytime"), atmosphere (e.g., "relaxed and pleasant"), etc.

[0051] Event List: Atomic events arranged in chronological order for each scene. Each event can include a timestamp (e.g., "00:00"), participating characters, dialogue (or lines) (e.g., "Come here..."), actions, expressions (e.g., "anxious"), and audio cues (e.g., background music, sound effects).

[0052] In some embodiments, obtaining text feature samples of preset prompt word samples can be achieved by performing the following processing through a pre-configured text tokenizer: segmenting the prompt word samples to obtain multiple tokens; embedding and encoding each token to obtain the embedding feature of each token; positionally encoding each token to obtain the position feature of each token; fusing the embedding feature and position feature of each token to obtain the fused feature corresponding to the token; and combining the fused features corresponding to multiple tokens to form the text feature sample of the prompt word samples.

[0053] For example, segmenting the prompt word sample to obtain multiple tokens can be achieved in the following ways: using punctuation marks such as spaces, periods, and commas as tokenization markers to perform tokenization processing on the prompt word sample; or using a tokenizer based on Byte-Pair Encoding (BPE) or a tokenizer based on Byte-Level Byte Pair Encoding, etc. The embodiments of this application do not limit the specific tokenizer or segmentation method used to perform tokenization processing on the prompt word sample.

[0054] For example, embedding layers can be used to encode lexical units to obtain embedding features. For example, embedding layers can use embedding methods such as word embeddings, character embeddings, and subword embeddings to encode lexical units and obtain the embedding features of each lexical unit.

[0055] For example, positional encoding is performed on each word to obtain its positional features. This can be achieved by constructing an absolute positional encoding (NPI) or a relative positional encoding mechanism (such as Rotational Positional Encoding, RoPE). Taking alternating sine and cosine absolute positional encoding as an example, assuming the word's position sequence index is... The feature hidden layer dimension index is When in an even dimension (i.e. When ), the generated position feature component value is For odd-numbered dimensions (i.e.) The generated location feature component values ​​are This generates a positional feature of the same size as the embedded feature for each lexical unit.

[0056] For example, for each lexical unit, the embedding features and positional features of the lexical unit are fused to obtain the corresponding fused features. This can be achieved through element-wise addition. That is, the values ​​of the lexical unit embedding feature vector and the lexical unit positional feature vector are added along the same index dimension, so that a single vector simultaneously carries local contextual semantic attributes and global word order absolute positional constraints.

[0057] For example, combining the fusion features corresponding to multiple lexical units into a text feature sample for a prompt word sample can be achieved by concatenating the fusion features corresponding to each lexical unit along the sequence length dimension.

[0058] In some embodiments, see Figure 3B The acquisition of the first visual feature sample and the first audio feature sample of the video sample can be achieved through the following steps 201 to 203, which are explained in detail below.

[0059] In step 201, multiple image frames in the video sample are encoded to obtain a second visual feature sample for each image frame.

[0060] Here, multiple image frames in a video sample can be obtained by performing frame extraction on the video sample according to a preset frame extraction sampling rate (e.g., 1 frame / second, i.e., 1 FPS), converting the video sample into a sequence of image frames that exhibit a discrete arrangement pattern in physical time.

[0061] In some embodiments, encoding multiple image frames in a video sample to obtain a second visual feature sample for each image frame can be achieved by performing the following processing on each image frame: dividing the image frame to obtain multiple pixel blocks; sorting the multiple pixel blocks into a sequence of pixel blocks to be encoded; and encoding the sequence of pixel blocks to be encoded to obtain a second visual feature sample for the image frame.

[0062] For example, assuming the image frame size is 224×224, it is divided according to the set pixel block size (e.g., 16×16 pixel block size), resulting in (224 / 16)² = 196 pixel blocks. These multiple pixel blocks are then arranged (e.g., in order from left to right and from top to bottom) to form the pixel block sequence to be encoded.

[0063] For example, encoding the sequence of pixel blocks to be encoded to obtain the encoded features of the image frame can be achieved in the following way: performing embedding encoding on the sequence of pixel blocks to be encoded to obtain embedding features; performing attention encoding on the embedding features to obtain attention encoded features; and performing mapping on the attention encoded features to obtain the second visual feature samples of the image frame.

[0064] For example, embedding encoding can be performed on the sequence of pixel blocks to be encoded through an embedding layer to obtain embedded features. For instance, convolution operations can be performed on the sequence of pixel blocks to be encoded in the embedding layer, and position embedding can be performed on the convolutionally processed sequence of pixel blocks to be encoded to obtain embedded features. Before the embedding encoding process, each pixel block to be encoded in the sequence of pixel blocks to be encoded can be flattened into a one-dimensional vector. For example, for a 16×16×3 pixel block to be encoded, the length of the flattened vector is 768. Position encoding can be generated by a fixed algorithm, such as a combination of sine and cosine functions. For each position (e.g., the pixel position of each pixel in each pixel block to be encoded) pos and the dimension i of the feature, the position encoding of even-dimensional position uses the sine function, while the position encoding of odd-dimensional position uses the cosine function. The embodiments of this application do not limit the specific implementation of position encoding.

[0065] For example, before attention encoding, the embedded features can be normalized (e.g., layer normalization) to obtain normalized features. For the normalized features used for attention encoding, a linear transformation is first performed to generate three matrices: a query vector (Q), a key vector (K), and a value vector (V). For each attention head, the dot product of Q and K is calculated to obtain the attention score. Next, the attention score is normalized, for example, by applying a normalization function (such as the softmax function) to transform the attention score into a probability distribution. The attention probability distribution is then used to weight V to generate a new feature representation. Finally, the outputs of all attention heads are concatenated and a linear transformation is performed to obtain a rich representation that includes the correlation between different positions in the first normalized features, i.e., the attention-encoded features.

[0066] For example, attention-encoded features can be mapped using a feed-forward neural network (FFN) layer to obtain second visual feature samples of the image frame. The feed-forward neural network layer can, for example, adopt a multilayer perceptron (MLP) structure.

[0067] In other embodiments, image frames can be encoded (convolutional processing) using a convolutional neural network (CNN) to obtain second visual feature samples of the image frames. The embodiments of this application do not limit the specific implementation of encoding image frames.

[0068] In step 202, multiple audio segments in the video sample are encoded to obtain a second audio feature sample for each audio segment. The multiple image frames and multiple audio segments correspond one-to-one according to the temporal relationship.

[0069] Here, multiple audio segments in a video sample can be obtained by: demultiplexing the video sample to extract the audio track; and cutting the audio stream into multiple audio segments according to the physical time span synchronized with the image frame (e.g., in 1-second units).

[0070] Here, the one-to-one correspondence between multiple image frames and multiple audio segments according to their temporal relationship means that, on the physical playback timeline of the video sample, the timestamp or duration range corresponding to each image frame is completely consistent with the absolute physical time span of the corresponding audio segment.

[0071] In some embodiments, encoding multiple audio segments in a video sample to obtain a second audio feature sample for each audio segment can be achieved by performing the following processing on each audio segment: performing spectral transformation on the audio segment to obtain an acoustic spectrogram to be encoded; performing embedding encoding on the acoustic spectrogram to be encoded to obtain audio embedding features; performing attention encoding on the audio embedding features to obtain audio attention encoded features; and performing mapping processing on the audio attention encoded features to obtain a second audio feature sample for the audio segment.

[0072] For example, the audio segment can be spectrally transformed to obtain the acoustic spectrum to be encoded, which can be achieved as follows: the audio segment is resampled to a set global sampling rate (e.g., 16 kHz), and a short-time fourier transform (STFT) is performed on the audio segment using a set window length (e.g., 25 ms) and a window shift interval (e.g., 10 ms). Subsequently, the segment is mapped through a filter bank with a set number of large-scale Mel channels (e.g., 128 Mel-filterBanks) to obtain a log-Mel spectrogram, which is then used as the acoustic spectrum to be encoded.

[0073] For example, to obtain audio embedding features by embedding the acoustic spectrogram to be encoded, the following method can be used: Perform downsampling convolution processing on the acoustic spectrogram based on one-dimensional convolutional layers (e.g., passing it through a first convolutional network layer (e.g., kernel size 3, stride 1) and a second convolutional network layer (e.g., kernel size 3, stride 2), and combine this with an activation function (e.g., GELU activation function) to map the features to a set hidden layer feature dimension (e.g., 1280 dimensions). Due to the application of downsampling with a stride of 2, the sequence units generated for every 10 milliseconds of acoustic window shift are compressed by half in the temporal dimension. Therefore, for an audio segment with a duration of 1 second (containing 100 initial time steps), after downsampling convolution processing, it will be transformed into a one-dimensional sequence with a sequence length of 50 and a feature dimension of 1280. After obtaining the sequence obtained from the convolution processing, position embedding processing can be performed on the sequence to finally obtain the audio embedding features.

[0074] For example, the process of attention encoding and mapping audio embedding features is similar to image frame encoding and can be implemented using a multi-layer Transformer module (e.g., a structure with 32 stacked encoding layers and 20 attention heads). For the input audio embedding features, they are first normalized using Layer Normalization to obtain normalized features. Then, three matrices—query vector (Q), key vector (K), and value vector (V)—are generated. The dot product of Q and K is calculated to obtain the attention score, which is then transformed into a probability distribution using a softmax function. Weighted V is then applied to generate an output that encompasses the global audio contextual relevance. Finally, the attention output features are mapped through a feedforward neural network (FFN) layer to obtain a highly semantically abstract audio representation, i.e., the second audio feature sample of the audio segment.

[0075] In step 203, based on the temporal relationship, the second visual feature sample and the second audio feature sample within the same preset time interval are subjected to feature alignment processing to obtain the first visual feature sample and the first audio feature sample within the preset time interval.

[0076] Here, temporal relationship refers to the order in which image frames and audio segments occur on the original physical playback timeline of the video sample, as well as the overlapping binding relationship between the video stream and the audio stream with timestamp mapping association.

[0077] Here, the preset time interval refers to a pre-set duration window used to divide video samples into blocks and perform batch processing according to a fixed window step size in the temporal dimension (for example, setting a continuous "2 seconds" as a processing interval period).

[0078] In some embodiments, feature alignment processing is performed on second visual feature samples and second audio feature samples within the same preset time interval according to temporal relationships to obtain first visual feature samples and first audio feature samples within the preset time interval. This can be achieved by: determining the first total sequence length of the second visual feature samples within the preset time interval, and determining the second total sequence length of the second audio feature samples within the preset time interval; calculating the sequence length difference between the first total sequence length and the second total sequence length; if the first total sequence length is greater than the second total sequence length, supplementing the end of the last second audio feature sample within the preset time interval with a preset number of feature vectors corresponding to the sequence length difference, based on the sequence length difference, to obtain the first audio feature sample corresponding to the preset time interval; and combining the second visual feature samples within the preset time interval into the first visual feature sample corresponding to the preset time interval.

[0079] For example, determining the first total sequence length of the second visual feature samples within a preset time interval, and determining the second total sequence length of the second audio feature samples within a preset time interval, can be achieved in the following way: respectively, the sequence length of the second visual feature samples corresponding to all image frames included within the preset time interval is counted and used as the first total sequence length in the sequence length dimension, and the sequence length of the second audio feature samples corresponding to all audio segments included within the preset time interval is used as the second total sequence length.

[0080] For example, assuming the preset time interval is 2 seconds, and the sampling rate is 1 frame / second, the preset time interval includes 2 image frames. Assuming the sequence length of each image frame is 56, the total length of the first sequence is 56×2=112. At the same time, the preset time interval includes 2 audio segments with a corresponding duration of 1 second. Assuming the sequence length of each audio segment is 50, the total length of the second sequence is 50×2=100.

[0081] For example, the difference in sequence length between the total length of the first sequence and the total length of the second sequence can be achieved by calculating the absolute value of the difference between the total length of the first sequence and the total length of the second sequence.

[0082] For example, by subtracting the total length of the second sequence from the total length of the first sequence (112) obtained from statistics, the difference in sequence length between these two heterogeneous modes after their respective branch encodings due to the different sequence downsampling rates is calculated to be 112 - 100 = 12.

[0083] For example, when the total length of the first sequence is greater than the total length of the second sequence, based on the sequence length difference, a number of preset feature vectors corresponding to the sequence length difference are added to the end of the last second audio feature sample within the preset time interval to obtain the first audio feature sample corresponding to the preset time interval. This can be achieved by generating multiple preset feature vectors equal to the sequence length difference (for example, placeholder vectors with all zero values ​​can be used, and the hidden feature dimension of the preset feature vectors must be consistent with the original audio feature dimension). Then, through the vector concatenation operator, the multiple preset feature vectors are concatenated to the end of the temporal sequence of the second audio feature sample set within the preset time interval according to the temporal direction.

[0084] Continuing with the previous example, for the calculated difference, 12 all-zero feature vectors are generated (i.e., the number of preset feature vectors corresponding to the sequence length difference). Then, these 12 all-zero feature vectors are appended to the end of the two second audio feature samples within the preset time interval. After expansion, the sequence length of the audio features in the preset time interval changes from 100 to 100 + 12 = 112, the same as the total length of the first sequence.

[0085] For example, combining second visual feature samples within a preset time interval into first visual feature samples corresponding to the preset time interval can be achieved in the following way: according to the time sequence indicated by the temporal relationship, combine the second visual feature samples corresponding to each single frame image frame belonging to the same preset time interval to obtain the first visual feature samples corresponding to the preset time interval.

[0086] For example, the second visual feature sample of the first second image frame and the second visual feature sample of the second second image frame, which fall within the aforementioned consecutive "2 seconds" preset time interval, are combined and spliced ​​in chronological order to obtain the first visual feature sample corresponding to the "2 seconds" preset time interval.

[0087] By employing a dynamic feature alignment mechanism based on strict temporal relationships, the difference in length between audio and video feature sequences within the same time interval is calculated and targeted end-feature padding is performed. This avoids the sequence dimension mismatch problem caused by differences in sampling rate, network stride, or downsampling mechanism in heterogeneous multimodal data. It not only achieves precise and strict synchronization of visual and audio signals on the physical time axis and feature dimension, but also greatly reduces the computational difficulty of heterogeneous data fusion. This lays a unified data form foundation for subsequent video understanding models to perform efficient multimodal joint computation and high-quality structured content transcription inference.

[0088] See also Figure 3AIn step 102, a first thought process sample is generated based on a first visual feature sample, a first audio feature sample, and a text feature sample using a first video understanding model. The first visual feature sample and the first audio feature sample are injected into network layers of different depths in the first video understanding model.

[0089] Here, the first thought process sample (or first thought process text sample) refers to the natural language description (e.g., the Chain of Thought (CoT) text based on a large language model) generated by the first video understanding model before outputting the final conclusion or response (i.e., the first prediction text sample below), which reflects the internal intermediate analysis steps, cross-modal information combination logic, or causal reasoning chain. It is used to demonstrate to the outside world how the first video understanding model gradually understands the video images and sounds and draws conclusions.

[0090] Here, the different depths of the network layers in the first video understanding model refer to the hidden network module units in the structure of the first video understanding model (e.g., a large language model architecture composed of multiple Transformer decoder modules stacked together), which are located at different levels such as the front end (shallow layer), the middle layer, or the back end (deep layer). (For example, assuming that the first video understanding model contains a total of 32 network layers, the 8th, 16th, and 24th layers represent network layers at different depths.)

[0091] In some embodiments, a first thought process sample is generated based on a first visual feature sample, a first audio feature sample, and a text feature sample using a first video understanding model. This can be achieved by: fusing the first visual feature sample and the first audio feature sample to obtain a multimodal feature sample; selecting multiple target network layers of different depths from multiple network layers of the first video understanding model; inputting the text feature sample and the multimodal feature sample into the input layer of the first video understanding model, and inputting the multimodal feature sample into each target network layer respectively, so that the output layer of the first video understanding model outputs the first thought process sample.

[0092] For example, the fusion of the first visual feature sample and the first audio feature sample to obtain the multimodal feature sample can be achieved in the following way: the first visual feature sample and the first audio feature sample corresponding to the same preset time interval are transformed in feature dimension and mapped to a unified space that matches the input dimension of the first video understanding model. Then, they are superimposed pixel by pixel according to the corresponding element position to obtain the multimodal feature sample.

[0093] For example, fusing the first visual feature sample and the first audio feature sample to obtain multimodal feature samples can also be achieved in the following way: concatenating the first visual feature sample and the first audio feature sample along the channel dimension of the feature, and then mapping the concatenated high-dimensional matrix into a unified multimodal feature sample through a multilayer perceptron (MLP).

[0094] For example, selecting multiple target network layers of different depths from multiple network layers of the first video understanding model can be achieved in the following ways: Based on a preset equidistant sampling strategy, multiple target network layers of different depths can be selected from multiple network layers of the first video understanding model. For example, in a network structure with a total of 32 layers, a step size of 8 can be set, and layers 8, 16, 24, and 32 can be selected as multiple fusion target network layers in sequence. Alternatively, non-equidistant specification can be made based on the functional tendency of semantic reasoning. For example, mid-to-late network layers (such as selecting layers 20, 25, and 30) can be selected as target network layers to enhance the extraction of original audiovisual details by deep networks.

[0095] For example, inputting text feature samples and multimodal feature samples into the input layer of the first video understanding model can be achieved as follows: the fused multimodal feature samples are used as prefix tokens for visual and auditory cues, and concatenated at the front (or back) of the text feature sample sequence to obtain a concatenated sequence; after adding the corresponding positional encoding to the concatenated sequence, it is input into the input layer (i.e., the embedding layer) of the first video understanding model as the initial multimodal context features for the forward propagation inference of the first video understanding model.

[0096] In some embodiments, multimodal feature samples are input into each target network layer, which can be achieved by performing the following processing on each target network layer: using the output of the previous network layer as the query feature; performing feature mapping on the multimodal feature samples input into the target network layer to obtain key features and value features; performing attention encoding based on the query features, key features, and value features to obtain attention features; and fusing the attention features with the output of the previous network layer to obtain the output of the target network layer.

[0097] For example, the process described above of inputting multimodal feature samples into each target network layer is actually constructing a cross-modal attention mechanism. For instance, for a selected target network layer (e.g., layer 16), the hidden states output by the previous layer (i.e., layer 15) are used as query features (Q). The hidden states are features evolved from text and shallow multimodal information through multiple self-attention calculations. Simultaneously, "multimodal feature samples" that have not undergone attenuation processing by the previous 15 layers and retain the original audiovisual details are directly introduced through external skip connections, and high-fidelity key features (K) and value features (V) are generated through linear calculation. Then, attention calculation is initiated on the key features (K) using the query features (Q), dynamically extracting the audiovisual details required for the current inference stage and weighted and fused with the value features (V) to obtain the attention features. Finally, learnable gating parameters or residual structures (ResidualConnection) are introduced to supplement and fuse the attention features, which include cross-modal content, into the feedforward result originally calculated by the target network layer based on the output of the previous layer. This refreshes and strengthens the feature representation of the target network layer and continues to pass it to the next network layer.

[0098] For example, text feature samples and multimodal feature samples are input into the input layer of the first video understanding model, and the multimodal feature samples are input into each target network layer separately, so that the output layer of the first video understanding model outputs the first thought process sample. This can be achieved as follows: Multimodal feature samples and text feature samples are injected together as initial context into the input layer of the first video understanding model, initiating bottom-up feature parsing computation; as the information flow deepens layer by layer in the multi-layer network, when it reaches the target network layers of different pre-configured depths, the aforementioned cross-attention bypass is used to force a "refresh" and reintroduce the undistorted original multimodal feature samples. This layered, deep multimodal injection mechanism allows the model to not only rely on its internalized common-sense semantics for deduction when generating long sequences of logical reasoning text, but also continuously review and anchor the original video footage and original sound features at each key stage of reasoning, effectively avoiding the forgetting of modal details caused by long network stacking. Ultimately, the output layer of the first video understanding model (such as the vocabulary classification layer) performs autoregressive decoding on the vocabulary based on the deep network hidden layer representation after layer-by-layer calibration. It outputs a sequence of sentences with strict causal attributes and logical coherence word by word (or word unit), thus completely outputting the internal reasoning basis and steps, which is the first thought process sample.

[0099] For example, the first thought process sample includes a summary of the video sample's content and character relationship information, such as "[Plot] (content summary): a macro-level overview of the current scene's plot, such as 'Character A is tidying up at home in the early morning when the holographic projection butler issues a notification audio, reminding her that an external visitor has arrived'; [Character Relationships] (character relationship information): an analysis of character interactions and potential relationships, such as 'the unfamiliar interaction between the courier and the customer (Character A) suggests that their relationship is only temporary, and the courier's statement, "Please sign for this," indicates the core event focus of the current scene and the expected subsequent actions'."

[0100] By introducing a cross-modal attention mechanism into different depth target network layers of the model architecture, the undistorted multimodal feature samples are continuously injected and "refreshed" in a bypass jump manner. This guides the first video understanding model to generate first thought process samples that reflect the detailed internal reasoning steps and causal logic before outputting the final conclusion. This not only effectively avoids the problems of forgetting original audiovisual details and feature decay commonly encountered in deep network stacking and long sequence generation, but also suppresses the modal prior "illusion" that is easily induced in the first video understanding model during deep logical deduction. Furthermore, it forces the first video understanding model to accurately anchor the underlying objective audiovisual features when performing high-order semantic abstraction (such as extracting content summaries and character relationships). This significantly improves the transparency and robustness of multimodal complex reasoning, laying a highly deterministic reasoning foundation for outputting a final structured first predicted text sample with rigorous causal attributes and high logical self-consistency.

[0101] In step 103, a first predicted text sample is generated based on a first thought process sample using a first video understanding model. The first predicted text sample includes structured transcribed content of multiple events, and the first thought process sample includes a content summary and role association information of the video sample.

[0102] Here, the first predicted text sample refers to the conclusion text that the first video understanding model outputs after completing internal logical reasoning (i.e. generating the first thought process sample), which meets the preset task requirements (video content transcription task). It is the result of refining, extracting and arranging various factual information that occurs in the video in a specific format.

[0103] Here, multiple events refer to the smallest descriptive units on the timeline of a video sample, which are divided according to action switching, scene changes, or the start and end of dialogue, and have independent time spans and specific behavioral meanings (such as "Character A picks up a water glass", "Character B pushes open the door and enters", etc.).

[0104] Here, the structured transcription of events refers to the text blocks formed by organizing the aforementioned events and related attributes (such as start and end times, action descriptions, dialogue, and audio cues) according to a predetermined hierarchical logic or a machine-readable format (such as JSON, XML, or Markdown tables).

[0105] Here, the content summary of the video sample refers to a highly generalized summary of the storyline that occurs within the current processing time interval or the entire video segment, aiming to grasp the central idea of ​​the video from a macro perspective.

[0106] Here, role association information refers to the role identity and logical relationships (such as cooperation, kinship, etc.) derived from the character interactions, dialogue, and behavioral feedback in the video footage.

[0107] In some embodiments, the generation of a first predicted text sample based on a first thought process sample by a first video understanding model can be achieved by: inputting the thought text features, text feature samples, and multimodal feature samples of the first thought process sample into the input layer of the first video understanding model, and inputting the multimodal feature samples into each target network layer respectively, so that the output layer of the first video understanding model outputs the first predicted text sample.

[0108] For example, the first video understanding model can employ an auto-regressive decoding approach. After generating the first thought process sample, the generated first thought process sample is transformed into thought text features, and these thought text features are then concatenated with the initial text feature samples and multimodal feature samples. At this point, the deep injection mechanism within the first video understanding model (i.e., the cross-modal cross-attention calculation performed in each target network layer) remains active. This allows the first video understanding model to simultaneously reference both the "reasoning conclusion" and the "original audio and video facts" when outputting the final structured transcription content, thus ensuring that the transcription result is both logically consistent and rigorously adheres to the original video.

[0109] For example, the first predicted text sample can be represented in the following structured form: { Location: "Character A's home", Environment: "In the early morning, sunlight streams in through the floor-to-ceiling windows, creating a bright and airy atmosphere. The interior is decorated with high-tech holographic equipment, giving it a modern yet cool ambiance." "Event List": [ { "Timestamp": "00:01", Role: "Courier" Action: "Hands over the package and points to the electronic signature area", "Emoji": "Smiling, but slightly stiff" Dialogue: "Please sign here and accept the goods." }, { "Timestamp": "00:04", "Character": "Character A", Action: "Takes the pen and quickly swipes it across the screen, glancing at the flashing holographic prompt to the side." "Expression": "Anxious and slightly rushed" Dialogue: "Okay, thank you." } ] }

[0110] As can be seen, the first video understanding model achieves a highly structured encapsulation of video content through standard JSON format data. Location and environment provide a scene-level global description, while the event list precisely aligns the multimodal details of each character at different points in time. Through fine-grained characterization of timestamps, characters, actions, expressions, and dialogues, the first predicted text sample can fully reconstruct the complex audiovisual information in the video sample.

[0111] By using the first thought process sample, which includes macro-level content summaries and character association information, as the logical lead sequence for autoregressive decoding, and combining it with a continuously activated deep cross-modal feature injection mechanism, the first video understanding model can simultaneously and bidirectionally anchor high-order internal inference conclusions and underlying objective audiovisual facts when generating the final first predicted text sample. This not only accurately aligns and efficiently encapsulates the complex global scene attributes and fine-grained temporal events (such as precise timestamps, actions, dialogues, and expressions) in the video into a highly standardized machine-readable structure (such as JSON format), but also completely opens up a complete paradigm loop from multimodal underlying perception, internal causal reasoning to hierarchical content transcription. This effectively ensures the high degree of consistency in factual fidelity, detail granularity, and logical self-consistency of the final output structured prediction results.

[0112] In step 104, the first predicted text sample is divided into multiple predicted text segment samples. Based on the multiple predicted text segment samples and at least one real text segment corresponding to the multiple predicted text segment samples, a first reward value is determined, and based on the first reward value, the parameters of the first video understanding model are updated.

[0113] In some embodiments, dividing the first predicted text sample into multiple predicted text fragment samples can be achieved by parsing the text structure or preset delimiter of the first predicted text sample, and dividing the first predicted text sample into multiple predicted text fragment samples according to the semantic event boundary, timestamp marker or scene switching node indicated by the text structure or preset delimiter.

[0114] For example, since the first predicted text sample is structured data (such as JSON format), various segmentation strategies can be flexibly adopted when dividing the data into segments: for example, it can be segmented according to the video's timestamp (e.g., using timestamp key-value pairs in the JSON sequence to extract time boundaries, and taking the transcribed content corresponding to each minute or each specific time period as a predicted text segment sample); it can be segmented according to the semantic structure of the plot (e.g., parsing the array of elements in the events list in the JSON, and identifying each round of independent dialogue, each character's micro-expression, or each scene description block as a predicted text segment sample); or it can use specific delimiter words (such as explicit truncation characters like "<|event_sep|>") output by the first video understanding model when generating text to directly perform string segmentation, thereby accurately obtaining multiple predicted text segment samples.

[0115] The embodiments of this application do not limit the specific implementation of dividing the first predicted text sample into multiple predicted text fragment samples.

[0116] In some embodiments, real text includes at least one real text fragment.

[0117] Here, ground truth refers to pre-annotated structured descriptive text data that corresponds to the objective content of the video samples. It serves as the ground truth (GT) for training the first video understanding model and provides a standard for "what is the correct transcription result" when calculating values ​​such as reward or loss functions.

[0118] It should be noted that the granularity of the predicted text segment samples can differ from the granularity of the actual text segments. The segmentation logic of each event favored by the first video understanding model when generating the first predicted text samples (e.g., a micro-level frame-by-frame / sentence-by-sentence parsing) may differ in the scope and granularity of events covered by the inductive method used when annotating actual text manually or through annotators (e.g., using LLM as an annotator).

[0119] For example, combining the aforementioned structured JSON data, assuming that the prediction stage adopts a finer semantic structure segmentation granularity, then a predicted text fragment sample may only include a single-step micro-interaction for a certain role (such as the single event with the timestamp "00:01" mentioned above), such as the "smiling" expression, the action of "handing over the package and pointing to the electronic signature" shown by the single role of "courier" during this time, and the corresponding dialogue "please sign here, please receive the goods", thus forming an independent sample unit.

[0120] In contrast, real-world text fragments may employ coarser-grained event-level summaries during annotation. Therefore, the content covered by a real-world text fragment may not be a single action of a particular character, but rather an integration of the interaction between the two parties during that time as a complete macro-semantic block. For example, it may include a complete scene event (such as the two events corresponding to the timestamps "00:01" and "00:04" mentioned above, which constitute a whole event) describing "the courier delivers the package to the door and asks the recipient to sign for it. Character A quickly and somewhat anxiously completes the signing on the holographic electronic screen and expresses gratitude," thus serving as a coarser-grained real-world text fragment.

[0121] As can be seen, due to the potential differences in the granularity (micro-level single-step information versus macro-level event summary) between predicted text fragment samples and real text fragments, a single predicted text fragment sample may correspond to a part of a real text fragment, or multiple predicted text fragment samples may correspond to a common real text fragment. This diversity of granularity is a key reason why subsequent temporal matching and cross-evaluation using the intersection-union ratio (IUU) of time intervals are necessary.

[0122] In some embodiments, see Figure 3C The first reward value is determined based on multiple predicted text fragment samples and at least one real text fragment corresponding to the multiple predicted text fragment samples. This can be achieved through the following steps 301 to 304, which are explained in detail below.

[0123] In step 301, the temporal intersection-union ratio (CIU) of each predicted text segment sample is determined based on the first time interval corresponding to each predicted text segment sample and the second time interval corresponding to each real text segment.

[0124] Here, the first time interval corresponding to the predicted text fragment sample refers to the start and end time range of the semantic action or event described by the predicted text fragment sample on the timeline of the video sample. For example, the first time interval can be determined based on the timestamps recorded in the structured transcribed content (e.g., a continuous time period starting from the timestamp of the current event and ending at the timestamp of the next event or before the scene switching node).

[0125] Here, the second time interval corresponding to the real text fragment refers to the start and end time range of the events or content covered by the real text fragment on the video sample timeline in the labeled data annotated manually or by annotators (such as using MLLM as an annotator).

[0126] In some embodiments, the temporal intersection-union ratio (CIU) for each predicted text segment sample is determined based on a first time interval corresponding to each predicted text segment sample and a second time interval corresponding to each real text segment. This can be achieved by performing the following processing on each predicted text segment sample: combining the first time interval corresponding to the predicted text segment sample with the second time interval corresponding to each real text segment to construct at least one temporal evaluation pair; for each temporal evaluation pair, calculating the temporal intersection length and the temporal union length of the first and second time intervals; and for each temporal evaluation pair, calculating the ratio between the temporal intersection length and the temporal union length as the temporal intersection-union ratio corresponding to the temporal evaluation pair.

[0127] Here, Temporal Intersection over Union (tIoU) is a quantitative indicator used to measure the degree of overlap between two time intervals (i.e., the first time interval and the second time interval) in the time series dimension.

[0128] For example, combining the first time interval corresponding to each predicted text segment sample with the second time interval corresponding to each real text segment to construct at least one temporal evaluation pair means calculating pairwise combinations between all predicted text segment samples and all real text segments. For instance, assuming the first predicted text sample is divided into M predicted text segment samples, and the real text includes N real text segments, by traversing and cross-calculating these M first time intervals and N second time intervals, M×N temporal evaluation pairs can be constructed. By calculating the tIoU values ​​of these M×N temporal evaluation pairs, a temporal intersection-union matrix (tIoU matrix) of size M×N is obtained. Each element in the temporal intersection-union matrix represents the degree of overlap between a predicted text segment sample and a real text segment in the time dimension.

[0129] For example, if two predicted text segment samples (denoted as P1 and P2) and two real text segments (denoted as G1 and G2) are obtained, P1 will be combined with G1 and G2 respectively, and P2 will be combined with G1 and G2 respectively, thus constructing four temporal evaluation pairs. After calculating the tIoU values ​​of these four pairs, a 2×2 temporal intersection-union matrix can be formed.

[0130] Continuing from the example above, suppose there are two predicted text fragment samples with a first time interval of [1,4] (corresponding to "the courier hands over the package", denoted as P1) and a predicted text fragment sample with a first time interval of [4,7] (corresponding to "role A signs for the package", denoted as P2); while the real text uses macro-labeling, and there is only one real text fragment with a second time interval of [0,6] (corresponding to "the courier delivers the package and role A signs for the package", denoted as G1).

[0131] At this point, two time series evaluation pairs will be constructed: For evaluation pair 1 (P1 and G1, i.e. [1,4] and [0,6]), the temporal intersection length is 3, the union length is 6, and the tIoU is 0.5; For the evaluation pair 2 (P2 and G1, i.e. [4,7] and [0,6]), the intersection range is [4,6] with a length of 2, the union range is [0,7] with a length of 7, and the tIoU is approximately 0.28.

[0132] Thus, a time-series intersection-union matrix of shape 2×1 is obtained [[0.5],[0.28]].

[0133] In step 302, based on the temporal intersection-union ratio corresponding to each predicted text segment sample, at least one real text segment that has a matching relationship with each predicted text segment sample is determined.

[0134] In some embodiments, determining at least one real text segment that matches each predicted text segment sample based on the temporal intersection-union ratio (TUIR) can be achieved as follows: traversing the data in descending order of TUIR; if no matching relationship is established between the predicted text segment sample and the real text segment corresponding to the TUIR, then establishing a matching relationship between the predicted text segment sample and the real text segment corresponding to the TUIR, and determining the predicted text segment sample corresponding to the TUIR as the predicted text segment sample with established matching relationship, and determining the real text segment corresponding to the TUIR as the real text segment with established matching relationship. After traversal, for the predicted text fragment samples without established matching relationships, real text fragments that satisfy the preset temporal overlap condition with the predicted text fragment samples without established matching relationships are selected from the real text fragments with established matching relationships, and a matching relationship is established between the predicted text fragment samples without established matching relationships and the selected real text fragments; for the real text fragments without established matching relationships, predicted text fragment samples that satisfy the temporal overlap condition with the real text fragments without established matching relationships are selected from the predicted text fragment samples with established matching relationships, and a matching relationship is established between the real text fragments without established matching relationships and the selected predicted text fragment samples.

[0135] Here, the matching relationship refers to the alignment mapping between predicted text fragment samples and real text fragments in terms of time and event semantics. Establishing a matching relationship means that when determining the first reward value, the two parties with a matching relationship will be regarded as "predicted descriptions" and "real descriptions" of the same video content, and used to calculate the similarity or consistency in dimensions such as text content and action entities. Due to differences in granularity, this matching relationship not only includes one-to-one mappings, but also accommodates many-to-one or one-to-many mappings.

[0136] For example, traversing the temporal intersection-union ratio (tIoU) from high to low, if no match is established between the predicted text fragment sample and the real text fragment corresponding to the temporal intersection-union ratio, a match is established between the predicted text fragment sample and the real text fragment, and the predicted text fragment sample corresponding to the temporal intersection-union ratio is identified as a predicted text fragment sample with an established match, and the real text fragment corresponding to the temporal intersection-union ratio is identified as a real text fragment with an established match. This refers to a process of establishing a globally optimal "one-to-one" backbone match based on a greedy strategy. For example, all tIoU values ​​in the temporal intersection-union ratio matrix are globally sorted in descending order, and each value is examined starting from the maximum value. If the predicted fragment and the real fragment corresponding to the current maximum tIoU have not been assigned before (i.e., both are in an idle state), they are bound as an optimal match, and both are marked as "matched". This process is repeated to construct the basic temporal alignment relationship.

[0137] For example, if we have predicted text fragment samples P1 and P2, and real text fragments G1 and G2, and tIoU(P1-G1)=0.8, tIoU(P2-G2)=0.6, and tIoU(P1-G2)=0.4, in descending order traversal, we first traverse to the maximum value of 0.8, determining that neither P1 nor G1 matches, thus establishing a matching relationship between P1 and G1; continuing to traverse to 0.6, determining that neither P2 nor G2 matches, thus establishing a matching relationship between P2 and G2; at this point, the basic one-to-one matching relationship is completed.

[0138] For example, after traversal, for predicted text fragment samples without established matching relationships, real text fragments that satisfy preset temporal overlap conditions with the predicted text fragment samples without established matching relationships are selected from the real text fragments with established matching relationships. Establishing a matching relationship between the predicted text fragment samples without established matching relationships and the selected real text fragments refers to a supplementary matching mechanism for "many-to-one" mapping scenarios (i.e., multiple predicted text fragment samples correspond to one real text fragment). When some unmatched predicted text fragments "fail" in the first round of greedy matching due to slightly smaller tIoU values ​​appear, it is allowed to associate them with those real text fragments that have already been matched.

[0139] For example, suppose the temporal intersection-union ratio (tIoU(P1-G1)) is 0.5 and tIoU(P2-G1) is 0.28. In the first round of greedy matching, since 0.5 is the maximum, P1 and G1 establish a matching relationship and are marked as matched. At this time, P2 becomes a predicted text segment sample without a matching relationship, and the overlap between P2 (interval [4,7]) and the matched G1 (interval [0,6]) is checked. The intersection of the two is [4,6] with a length of 2. Assuming the preset temporal overlap condition is "the proportion of the intersection length to the total length of the predicted time interval is greater than 50%", the total length of P2 is 3, and the proportion is 2 / 3 (about 66.7%) > 50%, which meets the condition. Therefore, a matching relationship is established between the unmatched P2 and the matched G1. Finally, P1, P2 and G1 all establish matching relationships.

[0140] For example, for real text fragments without established matching relationships, a supplementary matching mechanism is used to select predicted text fragment samples that satisfy the temporal overlap condition with the real text fragments without established matching relationships from the predicted text fragment samples with established matching relationships. This establishes a matching relationship between the real text fragments without established matching relationships and the selected predicted text fragment samples. This is a supplementary matching mechanism for "one-to-many" mapping scenarios (i.e., one predicted text fragment sample corresponds to multiple real text fragments). If unmatched real text fragments exist, those predicted text fragment samples that have already been matched will be detected. As long as the preset temporal overlap condition is met, this predicted text fragment sample is allowed to be "reused" for matching.

[0141] For example, suppose the predicted text fragment sample P3 (corresponding to the interval [0,9]), the real text fragment G3 (interval [0,5]), and the real text fragment G4 (interval [6,9]), with tIoU(P3-G3)=0.56 and tIoU(P3-G4)=0.33. In the first round of greedy matching based on temporal intersection-union ratio (TUIR) from high to low, P3 and G3 establish a matching relationship first because P3 and G3 have the highest tIoU (0.56), and are subsequently marked as matched. G4 becomes the real text fragment without a matching relationship. After entering the supplementary matching mechanism, for the unmatched G4, the matched predicted text fragment sample P3 is retrieved. It is calculated that the intersection range of the unmatched G4 (interval [6,9]) and the matched P3 (interval [0,9]) is [6,9], and the intersection length is 3. Assuming the preset temporal overlap condition is "the length of the temporal intersection is greater than 50% of the total length of the real text segment," the total length of G4 is 3 (9-6=3), and the overlap ratio is 3 / 3 (100%), which is greater than 50%, thus satisfying the condition. Therefore, a matching relationship is established between the unmatched real segment G4 and the matched predicted segment P3. Ultimately, the predicted text segment sample P3 is associated with both real text segments G3 and G4.

[0142] By combining a greedy matching strategy based on the global maximum temporal intersection-union ratio with a bidirectional supplementary reuse mechanism based on preset temporal overlap conditions, this approach not only prioritizes the establishment of a basic "one-to-one" global alignment mapping with the highest confidence but also significantly improves temporal inclusiveness. It is compatible with complex "many-to-one" and "one-to-many" mapping scenarios arising from differences in event segmentation granularity between predicted text fragment samples and actually labeled text fragments. This hierarchical and progressive flexible matching architecture effectively avoids the omission or erroneous discarding of valid event fragments due to stringent single constraints, achieving seamless alignment of local micro-actions and macro-time spans on the timeline. This lays the foundation for subsequent multi-dimensional semantic similarity comparisons at the execution depth and the extraction of reliable reward evaluation signals.

[0143] In step 303, multiple predicted role information from the first predicted text sample and multiple real role information from the real text are obtained. Based on the multiple predicted role information and the multiple real role information, a role mapping relationship is established.

[0144] Here, predicted role information refers to the subject object attributes identified from the first predicted text sample that participate in the video event, including the subject's title (corresponding to the object name below), identity attributes (corresponding to the name type below), and quantity characteristics (corresponding to the singular or plural identifier of the object name below).

[0145] Here, real character information refers to the attributes of the actual main objects in the video extracted from standard label data provided by human annotation or annotators (such as MLLM).

[0146] For example, each predicted role or each real role can be represented as structured data, such as: {name (object name): "Zhang Mou", type (name type): "specific (corresponding to the role name type below)", count (singular / plural identifier): "singular (singular)"} represents a specific role named Zhang Mou; {name: "students", type: "role (group name type)", count: "plural (plural)"} represents a group of roles with the identity of "student".

[0147] In some embodiments, obtaining multiple predicted role information in the first predicted text sample can be achieved by: using a pre-trained Named Entity Recognition (NER) model or a multimodal large model (MLLM) guided by a prompt word to parse the first predicted text sample, extracting all subject objects mentioned in the first predicted text sample, and labeling the names, types, and number of references of the objects in the text.

[0148] In some embodiments, obtaining information about multiple real characters in real text can be achieved by directly reading the character field from structured real text, or by extracting entities from unstructured real text fragments to obtain real character information.

[0149] In some embodiments, each predicted role information and each real role information includes an object name, a name type corresponding to the object name, and a singular / plural identifier indicating whether the object name is singular or plural; the role mapping relationship includes a first mapping relationship and a second mapping relationship; the role mapping relationship is established based on multiple predicted role information and multiple real role information, which can be achieved in the following way: based on the name type, the predicted role information and real role information with the name type of role name are matched by object name to obtain a first mapping relationship; the predicted role information and real role information without a first mapping relationship are paired to obtain at least one candidate mapping pair, and candidate mapping pairs with semantic conflicts are eliminated, wherein a semantic conflict means that there is a semantic contradiction between the object name corresponding to the predicted role information and the object name corresponding to the real role information in the candidate mapping pair; for the candidate mapping pairs obtained after eliminating semantic conflicts, the mapping score corresponding to the candidate mapping pair is determined based on the predicted text fragment corresponding to the predicted role information and the real text fragment corresponding to the real role information in the candidate mapping pair; the second mapping relationship is obtained based on the candidate mapping pair whose mapping score satisfies a first preset condition and whose singular / plural identifier in the predicted role information and the singular / plural identifier in the real role information satisfy a second preset condition.

[0150] Here, the first mapping relationship refers to the explicit correspondence established based on explicit name alignment.

[0151] Here, the second mapping relationship refers to the implicit correspondence established after deep reasoning based on semantic similarity, temporal overlap, and referential logical consistency, excluding the first mapping relationship.

[0152] Here, the semantic contradiction between the object name corresponding to the predicted role information and the object name corresponding to the real role information means that the two object names are obviously mutually exclusive or contradictory in terms of objective common sense, basic attributes (such as gender, species, age group, etc.) or ontology knowledge base classification, which makes it logically impossible for them to point to the same entity in the video.

[0153] For example, if the object name in the predicted role information is "grandma" (indicating an elderly woman), while the object name in the actual role information is "little boy" (indicating a young male), this gender and age contrast immediately constitutes a semantic contradiction.

[0154] For example, based on name type, the predicted role information and the real role information with the name type of "role name" are matched by object name to obtain the first mapping relationship. This can be achieved in the following way: filter out objects in both the predicted role information and the real role information whose name type is marked as "role name type" (i.e. pointing to a specific individual), and directly establish the explicit correspondence between them by verifying whether the object names of the two are completely consistent or belong to the same entity as synonyms.

[0155] For example, in the first predicted text sample, a predicted role information is identified with the object name "Zhang Mou" and the name type is "role name type". At the same time, there is also a real role information with the object name "Zhang Mou" and the name type is "role name type" in the real text. Since the two have the same type and completely identical names, a first mapping relationship is established between the two.

[0156] For example, the predicted role information that has not established a first mapping relationship is paired with the real role information to obtain at least one candidate mapping pair, and candidate mapping pairs with semantic conflicts are eliminated. This means that for the remaining predicted role information and real role information that cannot be directly matched by object name, a full permutation exhaustive combination is performed to form potential candidate mapping pairs. Then, according to the preset common sense rules or relation dictionary, combinations that are not logically valid (such as completely mutually exclusive identities, species mismatch, etc.) are deleted, thereby reducing the complexity of subsequent mapping score calculation.

[0157] For example, in data where no first mapping relationship has been established, the predicted role information includes the role with the object name "Doctor," while the actual role information includes the roles with the object names "Patient" and "Doctor-Patient." This results in two candidate mapping pairs: "Doctor-Patient" and "Doctor-Inpatient." If, according to preset rules, "Doctor" and "Patient" are determined to be roles with opposing identities in the current scenario (i.e., semantic conflict), then the "Doctor-Inpatient" pair is discarded, and only the "Doctor-Patient" pair is retained as a candidate mapping pair.

[0158] By matching object names based on name type to prioritize the establishment of the first mapping relationship, and then pairing the predicted role information without the first mapping relationship with the real role information, the candidate mapping pairs with semantic conflicts are eliminated in advance. This not only quickly establishes the explicit correspondence of specific individuals, but also effectively eliminates erroneous referential combinations that are mutually exclusive or contradictory in objective common sense or basic attributes. This significantly reduces the complexity of subsequent mapping score calculations and lays a reliable foundation for accurately and logically determining the second mapping relationship.

[0159] In some embodiments, the mapping score corresponding to the candidate mapping pair is determined based on the predicted text fragment corresponding to the predicted role information and the real text fragment corresponding to the real role information within the candidate mapping pair. This can be achieved by: calculating the text similarity between the predicted text fragment corresponding to the predicted role information and the real text fragment corresponding to the real role information; obtaining the temporal intersection-union ratio between the predicted text fragment corresponding to the predicted role information and the real text fragment corresponding to the real role information; and fusing the text similarity and the temporal intersection-union ratio to obtain the mapping score corresponding to the candidate mapping pair.

[0160] For example, calculating the text similarity between the predicted text fragment corresponding to the predicted character information and the real text fragment corresponding to the real character information can be achieved as follows: Extract the action text (such as "turn right" or "run") and dialogue text (such as "good afternoon") of the predicted character information in the corresponding predicted text fragment, as well as the action text and dialogue text of the real character information in the corresponding real text fragment; calculate the action semantic similarity between the action text in the predicted text fragment and the action text in the real text fragment using a pre-trained language model, and calculate the dialogue semantic similarity between the dialogue text in the predicted text fragment and the dialogue text in the real text fragment; use the action semantic similarity and dialogue semantic similarity as the text similarity between the predicted text fragment corresponding to the predicted character information and the real text fragment corresponding to the real character information.

[0161] For example, the mapping score corresponding to the candidate mapping pair can be obtained by fusing text similarity and temporal intersection-union ratio. This can be achieved by: weighting and summing the text similarity (including two dimensions: action semantic similarity and dialogue semantic similarity) that measures behavioral consistency with the temporal intersection-union ratio that measures the degree of overlap in the appearance time of the characters according to preset weight coefficients; and finally correcting any possible logical deviations through a penalty factor to obtain the final mapping score.

[0162] For example, the calculation of the corresponding mapping score for a candidate mapping can be expressed by formula (1): (1) in, This is the temporal intersection-over-union (IoU) between the predicted text fragments corresponding to the predicted character information and the real text fragments corresponding to the real character information. For semantic similarity of dialogue; For action semantic similarity; and These are preset weighting coefficients; This is a preset penalty factor.

[0163] In some embodiments, a second mapping relationship is determined based on candidate mapping pairs that satisfy a first preset condition in mapping scores and a second preset condition in singular / plural identifiers in predicted role information and real role information. This can be achieved by: using candidate mapping pairs whose mapping scores are greater than a preset matching score threshold as candidate mapping pairs that satisfy the first preset condition; using candidate mapping pairs that do not trigger a preset referential conflict condition as candidate mapping pairs that satisfy the second preset condition, wherein a referential conflict condition refers to a situation where, between the predicted role information and real role information of a candidate mapping pair, one party's singular / plural identifier is plural, and the other party's name type is a role name type or its singular / plural identifier is singular; and determining the second mapping relationship based on candidate mapping pairs that satisfy the first and second preset conditions.

[0164] Here, a referential conflict condition refers to a logically mutually exclusive matching error between predicted role information and actual role information in the candidate mapping pair, in terms of the number of objects or the uniqueness of identities. That is, the second mapping relationship allows mapping from "singular identity to specific name" (e.g., the common noun "boy" to the proper noun "Zhang"), but prohibits mapping from "specific name to plural identity" (e.g., "Zhang" to the group "boys") or "singular identity to plural identity" (e.g., the singular "boy" to the plural "boys").

[0165] For example, determining the second mapping relationship based on candidate mapping pairs that meet the first and second preset conditions refers to employing a dual filtering mechanism combining "score admission" and "logical verification" after quantifying the mapping scores of the candidate mapping pairs. First, only high-confidence candidate mapping pairs with mapping scores greater than a preset matching score threshold (i.e., meeting the first preset condition) are retained. Next, among these high-scoring candidate mapping pairs, the singular / plural identifiers and name types of the predicted role information and the real role information are further verified, and incorrect alignment combinations that trigger referential conflict conditions are eliminated. Finally, only candidate combinations that simultaneously meet the criteria at the spatiotemporal action semantic level and conform to logical judgments at the referential quantity level are formally established as implicit second mapping relationships.

[0166] For example, suppose the mapping scores of two candidate mapping pairs are both greater than a preset matching score threshold, meaning both satisfy the first preset condition. In the first candidate mapping pair, the predicted role information corresponds to the object name "boys" (single / plural identifier is plural), while the corresponding real role information corresponds to the object name "Zhang Mou" (name type is role name type, singular / plural identifier is singular). Since one is plural and the other is a role name type, a referential conflict condition is triggered, and the second preset condition is not met. Therefore, the first candidate mapping pair will not be determined as the second mapping relationship.

[0167] In the second candidate mapping pair, the predicted role information corresponds to the object name "boy" (single / plural identifier is singular), and the actual role information corresponds to the object name "Zhang Mou" (name type is role name type, singular / plural identifier is singular). At this time, the second candidate mapping pair belongs to the allowed mapping of "singular identity to specific person name", and the referential conflict condition is not triggered, thus satisfying the second preset condition, and the second candidate mapping pair is determined as the second mapping relationship.

[0168] By using a mapping score greater than a preset matching score threshold as the first preset condition and combining it with the condition of not triggering a referential conflict as the second preset condition, a dual filtering mechanism combining "score admission" and "logical verification" is implemented on candidate mapping pairs. This not only ensures that the retained candidate mapping pairs have high confidence in the spatiotemporal action semantic level, but also effectively intercepts and eliminates matching errors in singular / plural identifiers or name types where the number of objects and the uniqueness of the identity are logically mutually exclusive, thus rigorously and accurately establishing the implicit second mapping relationship.

[0169] In step 304, a first reward value is calculated based on the role mapping relationship, multiple predicted text fragment samples, and at least one real text fragment that matches each predicted text fragment sample.

[0170] In some embodiments, the calculation of a first reward value based on role mapping relationships, multiple predicted text fragment samples, and at least one real text fragment matching each predicted text fragment sample can be achieved as follows: For predicted text fragment samples and real text fragments with matching relationships, based on role mapping relationships, calculate the semantic similarity of multiple dimensions between the predicted text fragment samples and the matched real text fragments, and determine a first evaluation value based on the semantic similarity of multiple dimensions; For each of the preset multiple temporal intersection-union (CIU) thresholds, if the CIU between the predicted text fragment sample and the matched real text fragment is greater than or equal to the CIU threshold, and the first evaluation value is greater than or equal to a preset first evaluation value threshold, then the predicted text fragment sample is determined as a hit sample; Calculate the ratio between the number of hit samples corresponding to each CIU threshold and the total number of multiple predicted text fragment samples, as a second evaluation value; Average the second evaluation values ​​corresponding to the multiple CIU thresholds respectively to obtain a third evaluation value; Combine the first evaluation value, the second evaluation value, and the third evaluation value to obtain the first reward value.

[0171] In some embodiments, both the predicted text fragment sample and the real text fragment include at least a role field, a dialogue field, an action field, and an environment field. Based on the role mapping relationship, the semantic similarity between the predicted text fragment sample and the matched real text fragment is calculated in multiple dimensions, which can be achieved as follows: based on the role mapping relationship, the role field of the predicted text fragment sample is compared with the role field of the real text fragment, and the role semantic similarity is determined based on the comparison result; the dialogue semantic similarity between the dialogue field of the predicted text fragment sample and the dialogue field of the real text fragment is calculated; the action semantic similarity between the action field of the predicted text fragment sample and the action field of the real text fragment is determined through a pre-set language model; and the environment semantic similarity between the environment field of the predicted text fragment sample and the environment field of the real text fragment is determined through a language model.

[0172] Here, the "role field" corresponds to the "role" in the "event list" in the structured form example of the first predicted text sample above (e.g., the corresponding role name is "courier"); the "dialogue field" corresponds to the "dialogue" in the "event list" (e.g., the corresponding dialogue description is "Please sign here, please accept the goods"); the "action field" corresponds to the "action" in the "event list"; and the "environment field" corresponds to the "environment" (e.g., the corresponding environment description is "In the early morning, sunlight shines through the floor-to-ceiling windows, making the room bright. The interior is decorated with high-tech holographic equipment, and the overall atmosphere is modern and deserted").

[0173] For example, based on the role mapping relationship, the role field of the predicted text fragment sample is compared with the role field of the real text fragment, and the role semantic similarity is determined based on the comparison result. This can be achieved in the following way: Based on the preset role mapping table, it is determined whether the role name corresponding to the role field in the predicted text fragment sample matches the role name corresponding to the role field in the real text fragment. If they match (i.e., they are consistent), the role semantic similarity is determined to be 1; otherwise, the role semantic similarity is determined to be 0.

[0174] For example, the dialogue semantic similarity between the dialogue fields of the predicted text fragment sample and the dialogue fields of the real text fragment can be achieved as follows: extract the dialogue corresponding to the dialogue fields in the predicted text fragment sample and the real text fragment, calculate the text difference between the two using the Levenshtein Distance algorithm, and normalize the text difference to obtain a dialogue semantic similarity in the range of 0.0 to 1.0.

[0175] For example, by using a pre-set language model, the semantic similarity between the action fields of the predicted text fragment sample and the action fields of the real text fragment can be determined as follows: input the action descriptions in the action fields of the predicted text fragment sample and the action descriptions in the action fields of the real text fragment into the pre-set language model, use the language model to evaluate the similarity between the two action descriptions in terms of behavioral logic, and output a score value between 0.0 and 1.0 as the semantic similarity of the action.

[0176] For example, determining the environmental semantic similarity between the environment fields of the predicted text fragment sample and the environment fields of the real text fragment using a language model can be achieved as follows: input the environment descriptions in the environment fields of the predicted text fragment sample and the environment descriptions in the environment fields of the real text fragment into a pre-set language model, use the language model to evaluate the similarity between the two environment descriptions, and output a score value between 0.0 and 1.0 as the environmental semantic similarity.

[0177] It should be noted that although the above embodiments only list the calculation process for four dimensions: role, dialogue, action, and environment, in practical applications, the dimensions for calculating semantic similarity are not limited to the fields described above. All fields in the aforementioned structured examples (such as "location", "expression", etc.) can be included in the comparison and calculation of similarity as needed.

[0178] In some embodiments, the first evaluation value is determined based on semantic similarity across multiple dimensions, which can be achieved by fusing semantic similarity of roles, semantic similarity of dialogues, semantic similarity of actions, and semantic similarity of the environment to obtain the first evaluation value.

[0179] For example, the first evaluation value can be obtained by fusing character semantic similarity, dialogue semantic similarity, action semantic similarity, and environment semantic similarity. This can be achieved by: obtaining the pre-defined weight coefficients for character semantic similarity, dialogue semantic similarity, action semantic similarity, and environment semantic similarity respectively; multiplying each of these four similarities by their respective weight coefficients, and summing the products (or directly taking the arithmetic mean of the four similarities) to obtain the first evaluation value.

[0180] In some embodiments, for each of the preset temporal intersection-union (CIU) thresholds, if the CIU between the predicted text fragment sample and the matched real text fragment is greater than or equal to the CIU threshold, and the first evaluation value is greater than or equal to the preset first evaluation value threshold, then the predicted text fragment sample is determined as a hit sample. The ratio between the number of hit samples corresponding to each CIU threshold and the total number of multiple predicted text fragment samples is calculated as the second evaluation value. This can be achieved by: counting the total number of hit samples that satisfy the above two conditions (the CIU between the predicted text fragment sample and the matched real text fragment is greater than or equal to the CIU threshold, and the first evaluation value is greater than or equal to the preset first evaluation value threshold) at each CIU threshold, and obtaining the total number of predicted text fragment samples; dividing the calculated total number of hit samples by the total number of predicted text fragment samples, thus obtaining the second evaluation value corresponding to the current CIU threshold.

[0181] For example, given a temporal intersection-union ratio (tIoU) threshold (e.g., tIoU=0.5), iterate through all predicted text fragment samples and matching results, and select the predicted text fragment samples that satisfy both "temporal intersection-union ratio ≥ 0.5" and "first evaluation value ≥ preset first evaluation value threshold" as the hit samples; then divide the total number of selected hit samples by the total number of all predicted text fragment samples to obtain the corresponding accuracy index, which can be used as the current second evaluation value.

[0182] For example, the calculation of the second evaluation value can be expressed by formula (2): (2) in, "Takes the second evaluation value when the time-series intersection-union threshold is t"; "Hit Groups" represents the total number of hit samples (i.e., predicted text fragment samples that meet the conditions); "Total Pred Groups" represents the total number of all predicted text fragment samples.

[0183] In some embodiments, the third evaluation value is obtained by averaging the second evaluation values ​​corresponding to the multiple time-series intersection-union ratio (CIRR) thresholds. This can be achieved by: obtaining multiple second evaluation values ​​corresponding to the multiple preset CIRR thresholds; summing the multiple second evaluation values ​​and dividing the sum by the total number of CIRR thresholds to obtain the average value of the multiple second evaluation values, which is then used as the third evaluation value.

[0184] For example, the calculation of the third evaluation value can be expressed by formula (3): (3) in, This represents the third evaluation value; T represents a predefined set of discrete thresholds (e.g., setting...). |T| represents the total number of thresholds in the discrete threshold set (e.g., 5); Precision(t) represents the second evaluation value calculated at the temporal intersection-union threshold t. By introducing the third evaluation value (mAP), the video understanding model is forced to not only be correct in understanding macroscopic entities and detailed textual descriptions throughout the entire forward inference cycle, but also to have high accuracy in micro-timestamp delineation.

[0185] By calculating the semantic similarity between predicted text fragment samples and real text fragments in multiple dimensions such as roles, dialogues, actions, and environment based on role mapping relationships, a first evaluation value is determined. Then, by combining multiple temporal intersection and union ratio thresholds, the ratio of hit samples is calculated to obtain the second and third evaluation values. Finally, the first reward value obtained by fusion constructs a comprehensive evaluation system that takes into account the semantic accuracy of multi-dimensional expression and the precision of micro-temporal positioning. This effectively guides the first video understanding model to not only express meaning correctly in macro-entity understanding and detailed text narration, but also to have high precision in micro-time stamp delineation.

[0186] In some embodiments, the first reward value is obtained by merging the first evaluation value, the second evaluation value, and the third evaluation value. This can be achieved by configuring corresponding reward weight ratios for the first evaluation value, the second evaluation value, and the third evaluation value respectively; multiplying the first evaluation value, the second evaluation value, and the third evaluation value by their corresponding reward weight ratios respectively, and then merging them into a final score by weighted summation, which serves as the first reward value.

[0187] Through steps 301 to 304, a comprehensive evaluation mechanism is constructed, including multi-level temporal intersection-union ratio (CIU) precise matching (compatible with one-to-one, one-to-many, and many-to-one mappings), dual entity role mapping verification based on logical anti-conflict, and a multi-dimensional semantic similarity fusion mechanism covering roles, dialogues, actions, and environment. Furthermore, global accuracy (mAP) under multiple temporal thresholds is introduced to calculate the final first reward value. This not only achieves strict alignment between the predicted transcribed content and the real annotations on complex timelines and deep semantic references, effectively avoiding entity reference fallacies and spatiotemporal misalignment illusions, but also imposes a strong constraint on the first video understanding model that balances the accuracy of micro-timestamp delineation with the accuracy of macro-multimodal content narration. Thus, it provides a high-quality reward feedback signal with extremely fine-grained, high fidelity, and strict adherence to logical consistency for the adaptive optimization or reinforcement learning of the first video understanding model.

[0188] In some embodiments, updating the parameters of the first video understanding model based on the first reward value can be achieved in the following ways: using the first reward value as a feedback signal in the reinforcement learning fine-tuning process, constructing a target loss function (e.g., a policy gradient loss function) based on the first reward value; calculating the gradient of the target loss function corresponding to the current parameters of the first video understanding model using the backpropagation algorithm; and iteratively updating the parameters of the first video understanding model according to the calculated gradient using a preset optimizer (e.g., the Adam optimization algorithm), so that the updated first video understanding model is optimized in the direction of obtaining higher reward values ​​in subsequent prediction generation.

[0189] For example, updating the parameters of the first video understanding model based on the first reward value can be achieved through the following specific steps: introducing reinforcement learning algorithms such as Proximal Policy Optimization (PPO) or Direct Preference Optimization (DPO); in this process, the first reward value obtained by fusing the first evaluation value, the second evaluation value, and the third evaluation value is used as the return for the action (i.e., generating the first predicted text sample); the loss value is calculated by maximizing the expected value of the return, and the weights and bias parameters in the network layer of the first video understanding model are updated along the direction of loss descent.

[0190] Through steps 101 to 104, first visual feature samples and first audio feature samples with the same sequence length are injected into network layers of different depths in the first video understanding model. This ensures that the video understanding model can continuously acquire original visual and audio information at multiple levels of forward propagation. By first generating a first thought process sample that includes a summary of the video content and information about the characters, and then generating predicted text containing structured transcription content of multiple events based on the first thought process sample, the video understanding model can be guided to prioritize the construction of a clear macro-level content logic and a framework of character relationships. This provides coherent semantic support for the generation of structured transcription content of multiple subsequent events, improving the logical rationality of complex video content transcription. By dividing the first predicted text sample into multiple predicted text fragment samples and combining them with the corresponding real text fragments to determine the first reward value, the video understanding model can be provided with highly discriminative and fine-grained supervision signals when processing long structured text. This makes the optimization direction of the video understanding model parameters more targeted, thereby improving the accuracy of the video understanding model in generating structured transcription content corresponding to the video.

[0191] In other embodiments, before updating the parameters of the first video understanding model based on the first reward value, the following processing may also be performed: generating a second thought process sample corresponding to the video sample through a second video understanding model, and generating a second predicted text sample based on the second thought process sample, wherein the second video understanding model has the same network architecture as the first video understanding model; determining the difference between the second predicted text sample and the real text, and determining a question text sample and an answer text sample corresponding to the question text sample based on the difference, wherein the real text includes at least one real text fragment; and combining the video sample, the question text sample, and the answer text sample into a question-answering task dataset.

[0192] Here, the second video understanding model can be the initial video understanding model trained through the supervised fine-tuning stage, or a teacher model that can serve as an evaluator (such as a model based on the multimodal large language model MLLM architecture).

[0193] For example, generating a second thought process sample corresponding to a video sample and a second predicted text sample based on the second thought process sample through a second video understanding model can be achieved in the following way: inputting the video sample into the second video understanding model, driving the second video understanding model to first output the intermediate reasoning steps for structured parsing of the video content as the second thought process sample, and then outputting the final structured transcription content based on the second thought process sample as the second predicted text sample.

[0194] The specific implementation method is similar to the processing in steps 102 and 103 above, and will not be repeated here.

[0195] For example, the difference between the second predicted text sample and the real text can be determined by comparing the pre-annotated real text with the second predicted text sample using a teacher model (or evaluator model), identifying and locating specific errors in the second predicted text sample (such as incorrect descriptions of character actions, missing background sounds, or confusion about dialogue attribution), and determining these errors as the differences between the two.

[0196] For example, determining the question text sample and the corresponding answer text sample based on the differences can be achieved in the following way: For each identified difference (i.e., a specific error point), the teacher model automatically generates a question about the video details corresponding to the error point as a question text sample; at the same time, based on the correct content structure in the real text, the correct answer corresponding to the question text sample is generated as the standard answer text sample.

[0197] For example, suppose the structured description of a video clip [00:10-00:15] in the real text is: "Character A picked up a coffee cup from the table, with saxophone music playing in the background"; while the second predicted text sample output by the second video understanding model describes the [00:10-00:15] time period as: "Character A picked up a mobile phone from the table, with saxophone music playing in the background". The teacher model, through comparison, identifies the difference as "incorrect description of the character's action target (mispredicting the coffee cup as a mobile phone)". Based on this specific difference (i.e., the error point), the teacher model automatically generates a question text sample: "What specific item did Character A pick up from the table during the [00:10-00:15] time period?"; simultaneously, based on the correct content structure in the real text, it generates the corresponding standard answer text sample: "coffee cup". In this way, the fuzzy evaluation originally generated for structured long text is accurately reduced in dimensionality and transformed into specific question-and-answer questions that can be used for objective automated judgment, thereby constructing a high-quality adversarial question-and-answer task dataset.

[0198] In some embodiments, the prompt word samples are also used to instruct the first video understanding model to perform a video content question-answering task for the video samples; updating the parameters of the first video understanding model based on the first reward value can be achieved by: inputting the video samples, question text samples, and prompt word samples from the question-answering task dataset into the first video understanding model for processing to obtain a third thinking process sample, and generating a predicted answer text sample based on the third thinking process sample; determining a second reward value based on the predicted answer text sample and the answer text sample; fusing the first reward value and the second reward value to obtain a third reward value; and updating the parameters of the first video understanding model based on the third reward value.

[0199] For example, inputting video samples, question text samples, and prompt word samples from the question-answering task dataset into the first video understanding model for processing to obtain a third thinking process sample, and generating a predicted answer text sample based on the third thinking process sample, can be achieved as follows: the prompt word sample instructs the first video understanding model to adopt a "thinking-Answer" reasoning mode; after receiving the video samples and question text samples as multimodal input, the first video understanding model is required to first output structured transcription content including video details as the third thinking process sample (i.e., the thinking process) to assist reasoning; subsequently, the first video understanding model performs logical reasoning based on the structured information in the generated third thinking process sample and outputs the answer to the above question text sample as the predicted answer text sample.

[0200] For example, the second reward value can be determined based on the predicted answer text sample and the answer text sample as follows: The predicted answer text sample output by the first video understanding model is compared with the standard answer text sample in the question-answering task dataset, and the semantic similarity between the two is calculated; if the two match under the judgment rule (e.g., the semantic similarity is greater than the preset semantic similarity threshold), that is, it proves that the structured transcription content (i.e., the third thinking process sample) output by the first video understanding model in the "thinking process" is accurate about the error detection point or details, then a positive reward value is fed back to the first video understanding model as the second reward value; if the judgment does not match (e.g., the semantic similarity is less than or equal to the preset semantic similarity threshold), then a zero or negative penalty value is fed back as the second reward value.

[0201] For example, the third reward value can be obtained by merging the first reward value and the second reward value in advance by setting corresponding merging weight ratios for the first reward value and the second reward value respectively; multiply the first reward value by its corresponding weight and multiply the second reward value by its corresponding weight; then linearly sum the weighted results of the two to obtain the third reward value.

[0202] For example, updating the parameters of the first video understanding model based on the third reward value can be achieved as follows: the fused third reward value is used as the comprehensive return signal for this multi-inference path; in reinforcement learning stages such as group relative policy optimization (GRPO), the advantage function is calculated using the comprehensive return signal, and a policy gradient loss function is constructed accordingly; the gradient of the loss function with respect to the current parameters of the first video understanding model is calculated using the backpropagation algorithm, and the weights and bias parameters in the network structure are iteratively adjusted using a specified optimization algorithm, thereby driving the first video understanding model to tend to output high-quality predicted text in subsequent generation that meets both macroscopic structural requirements and can withstand scrutiny in microscopic details.

[0203] By constructing a specific question-and-answer task dataset using the difference between the second predicted text sample generated by the second video understanding model and the real text, and instructing the first video understanding model to perform the question-and-answer task to obtain a second reward value, the first reward value and the second reward value are then merged into a third reward value to update the parameters of the first video understanding model. This not only accurately transforms the fuzzy evaluation originally designed for generating structured long text into specific question-and-answer questions that can be used for objective and automated judgment, but also constructs a comprehensive reward signal with multiple inference paths. This effectively drives the first video understanding model to tend to output high-quality predicted text in subsequent generation that meets both macroscopic structural requirements and can withstand scrutiny of question-and-answer details.

[0204] The following will describe the training method of the video understanding model provided in this application embodiment, with the server as the execution subject, using exemplary applications and implementations of the server provided in the embodiments of this application. See also Figure 4 , Figure 4 This is a flowchart illustrating the video data processing method provided in the embodiments of this application, which will be combined with... Figure 4 The steps shown are explained.

[0205] In step 401, the first visual feature and the first audio feature of the video data to be processed are obtained, as well as the text features of preset prompt words are obtained. The prompt words are used to instruct the video understanding model to perform a video content transcription task for the video data. The sequence length of the first visual feature is the same as the sequence length of the first audio feature. The video understanding model is trained by the training method of the video understanding model provided in the embodiments of this application.

[0206] In some embodiments, obtaining the text features of a preset prompt word can be achieved by performing the following processing through a pre-configured text tokenizer: segmenting the prompt word to obtain multiple tokens; embedding each token to obtain the embedding feature of each token; positionally encoding each token to obtain the position feature of each token; fusing the embedding feature and position feature of each token to obtain the fused feature corresponding to the token; and combining the fused features corresponding to multiple tokens to form the text features of the prompt word.

[0207] For specific implementation details, please refer to the description of step 101 above, which will not be repeated here.

[0208] In some embodiments, obtaining the first visual feature and the first audio feature of the video data to be processed can be achieved by: encoding multiple image frames in the video data respectively to obtain the second visual feature of each image frame (see the description of step 201 above); encoding multiple audio segments in the video data respectively to obtain the second audio feature of each audio segment, wherein the multiple image frames and multiple audio segments correspond one-to-one according to the temporal relationship (see the description of step 202 above); and performing feature alignment processing on the second visual feature and the second audio feature within the same preset time interval according to the temporal relationship to obtain the first visual feature and the first audio feature within the preset time interval (see the description of step 203 above).

[0209] In step 402, a thought process is generated based on the first visual feature, the first audio feature, and the text feature through the video understanding model, and a predicted text is generated based on the thought process. The first visual feature and the first audio feature are injected into network layers of different depths in the video understanding model. The thought process includes a summary of the video data and role association information. The predicted text includes structured transcription content of multiple events.

[0210] In some embodiments, the generation of a thought process based on a first visual feature, a first audio feature, and a text feature using a video understanding model, and the generation of predicted text based on the thought process, can be achieved in the following ways: the generation of a thought process based on the first visual feature, the first audio feature, and the text feature using a pre-trained first video understanding model (see the description of step 102 above); and the generation of predicted text based on the thought process using the first video understanding model (see the description of step 103 above).

[0211] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0212] With the development of artificial intelligence technology, video understanding models have been widely used in scenarios such as video content analysis and narrative transcription.

[0213] Among related technologies, video understanding solutions include: 1) Pipeline approach: This approach combines multiple independent models, such as using an Automatic Speech Recognition (ASR) model to extract dialogue, using an Optical Character Recognition (OCR) model to extract subtitles, and combining this with a screen description generated by an image description model. Finally, the above text (dialogue and subtitles) is concatenated and input into a large language model for inference.

[0214] 2) End-to-end video large model solution: Extract video frame features through a pre-trained visual encoder, and project the visual features into the text space through a linear layer or mapping network (such as Q-Former), and directly feed them into the large language model for unified modeling.

[0215] However, video understanding solutions in related technologies still have the following problems: 1) Insufficient audio information perception: It relies heavily on visual features and lacks effective participation of audio features in the deep network of the model. This makes it difficult for video understanding models (such as multimodal large models) to extract key information "behind the scenes". For example, they cannot perceive tension based on the atmosphere created by the background music, or they cannot judge the true intention of the character through tone of voice, resulting in comprehension bias.

[0216] 2) Layer-by-layer forgetting of long sequence information: Multimodal features are usually only injected into the input layer. As the model network depth increases (usually up to dozens of layers), the original audio and video detail features (such as micro-expressions and faint background sounds) are easily diluted or forgotten by the text semantic features, resulting in a significant decrease in the video understanding model's ability to describe temporal details as the depth increases.

[0217] 3) Lack of logical reasoning and low degree of structuring: Directly outputting natural language can easily lead to "illusion" phenomena when faced with complex networks of relationships (i.e., generating content that does not match the original video), and the output of unstructured long text is difficult for downstream automated programs to effectively parse and reuse.

[0218] 4) The "Credit Allocation" Dilemma in Reinforcement Learning: When using reinforcement learning to optimize models, related techniques generate a single reward value based on the entire transcribed text. For videos lasting tens of minutes, if the video understanding model only makes a dialogue error at a certain local time point (such as the 50th minute), a single scalar reward cannot accurately anchor this local bias, which can easily lead to the phenomenon of "over-punishing correct segments" or "falsely encouraging local illusions," resulting in unclear direction for model parameter optimization.

[0219] 5) Evaluation metrics lack semantic robustness: Existing n-gram evaluations (such as Bilingual Evaluation Understudy (BLEU) and Recall-Oriented Understudy for Gisting Evaluation (ROUGE)) rely on literal matching and cannot identify cases where the expressions are semantically equivalent but different. For example, the video understanding model's output "He angrily smashed the cup" is semantically consistent with the real label "He angrily threw away the cup," but it will be judged as low-scoring due to lexical differences.

[0220] 6) Failure of alignment and fine-grained evaluation due to temporal drift: Slight timestamp offsets or inconsistent event segmentation granularity often occur in video transcription, and the one-to-one matching mechanism in related technologies often produces misjudgments. At the same time, the "overall scoring" mechanism in related technologies cannot distinguish whether the video understanding model has a defect in "dialogue extraction" or a deficiency in "scene action understanding", lacking fine-grained evaluation feedback on structured fields.

[0221] In view of this, this application provides a training method for a video understanding model, which will be described below.

[0222] First, the application scenarios of the video understanding model trained through the embodiments of this application will be explained.

[0223] In some embodiments, the video understanding model trained using the examples of this application can be used in an automated film and television plot transcription system. The input, processing, and output of the film and television plot transcription system are as follows: Input: Video clips uploaded by users, including visuals and audio.

[0224] Processing: The video understanding model trained in the embodiments of this application is called to perform multimodal encoding and inference.

[0225] Output: Text results consisting of two parts: 1) Thinking Block (corresponding to the thinking process above): [Plot] (corresponding to the content summary above): A macro-level overview of the current scene's plot (e.g., "Character A is tidying up in the mansion in the early morning when a holographic projection reminds her...").

[0226] [Character Relationships] (corresponding to the role association information above): Analysis of character interactions and potential relationships (e.g., "The interaction between the courier and the customer implies... The phrase 'Please sign for delivery' indicates...").

[0227] 2) Output Block: (corresponding to the predicted text above) The JSON format data (corresponding to the structured transcription of the multiple events mentioned above) includes: location, environment description, and event list.

[0228] The event list includes: timestamp, character, action, expression, and dialogue.

[0229] For example, see Figure 5 , Figure 5 This is a schematic diagram of the video transcription results provided in an embodiment of this application. For example... Figure 5 As shown, the predicted text generated by the video understanding model presents a structured form that is multi-dimensional and collaborative, ranging from macro to micro.

[0230] For example, the output generated by the video understanding model not only includes visual descriptions of each scene in the video (such as scene-1 to scene-k), such as specific locations (e.g., Daisy's house), types (e.g., indoor), environmental details (e.g., interior of a luxurious European-style villa, exquisitely decorated, equipped with a spiral staircase and holographic projection equipment), time (e.g., early morning), and the overall narrative atmosphere of the scene (e.g., relaxed, lifelike); but also provides a detailed, fine-grained decomposition of the narrative logic within the scene, breaking it down into multiple events (e.g., event-1 to event-n) driven by timestamps (e.g., 00:04, 00:08).

[0231] Furthermore, each event encompasses extremely detailed content dimensions: including the identities of the participating characters, the specific sequence of their actions (e.g., wiping the stair railing with a rag, stopping the action, putting their hands on their hips, and looking ahead), precise facial expressions (e.g., slightly helpless, seeking confirmation), complete dialogue text (supporting the distinction between on-screen dialogue and voice-over VO), corresponding sound effects and background (e.g., light and lively orchestral music, mainly pizzicato, with a playful melody, creating a pleasant atmosphere of life), and even in-depth subtext analysis.

[0232] The training method for the video understanding model provided in the embodiments of this application will now be described in detail.

[0233] See Figure 6, Figure 6 This is a schematic diagram of the architecture of the video understanding model provided in the embodiments of this application, which will be combined with Figure 6 Please provide an explanation.

[0234] 1. Multimodal coding and timing alignment mechanism like Figure 6 As shown, in the multimodal preprocessing stage, the video understanding model processes the input video stream and audio stream in parallel.

[0235] First, separate encoding is performed: the video stream is processed by a vision encoder to extract a series of visual feature vectors (such as...). Figure 6 The audio stream (Audio) is processed by the Audio Encoder to extract a series of audio feature vectors (such as...). Figure 6 The audio token in the code); at the same time, the instruction (i.e., the prompt word) is processed by the text tokenizer and converted into text data units (such as audio tokens). Figure 6 Text data unit (Token) in the text.

[0236] Secondly, timing alignment is performed. Because the audio and video modalities have different sampling rates, timestamps based on the physical time series are used, such as... Figure 6 In This explicitly binds visual features and audio features within the same time segment.

[0237] For example, a given video is sampled at a frequency of 1 frame per second (1fps), and the image is adaptively scaled (e.g., 448×256 for portrait mode and 256×448 for landscape mode). After passing through the visual encoder, 56 visual tokens are output per frame; simultaneously, the audio stream is passed through the audio encoder and outputs 50 audio tokens per second. A dynamic processing interval is defined as every 2 seconds (corresponding to the preset time interval mentioned above). Within a consecutive 2-second interval, the total number of visual tokens is 112 (56×2), and the total number of audio tokens is 100 (50×2). At the end of the audio token sequence within this 2-second interval, 12 all-zero tokens are added (i.e., 112-100=12) to ensure that the total number of audio tokens strictly matches the dimension and number of visual tokens. After length alignment is completed, the visual tokens and audio tokens within the corresponding time window are added element by element (or their dimensions are concatenated, which is not limited in this embodiment) and used as joint multimodal features to be input into the subsequent video understanding model (such as MLLM), thereby ensuring the tight integration of audio and video features in the temporal space.

[0238] 2. Deep Injection Architecture like Figure 6 As shown, the video understanding model is logically divided into several concatenated network blocks (Block 1 to Block N). In related technologies, all features are fed only into the input layer (e.g., Block 1), while this application proposes a deep injection architecture. In this deep injection architecture, the features output from the multimodal coding stage are not only connected to the basic input layer of the video understanding model, but also directly distributed and injected into the intermediate and deeper layers of the video understanding model (e.g., through skip connections or a lightweight adapter network) via skip connections or a lightweight adapter network. Figure 6 The model uses Blocks 2, 3, and up to Block N to ensure that original features are not degraded or forgotten when processing features from deeper network layers (such as the Nth layer). The video understanding model can still directly access original acoustic details (such as the tone of a person's sigh) and visual micro-details (such as fleeting micro-expressions or the movement of distant objects) across layers. This makes the "thought process" and "transcribed text" output by the video understanding model more accurate in describing voice-over, specific background atmosphere, and hidden physical emotions.

[0239] 3. Multi-stage training framework Supervised fine-tuning phase (the video understanding model trained in this phase corresponds to the second video understanding model mentioned above): Long videos are annotated using a teacher model with long logical reasoning capabilities (such as MLLM), and an instruction dataset including "implicit thinking-structured transcription" is constructed. The initialized video understanding model is then subjected to basic supervised fine-tuning using the instruction dataset, endowing the video understanding model with the basic ability to output "thinking process" and "transcribed text".

[0240] Post-reinforcement learning training phase (the video understanding model trained in this phase corresponds to the first video understanding model after parameter updates mentioned above): After the supervised fine-tuning phase, the Group Relative Policy Optimization (GRPO) algorithm is introduced to perform deep logical and spatiotemporal accuracy correction on the video understanding model. At this point, the accuracy of time attachment in the script, the stability of character identity overlap, and the completeness of dialogue extraction are set as the basic reward function. To address the difficulties in allocating contribution points (the credit allocation dilemma) and locating local detail errors when performing global scoring on extremely long scripts, this application embodiment incorporates the following training strategy in reinforcement learning: 1) Segmented Dense Reward Strategy (corresponding to steps 301 to 304 above): The segmented dense reward strategy is used to address the credit allocation barrier inherent in long-sequence evaluations in reinforcement learning, and will be explained in detail below.

[0241] First, the structured transcribed content generated by the video understanding model (corresponding to the predicted text samples above) is segmented into multiple consecutive evaluation fields (corresponding to the predicted text fragment samples above) using a pre-defined segmentation logic. The segmentation logic can be based on timestamps (e.g., treating the sequence generated every minute as a field), the transcribed event structure (treating each round of complete story interaction as a field), or rely on specific delimiters in the predicted output to complete the segmentation.

[0242] Subsequently, for each independent segmentation field, a fine-grained matching reward is calculated relative to the ground truth (GT) (corresponding to the real text above). The metrics include N-gram similarity, semantic overlap of sub-events, and compliance of JSON structure syntax within the field.

[0243] Finally, a Local Advantage update is performed. When calculating the improvement direction and weight of a certain feature token in this generation, only the local performance of tokens belonging to the same field is adopted for horizontal normalization comparison. This is equivalent to establishing a local correction mechanism: if the video generation model understands the plot very well in the first half, but forgets the characters in the second half due to time constraints, the truncated reward operator can ensure that the reward signal is accurate and protects the correct network parameters in the first half from interference, while strongly correcting the drifting parts of the parameters in the second half, thus achieving precise multimodal quality control with granularity down to the segment level.

[0244] The following section explains the details of constructing the ground truth (GT). See [link to documentation]. Figure 7 , Figure 7 This is a schematic diagram illustrating the actual text construction principle provided in the embodiments of this application, such as... Figure 7 As shown, by simulating the human long-term memory mechanism, the original video can be processed into a structured script (i.e., real text ground) with global consistency and fine-grained multimodal features. This can be achieved through the following processing: Video Segmentation and Input: The original long video (such as a movie or TV series) is divided into multiple semantically coherent video segments based on scene boundaries and other logical principles (e.g., Figure 7 The text describes a process involving extracting image frames and audio information from the current segment, using the previously shown segments as inputs.

[0245] The narrative and character reasoning based on the Character Profile Manager (CPM) involves inputting the audio and video of the current segment into an internal expert model (such as an LLM) for "character-centric narrative reasoning." During this process, historical profiles are injected from the "Character Information Storage" module. By combining the current audio and video content with historical character profiles across segments (such as voice, gait, and clothing), the problem of character identification across scenes can be accurately solved. After reasoning is complete, a globally consistent "character matching" result and a high-dimensional "narrative description" (such as scene evolution, plot climax, and ending) are output. Simultaneously, evolving multimodal attributes (such as character costume changes) within the current segment are fed back to dynamically update character memories (i.e., profile update), ensuring the consistency of the global identity state across the long video.

[0246] Generating fine-grained scripts through a script annotator: The high-level plot descriptions with accurate character identities generated in the previous step, along with the original audio and video content (image frames, audio), are input into the script annotator (large language model). The script annotator then transforms the general plot into fine-grained, scene-by-scene video scripts.

[0247] Output structured real text (GT): such as Figure 7 As shown, the final generated video script (i.e., the GT text used for model training and reward evaluation) follows a strict hierarchical structure (similar to JSON format), including, for example: Metadata: Global attributes at the video level, such as title and global character list (e.g., "Zhang XX").

[0248] Scene information (Scene / Script Text): Scene identifier (scene_id), physical environment (location: such as "outdoor street"), and time attribute (time: such as "daytime").

[0249] Event List: Atomic events arranged chronologically for each scene, including precise timestamps (e.g., "00:00"), participating characters, dialogue (e.g., "Come here..."), actions, expressions (e.g., "anxious"), and audio cues (e.g., background music, sound effects).

[0250] Furthermore, during the generation of the structured script (GT), the intermediate "thinking" trajectory (including plot development summaries and character relationship mappings) can be extracted and solidified, forming a complete "video-thinking-script" thought chain (CoT) data format. The high-quality, hierarchical, and timestamp-aligned structured script constructed above serves as the authentic reference label required for the segmented, dense reward phases.

[0251] The following section provides a detailed explanation of the specific calculation details and evaluation system for segmented dense rewards.

[0252] (1) Soft and precise role mapping mechanism (corresponding to the role mapping relationship establishment in step 303) Because video understanding models may use different names for the same character during prediction (e.g., "Zhang Mou" is predicted as "male teacher"), the soft, precise character mapping mechanism aims to resolve misjudgments caused by inconsistent character referencing. Specific implementation details are as follows: Entity attribute classification: Using a pre-defined language model, attribute parsing is performed on role names in the predicted text and ground truth (GT) to extract their structured tags. For example, the parsing includes: name content (name, such as "Zhang Mou"), name type (type, divided into specific personal name (corresponding to the role name type above) or role general identity name), and singular / plural identifier (count, divided into singular or plural).

[0253] Core semantic matching and conflict detection: The language model first identifies "name-name" pairs of type "specific" to confirm whether they are aliases. Then, a conflict verification task is performed on "role" identity names to detect whether there is gender bias or absolute semantic conflict of identity between candidate pairs (for example, when the predicted "old lady" is mapped to the real "young man", a conflict is triggered and the pair is eliminated).

[0254] Hybrid Fallback Matching: For the remaining candidate character pairs that failed to match, a hybrid score (corresponding to the mapping score above) is calculated for quantitative evaluation. Hybrid score ( The calculation formula for ) can be found in the explanation of formula (1) above.

[0255] Logical rule filtering: For candidate role pairs whose mixed scores in the above calculation results are greater than a preset score threshold (corresponding to the first preset condition above), a hard physical constraint on singularity and plurality is imposed (e.g., strictly prohibiting the mapping of singular type predicted roles to plural type real role sets, corresponding to the second preset condition above). Finally, a set of deterministic and semantically robust dictionary mapping tables for "predicted roles to real roles" is output (corresponding to the role mapping relationship above).

[0256] (2) Temporal greedy matching and event aggregation fusion (corresponding to steps 301 to 302) Since the event segmentation granularity predicted by the video understanding model may differ from that of the real text (for example, a long real event may be segmented into two short segments by the video understanding model), direct one-to-one comparison will introduce significant errors. Temporal greedy matching and event aggregation fusion aim to eliminate the evaluation noise caused by segmentation fragmentation, which will be explained in detail below.

[0257] Intersection over Union (IoU) Matrix Construction and Constrained Greedy Strategy: Calculate the temporal IoU matrix between all ground truth text fragments (GT events) and predicted text fragment samples (Pred events). Then, sort all possible temporal evaluation pairs in descending order of IoU value; iterate through the sorted queue, and establish pairing connections between GT events and Pred events if neither is currently bound.

[0258] One-to-many (1:N) temporal extension: After the basic connection is completed, if a GT event has been matched by a Pred event, but there are still other adjacent Pred events that have not yet been matched, and the IoU between these Pred events and the GT event (one GT event corresponds to one real text fragment) exceeds the preset IoU threshold, then these additional Pred events (one Pred event corresponds to one predicted text fragment sample) are allowed to be grouped into the same matching group to form a topology of "1 real fragment corresponds to N predicted fragments".

[0259] Granularity Elimination and Content Fusion: For matching groups that form a one-to-many mapping (e.g., one GT event corresponds to two consecutive Pred events), the structured fields (such as Action, Dialogue, etc.) within these two Pred events are concatenated and recombined at the text level using a preset delimiter, according to the chronological order of their timestamps, and aggregated into a coherent "fused prediction event". Subsequent reward calculations will be directly based on the "fused prediction event" and the single GT event, completely bridging the accuracy loss caused by inconsistent event segmentation granularity.

[0260] It should be noted that only one-to-many (1:N) timing extension implementation is shown here. It is understood that the embodiments of this application are not limited to one-to-many (1:N) mapping structures, and many-to-one (N:1) timing extension structures are also applicable.

[0261] Specifically, when the event segmentation granularity predicted by the video understanding model is coarse (e.g., the video understanding model merges multiple consecutive short events segmented from real text into a single long event), after the basic connections are completed, if a Pred event has been matched by a GT event, but there are still other adjacent GT events that have not yet been matched, and the IoU between these GT events and the Pred event exceeds a preset IoU threshold, then these additional GT events are allowed to be grouped into the same matching group, forming a topology of "N real segments corresponding to 1 predicted segment". In the corresponding content fusion stage, the structured fields (such as Action, Dialogue, etc.) within these N GT events are concatenated and reassembled at the text level using a preset delimiter according to the timestamp order, aggregating them into a coherent "fused real event". Subsequent reward calculations will be directly based on a single Pred event and this "fused real event", thus effectively eliminating the evaluation error caused by excessively coarse event segmentation granularity.

[0262] (3) Multi-dimensional comprehensive index calculation system (corresponding to the calculation of the first reward value in step 304) The final reward value is constructed by evaluating multiple modal semantic dimensions: Fine-grained semantic similarity assessment: This involves evaluating each field of the fused event group. For highly unstructured action and environment description fields, the language model outputs a semantic equivalence score ranging from 0.0 to 1.0. For the dialogue field, string-level normalized similarity is calculated using Levenshtein distance (edit distance). For the character field, the aforementioned mapping table is directly applied to determine if it's an exact match (score of 0 or 1). The first evaluation value is obtained by weighted fusion of these subdivided dimensions.

[0263] Temporal accuracy calculation (tIoU Hit Rate): Under constraints of temporal intersection-union ratio (IoU) thresholds with different levels of stringency (e.g., IoU greater than 0.5 or IoU greater than 0.7), the number of predicted segment event groups (i.e., hit sample groups) that meet the above-mentioned "fine-grained semantic similarity admission criteria" is counted according to the corresponding threshold. The formula for the hit rate (corresponding to the second evaluation value above) can be found in the explanation of formula (2) above.

[0264] Convergence of Mean Average Precision (mAP): Combining the above temporal evaluation constraints and semantic scoring, the calculation formula for the global comprehensive evaluation index (corresponding to the third evaluation value above) for long sequence output can be found in formula (3) above.

[0265] For example, see Figure 8 , Figure 8 This is a schematic diagram of the evaluation process provided in an embodiment of this application. For example... Figure 8 As shown, firstly, the system receives a JSON file containing labeled ground truth (GT) and model predictions (Pred) as input. This file then enters the preprocessing module to perform timestamp parsing, multi-role splitting, and text normalization / standardization to eliminate interference from the basic format for subsequent evaluation.

[0266] Subsequently, the preprocessed data enters the parallel processing core, executing dual-path tasks asynchronously: one is path A (role mapping task), responsible for extracting role information and using a large language model for classification and soft-precise matching (corresponding to...). Figure 8 The first step involves a large language model for classification and matching, outputting the final "role mapping table." The second step is path B (semantic evaluation task), which extracts scene or event content, and the large language model calculates the semantic similarity of unstructured fields such as actions, emotions, and locations (corresponding to...). Figure 8 The semantic similarity of the large language model is calculated to generate an "intermediate score" (at which point the score of the role field is pending).

[0267] Next, the role mapping table generated by path A is added to the evaluation system as "injected mapping results" to improve the score. The processing flow then enters the time alignment and fusion module. In this stage, the calculation of the IoU (Intersection over Union) matrix and greedy matching (covering complex mapping relationships such as one-to-one, one-to-many, and many-to-one) are completed in sequence, and dynamic event merging is achieved through text concatenation to completely eliminate the granularity difference of event segmentation.

[0268] Finally, the data stream, aligned with both time sequence and content, is fed into the metric aggregation module. The evaluation results from each of the aforementioned dimensions are summarized, and semantic similarity, temporal accuracy, and mean average precision (mAP) are calculated. The resulting multi-dimensional evaluation metrics serve as a high-quality reward signal, feeding back into the video understanding model to guide further network weight updates and iterative optimization.

[0269] 2) Fine-grained error localization strategy based on question answering (QA) assisted tasks The fine-grained error localization strategy aims to reduce the fuzzy evaluation of massive amounts of unstructured / semi-structured text and transform it into objective selection and question-answering intentions that can be rigorously and automatically determined.

[0270] First, the adversarial QA dataset (corresponding to the question-answering task dataset mentioned above) is constructed. Inputs include video data, manually annotated real text (i.e., ground truth), and predicted text (i.e., structured transcribed content) generated by the video understanding model trained during the basic supervised fine-tuning phase. A teacher model (such as a model based on an MLLM architecture) is used as the evaluator to compare the real text and predicted text in detail, identifying specific errors in the predicted text, such as incorrect descriptions of character actions, missing background sounds, or confusion regarding dialogue attribution. For each identified error, the teacher model automatically generates a corresponding question (corresponding to the question text sample mentioned above) and a standard answer (corresponding to the answer text sample mentioned above), thus constructing the adversarial QA dataset consisting of "video, question, and standard answer."

[0271] In the subsequent GRPO training phase, a "thinking chain-answer" reasoning model is employed. The video understanding model receives the video and the generated question as input, and is required to first output "structured transcribed content" as a thinking process, and then output the "answer" based on the structured transcribed content. Rewards are calculated by matching the model's predicted answer with the standard answer. If the answer is correct, it means that the structured transcribed content generated by the model in the "thinking process" is accurate in its details regarding the question, thus awarding a positive reward.

[0272] The following experimental data further illustrates the effectiveness of the video understanding model trained through the embodiments of this application.

[0273] During the evaluation process, this application adopted a set of refined content quality evaluation indicators.

[0274] Semantic accuracy is evaluated by calculating the similarity between the predicted results and the actual labeled data. For semantic fields (such as actions, expressions, environment, and atmosphere), a pre-defined language model (LLM) is used to measure the semantic similarity between the predicted content and the actual field content. For lexical fields (such as dialogue), normalized Levenshtein distance is used; for category fields (such as roles), precise string matching is performed based on the aforementioned role mapping relationship. Based on the matching scores for each field, the precision is further calculated. ) and recall rate ( The scores are then aggregated to obtain the final F1 score as the evaluation score, for example, as follows: .

[0275] For each field, the corresponding precision ( The calculation of ) can be achieved by performing the following process: determining the total number of field contents output for that specific field in the prediction results (denoted as ). ); In all aligned groups (i.e., groups where the predicted event fragment matches the event fragment corresponding to the actual labeled data), the similarity between the predicted content and the actual field content of this field. Perform summation (i.e.) Subsequently, the sum of the above similarities is calculated and then summed with the total number of field contents. The ratio between these two values ​​gives the precision of the corresponding field. For example, it can be expressed as: .

[0276] For each field, the recall rate ( The calculation of ) can be achieved in the following way: determine the total number of actual field contents corresponding to the fields in the actual labeled data (denoted as ). Similarly, within all the aligned groups mentioned above, obtain the content similarity of this field. The summation result (i.e. Then, the sum of the above similarities is calculated and the total number of the actual field contents is calculated. The ratio between these two values ​​yields the recall rate for that field. For example, it can be expressed as: .

[0277] Furthermore, to evaluate the localization quality of the video understanding model for predicted timestamps, embodiments of this application can also use temporal intersection-union (TIU) metrics for evaluation. The evaluation process can be implemented in the following ways: Set a pre-specified threshold Example, threshold You can select a set Any value in the table; for a predicted text segment (such as structured text corresponding to an event) output by the video understanding model, calculate the overlap (also known as time-series intersection-union) on the time axis between the time interval corresponding to the predicted text segment and the real time interval of the corresponding real text segment in the real text. ).

[0278] If at the currently set strictness threshold Below, determine whether the above overlap satisfies If the predicted text segment is not found to be a true positive (i.e., the sequential localization is effective), then the predicted text segment is considered as a true positive (i.e., the localization is effective). Based on the above determination, the threshold values ​​for all predicted text segments are statistically analyzed and calculated. The time accuracy (denoted as) ) and time recall (denoted as Thus, a comprehensive evaluation index for the time dimension (denoted as ) is derived through the aggregation method of harmonic mean calculation. For example, it can be represented as: .

[0279] Referring to Table 1 (evaluation results on movie data) and Table 2 (evaluation results on short drama data), compared with the baseline model in related technologies, the video understanding model obtained by introducing the thought chain derivation and reinforcement learning training mechanism through the embodiments of this application has achieved significant accuracy improvements in event parsing and timing alignment tasks for both long videos (movies) and short videos (short dramas).

[0280] Table 1

[0281] Table 2

[0282] The effectiveness of the embodiments of this application will be comprehensively and thoroughly explained below, combining experimental setup, multi-dimensional index evaluation, long video robustness test, and ablation experiment results.

[0283] 1. Implementation details In this embodiment of the application, the training process follows a three-stage scheme: Modal alignment stage: Freeze the LLM backbone network, optimize only the modal projectors, and apply random frame masks to improve robustness in partial observation environments.

[0284] Pre-training phase: Full fine-tuning was performed on approximately 2.4 million videos in both Chinese and English, covering tasks such as ASR, dense video captioning, summary summarization, and time-series localization.

[0285] SFT and Reinforcement Learning Phase: Performed on approximately 45,000 carefully selected professional film and television videos. Before generating structured decoding, CoT (Cognitive Chain)-style intermediate trajectories are introduced, and reinforcement learning alignment is performed using segmented dense rewards.

[0286] 2. Performance analysis of 5-minute video segments (Tables 3 and 4) In benchmark tests of 5-minute videos, the video understanding model trained in this application's embodiments exhibits extremely high parameter efficiency: Event-level performance: As shown in Table 3, in the movie data test, the video understanding model trained by the embodiments of this application achieved an overall score of 37.0 and a temporal localization accuracy score of 69.0. This result significantly surpasses larger open-source models (such as reference model-9) and outperforms some proprietary closed-source models (such as reference model-4) in key indicators such as dialogue understanding and temporal localization.

[0287] Table 3

[0288] Scene-level performance: As shown in Table 4, the video understanding model trained in this application embodiment achieved an overall score of 52.4 and a temporal positioning accuracy score of 74.6 in the short drama data test.

[0289] Table 4

[0290] 3. Performance comparison on long videos To evaluate the robustness of the video understanding modal model in handling longer temporal contexts, performance was further benchmarked on different video lengths (extending up to 30 minutes of movie clips).

[0291] For example, see Figure 9 , Figure 9 This is a comparison chart of model performance changes under different video durations provided in the embodiments of this application. Figure 9 This shows the performance trends of each model in terms of Event-Level Overall F1 score and Event-Level Overall Recall across durations of 5 minutes, 10 minutes, 15 minutes, 20 minutes, 25 minutes, and 30 minutes. Combined with... Figure 9 The following conclusions can be drawn: Common challenges in long-term temporal context understanding: such as Figure 9 As shown, the overall F1 score exhibits a significant downward trend with increasing video length. This reflects the inherent difficulty of understanding long films: when models need to process massive audiovisual token streams, they are prone to errors, context dilution, and loss of fine-grained details.

[0292] The model framework provided in this application exhibits exceptional resilience and resistance to performance degradation: the performance degradation of the method in this application is the most gradual. Notably, even on the most challenging 30-minute video subset, the video understanding model trained in this application still maintains a high F1 score and recall. This sustained superiority demonstrates that the model architecture design and training strategies (such as the thought chain derivation mechanism) provided in this application can effectively mitigate catastrophic forgetting and maintain consistency in narrative understanding over extremely long computational spans.

[0293] Analysis of the trade-off between context window and model capacity: Observations Figure 9The performance of the reference model-5 shows that it exhibits extremely high recall for videos under 20 minutes, but the F1 score drops significantly after the video length exceeds 20 minutes. This indicates that although some models have a large theoretical context window, they are limited by the model parameter capacity and struggle to consistently adhere to a strict structured script format when faced with very long videos, easily falling into repetitive loops or outputting unstructured content.

[0294] In summary, Figure 9 The data results strongly demonstrate that the video understanding model trained through the embodiments of this application has significant robustness and practical value when processing ultra-long videos (such as those on the order of 30 minutes) in real-world scenarios.

[0295] 4. Ablation Experiment Analysis The gains from the Chain of Thought (CoT): As shown in Table 5, introducing CoT into the SFT baseline increased the overall score from 35.3% to 37.0%, and improved the F1 score of the dialogue by 2.8%. This demonstrates that CoT can serve as a cognitive scaffold, significantly reducing logical ambiguity in long contexts.

[0296] The advantages of segmented rewards: As shown in Table 5, the combination of "supervised fine-tuning + reinforcement learning post-training + segmented rewards" achieved the highest score (37.7%). Compared to global rewards, segmented rewards, through a more granular penalty mechanism, precisely optimize the accuracy and recall of each event segment.

[0297] Table 5

[0298] As observed in the preceding benchmark tests, some purely visual models that do not support audio input still achieved relatively good scores on the "Dialogue" metric. This suggests that they may have utilized visible subtitles as "auxiliary cues" in the video. To verify this, embodiments of this application conducted a masking (occlusion) test on the video subtitles during the inference phase, and the specific comparison results are shown in Table 6.

[0299] Test results show that when subtitles are blocked, the model supporting full-modal input only experienced a moderate drop in dialogue F1 score. This indicates that the full-modal model does not simply rely on "copying" the text on the screen, but truly combines audio, video, and other multimodal cues for dialogue understanding, thus exhibiting stronger robustness in scenarios without subtitles.

[0300] In contrast, large, purely visual models without audio alignment (such as reference model-9) experienced a significant drop in dialogue recognition performance (from 58.6% to 7.7%) after subtitles were removed. This reveals that such models heavily rely on explicit visual text presentation for dialogue understanding and lack true audiovisual modality-level semantic understanding capabilities.

[0301] Table 6

[0302] To rigorously verify the necessity of acoustic modalities and the effectiveness of the pre-training stage (supervised fine-tuning stage) of the embodiments of this application, the embodiments of this application were compared. As shown in Table 7, although the pure visual model can recognize basic visual actions ("action" score of 30.3%), it encountered serious performance bottlenecks in fields that are highly dependent on audio and implicit semantics, especially the "dialogue" recognition accuracy was limited to only 52.0%.

[0303] In stark contrast, the video understanding model trained by the method in this application (supporting full-modal data), through deep alignment of acoustic features with the backbone of the large language model during the pre-training phase, achieved a dialogue accuracy of 68.2%, resulting in a gain of +16.2%. This significant performance gap not only highlights the fundamental limitations of "vision-centric" models in long-form video script generation tasks, but also fully demonstrates the effectiveness and necessity of the audio injection and pre-training strategies proposed in this application.

[0304] Table 7

[0305] In summary, the embodiments of this application provide a complete technical solution for fine-grained understanding and structured transcription text generation of video data.

[0306] First, in terms of the underlying model architecture, the embodiments of this application break through the limitations of a single visual model by introducing a strict temporal alignment mechanism for audio and video features and a deep injection architecture. This not only avoids the degradation and forgetting of original audiovisual details in deep networks, but also fundamentally establishes the true full-modal (visual + acoustic) semantic perception capability of the video understanding model.

[0307] Secondly, regarding the model training strategy, this application's embodiments design a progressive multi-stage training framework: by constructing a long-context narrative logic of "thinking first, then generating" through supervised fine-tuning with a thought chain (CoT) trajectory; by introducing an innovative segmented dense reward mechanism and a fine-grained positioning strategy for QA auxiliary tasks in the post-reinforcement learning training, the "credit allocation problem" and the error of inconsistent event segmentation granularity in the ultra-long sequence evaluation process are solved, forcing the video understanding model to achieve a win-win situation of extremely high accuracy in both macro-level plot coherence and micro-level timestamp delineation.

[0308] Finally, in terms of empirical performance, the extremely detailed benchmark tests and ablation experiment data strongly demonstrate the effectiveness of the training method for the video understanding model provided in the embodiments of this application.

[0309] Overall, the training method for the video understanding model provided in this application provides an efficient, accurate, and highly commercially valuable solution for business scenarios such as automated film and television analysis, long-form video content retrieval, and intelligent assisted creation.

[0310] The following description continues to illustrate the exemplary structure of the training device 133 for the video understanding model provided in this application embodiment as a software module. In some embodiments, such as... Figure 2A As shown, the software modules in the training device 133 of the video understanding model stored in memory 130-1 may include: The data processing module 1331 is used to acquire a first visual feature sample and a first audio feature sample of a video sample, and to acquire a text feature sample of a preset prompt word sample, wherein the prompt word sample is used to instruct a first video understanding model to perform a video content transcription task for the video sample, and the sequence length of the first visual feature sample is the same as the sequence length of the first audio feature sample.

[0311] Training module 1332 is used to generate a first thought process sample based on the first visual feature sample, the first audio feature sample and the text feature sample through the first video understanding model, wherein the first visual feature sample and the first audio feature sample are injected into network layers of different depths of the first video understanding model.

[0312] In some embodiments, the training module 1332 is further configured to generate a first predicted text sample based on the first thought process sample using the first video understanding model, wherein the first predicted text sample includes structured transcribed content of multiple events, and the first thought process sample includes a content summary and role association information of the video sample.

[0313] In some embodiments, the training module 1332 is further configured to divide the first predicted text sample into multiple predicted text segment samples, determine a first reward value based on the multiple predicted text segment samples and at least one real text segment corresponding to the multiple predicted text segment samples, and update the parameters of the first video understanding model based on the first reward value.

[0314] In some embodiments, the data processing module 1331 is further configured to encode multiple image frames in the video sample respectively to obtain a second visual feature sample for each image frame; encode multiple audio segments in the video sample respectively to obtain a second audio feature sample for each audio segment, wherein the multiple image frames and the multiple audio segments correspond one-to-one according to a temporal relationship; and perform feature alignment processing on the second visual feature samples and the second audio feature samples within the same preset time interval according to the temporal relationship to obtain the first visual feature samples and the first audio feature samples within the preset time interval.

[0315] In some embodiments, the data processing module 1331 is further configured to: determine the first total sequence length of the second visual feature samples within the preset time interval; determine the second total sequence length of the second audio feature samples within the preset time interval; calculate the sequence length difference between the first total sequence length and the second total sequence length; if the first total sequence length is greater than the second total sequence length, supplement the end of the last second audio feature sample within the preset time interval with a preset number of feature vectors corresponding to the sequence length difference, based on the sequence length difference, to obtain the first audio feature sample corresponding to the preset time interval; and combine the second visual feature samples within the preset time interval into the first visual feature sample corresponding to the preset time interval.

[0316] In some embodiments, the training module 1332 is further configured to: fuse the first visual feature samples with the first audio feature samples to obtain multimodal feature samples; select multiple target network layers of different depths from multiple network layers of the first video understanding model; input the text feature samples and the multimodal feature samples into the input layer of the first video understanding model, and input the multimodal feature samples into each of the target network layers respectively, so that the output layer of the first video understanding model outputs the first thinking process sample; input the thinking text features of the first thinking process sample, the text feature samples, and the multimodal feature samples into the input layer of the first video understanding model, and input the multimodal feature samples into each of the target network layers respectively, so that the output layer of the first video understanding model outputs the first predicted text sample.

[0317] In some embodiments, the training module 1332 is further configured to perform the following processing for each target network layer: using the output of the previous network layer of the target network layer as a query feature; performing feature mapping on the multimodal feature samples input to the target network layer to obtain key features and value features; performing attention encoding based on the query features, the key features and the value features to obtain attention features, and fusing the attention features with the output of the previous network layer to obtain the output of the target network layer.

[0318] In some embodiments, the real text includes the at least one real text segment; the training module 1332 is further configured to: determine the temporal intersection-union ratio (CIU) corresponding to each predicted text segment sample based on a first time interval corresponding to each predicted text segment sample and a second time interval corresponding to each real text segment; determine at least one real text segment that has a matching relationship with each predicted text segment sample based on the CIU corresponding to each predicted text segment sample; obtain multiple predicted role information in the first predicted text sample and multiple real role information in the real text; establish a role mapping relationship based on the multiple predicted role information and the multiple real role information; and calculate the first reward value based on the role mapping relationship, the multiple predicted text segment samples, and at least one real text segment that matches each predicted text segment sample.

[0319] In some embodiments, each predicted role information and each real role information includes an object name, a name type corresponding to the object name, and a singular / plural identifier indicating whether the object name is singular or plural; the role mapping relationship includes a first mapping relationship and a second mapping relationship; the training module 1332 is further configured to perform object name matching on the predicted role information and the real role information whose name type is a role name type based on the name type to obtain the first mapping relationship; combine the predicted role information that does not have the first mapping relationship with the real role information in pairs to obtain at least one candidate mapping pair, and eliminate those with semantic differences. Conflicting candidate mapping pairs, wherein the semantic conflict refers to a semantic contradiction between the object name corresponding to the predicted role information and the object name corresponding to the real role information in the candidate mapping pair; for the candidate mapping pairs obtained after eliminating semantic conflicts, a mapping score is determined based on the predicted text fragment corresponding to the predicted role information and the real text fragment corresponding to the real role information in the candidate mapping pair; based on the candidate mapping pairs whose mapping scores satisfy a first preset condition and whose singular / plural identifiers in the predicted role information and the singular / plural identifiers in the real role information satisfy a second preset condition, the second mapping relationship is obtained.

[0320] In some embodiments, the training module 1332 is further configured to calculate the text similarity between the predicted text segment corresponding to the predicted role information and the real text segment corresponding to the real role information; obtain the temporal intersection-union ratio between the predicted text segment corresponding to the predicted role information and the real text segment corresponding to the real role information; and fuse the text similarity and the temporal intersection-union ratio to obtain the mapping score corresponding to the candidate mapping pair.

[0321] In some embodiments, the training module 1332 is further configured to: classify candidate mapping pairs whose mapping scores are greater than a preset matching score threshold as candidate mapping pairs that satisfy the first preset condition; classify candidate mapping pairs that do not trigger a preset referential conflict condition as candidate mapping pairs that satisfy the second preset condition, wherein the referential conflict condition refers to a situation where, between the predicted role information and the real role information of the candidate mapping pair, one of them has a singular / plural identifier that is plural, and the other has a name type that is a role name type or a singular / plural identifier that is singular; and determine the second mapping relationship based on the candidate mapping pairs that satisfy the first preset condition and the second preset condition.

[0322] In some embodiments, the training module 1332 is further configured to perform the following processing for each predicted text segment sample: combining the first time interval corresponding to the predicted text segment sample with the second time interval corresponding to each real text segment to construct at least one time-series evaluation pair; for each time-series evaluation pair, calculating the time intersection length and the time union length of the first time interval and the second time interval; for each time-series evaluation pair, calculating the ratio between the time intersection length and the time union length as the time-series intersection-union ratio corresponding to the time-series evaluation pair.

[0323] In some embodiments, the training module 1332 is further configured to traverse the temporal intersection-union ratio (CIU) in descending order; if no matching relationship is established between the predicted text segment sample corresponding to the CIU and the real text segment, then a matching relationship is established between the predicted text segment sample corresponding to the CIU and the real text segment, and the predicted text segment sample corresponding to the CIU is determined as a predicted text segment sample with an established matching relationship, and the real text segment corresponding to the CIU is determined as a real text segment with an established matching relationship; after the traversal is completed, for the predicted text segment sample with no established matching relationship, the training module 1332 is further configured to... This sample segment selects real text segments from the established matching real text segments that satisfy a preset temporal overlap condition with the predicted text segment samples that do not have a matching relationship, and establishes a matching relationship between the predicted text segment samples that do not have a matching relationship and the selected real text segments; for the real text segments that do not have a matching relationship, it selects predicted text segment samples from the established matching real text segment samples that satisfy the temporal overlap condition with the real text segments that do not have a matching relationship, and establishes a matching relationship between the real text segments that do not have a matching relationship and the selected predicted text segment samples.

[0324] In some embodiments, the training module 1332 is further configured to: calculate, based on the role mapping relationship, a semantic similarity of multiple dimensions between the predicted text fragment samples and the matched real text fragments, and determine a first evaluation value based on the semantic similarity of the multiple dimensions; for each of the preset multiple temporal intersection-union (CIU) thresholds, if the CIU between the predicted text fragment sample and the matched real text fragment is greater than or equal to the CIU threshold, and the first evaluation value is greater than or equal to a preset first evaluation value threshold, then the predicted text fragment sample is determined as a hit sample; calculate the ratio between the number of hit samples corresponding to each CIU threshold and the total number of multiple predicted text fragment samples, as a second evaluation value; average the second evaluation values ​​corresponding to the multiple CIU thresholds respectively to obtain a third evaluation value; and fuse the first evaluation value, the second evaluation value, and the third evaluation value to obtain the first reward value.

[0325] In some embodiments, both the predicted text fragment sample and the real text fragment include at least a role field, a dialogue field, an action field, and an environment field. The training module 1332 is further configured to: compare the role field of the predicted text fragment sample with the role field of the real text fragment based on the role mapping relationship; determine the role semantic similarity based on the comparison result; calculate the dialogue semantic similarity between the dialogue field of the predicted text fragment sample and the dialogue field of the real text fragment; determine the action semantic similarity between the action field of the predicted text fragment sample and the action field of the real text fragment using a pre-set language model; determine the environment semantic similarity between the environment field of the predicted text fragment sample and the environment field of the real text fragment using the language model; and fuse the role semantic similarity, the dialogue semantic similarity, the action semantic similarity, and the environment semantic similarity to obtain the first evaluation value.

[0326] In some embodiments, the training module 1332 is further configured to generate a second thought process sample corresponding to the video sample using a second video understanding model, and generate a second predicted text sample based on the second thought process sample, wherein the second video understanding model has the same network architecture as the first video understanding model; determine the difference between the second predicted text sample and the real text, and determine a question text sample and an answer text sample corresponding to the question text sample based on the difference, wherein the real text includes the at least one real text segment; and combine the video sample, the question text sample, and the answer text sample into a question-answering task dataset.

[0327] In some embodiments, the prompt word samples are further used to instruct the first video understanding model to perform a video content question-and-answer task for the video samples; the training module 1332 is further used to input the video samples, the question text samples, and the prompt word samples from the question-and-answer task dataset into the first video understanding model for processing to obtain a third thinking process sample, and generate a predicted answer text sample based on the third thinking process sample; determine a second reward value based on the predicted answer text sample and the answer text sample; fuse the first reward value and the second reward value to obtain a third reward value; and update the parameters of the first video understanding model based on the third reward value.

[0328] The following description continues to illustrate the exemplary structure of the video data processing apparatus 134 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2B As shown, the software module in the video data processing device 134 stored in the memory 130-2 may include: The data processing module 1341 is used to acquire first visual features and first audio features of the video data to be processed, and to acquire text features of preset prompt words, wherein the prompt words are used to instruct the video understanding model to perform a video content transcription task for the video data, the sequence length of the first visual features is the same as the sequence length of the first audio features, and the video understanding model is trained by the training method of the video understanding model provided in the embodiments of this application.

[0329] The text generation module 1342 is used to generate a thought process based on the first visual feature, the first audio feature and the text feature through the video understanding model, and to generate predicted text based on the thought process, wherein the first visual feature and the first audio feature are injected into network layers of different depths of the video understanding model, the thought process includes a content summary and role association information of the video data, and the predicted text includes structured transcription content of multiple events.

[0330] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the video understanding model training method described in this application embodiment, or the video data processing method described in this application embodiment.

[0331] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the video data processing method provided in this application, or a video data processing method, for example... Figure 3A The video data processing method shown, or Figure 4 The video data processing method shown is illustrated.

[0332] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0333] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0334] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0335] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0336] In summary, by injecting first visual feature samples and first audio feature samples with the same sequence length into network layers of different depths in the first video understanding model through the embodiments of this application, it is ensured that the video understanding model can continuously acquire original visual and audio information at multiple levels of forward propagation. By first generating a first thought process sample including a summary of the video content and character association information, and then generating predicted text containing structured transcribed content including multiple events based on the first thought process sample, the video understanding model can be guided to prioritize the construction of a clear macro-content logic and character relationship framework, providing coherent semantic support for the subsequent generation of structured transcribed content for multiple events, and improving the logical rationality of complex video content transcription. By dividing the first predicted text sample into multiple predicted text fragment samples and combining them with the corresponding real text fragments to determine the first reward value, it is possible to provide the video understanding model with highly discriminative and fine-grained supervision signals when processing long structured text, making the optimization direction of the video understanding model parameters more targeted, thereby improving the accuracy of the video understanding model in generating structured transcribed content corresponding to the video.

[0337] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A training method for a video understanding model, characterized in that, The method includes: The system acquires a first visual feature sample and a first audio feature sample of a video sample, as well as a text feature sample of a preset prompt word sample. The prompt word sample is used to instruct the first video understanding model to perform a video content transcription task for the video sample. The sequence length of the first visual feature sample is the same as the sequence length of the first audio feature sample. The first video understanding model generates a first thought process sample based on the first visual feature sample, the first audio feature sample, and the text feature sample, wherein the first visual feature sample and the first audio feature sample are injected into network layers of different depths in the first video understanding model. The first video understanding model generates a first predicted text sample based on the first thought process sample. The first predicted text sample includes structured transcription content of multiple events, and the first thought process sample includes a content summary and role association information of the video sample. The first predicted text sample is divided into multiple predicted text segment samples. Based on the multiple predicted text segment samples and at least one real text segment corresponding to the multiple predicted text segment samples, a first reward value is determined, and based on the first reward value, the parameters of the first video understanding model are updated.

2. The method according to claim 1, characterized in that, The acquisition of the first visual feature sample and the first audio feature sample of the video sample includes: Encode each of the multiple image frames in the video sample to obtain a second visual feature sample for each image frame; The multiple audio segments in the video sample are encoded to obtain a second audio feature sample for each audio segment, wherein the multiple image frames and the multiple audio segments correspond one-to-one according to a temporal relationship; Based on the temporal relationship, feature alignment processing is performed on the second visual feature samples and the second audio feature samples within the same preset time interval to obtain the first visual feature samples and the first audio feature samples within the preset time interval.

3. The method according to claim 2, characterized in that, The step of performing feature alignment processing on the second visual feature samples and the second audio feature samples within the same preset time interval according to the temporal relationship to obtain the first visual feature samples and the first audio feature samples within the preset time interval includes: Determine the total length of the first sequence of the second visual feature samples within the preset time interval, and determine the total length of the second sequence of the second audio feature samples within the preset time interval; Calculate the sequence length difference between the total length of the first sequence and the total length of the second sequence; When the total length of the first sequence is greater than the total length of the second sequence, according to the sequence length difference, a preset number of preset feature vectors corresponding to the sequence length difference are added to the end of the last second audio feature sample in the preset time interval to obtain the first audio feature sample corresponding to the preset time interval. The second visual feature samples within a preset time interval are combined into the first visual feature samples corresponding to the preset time interval.

4. The method according to any one of claims 1 to 2, characterized in that, The step of generating a first thought process sample based on the first visual feature sample, the first audio feature sample, and the text feature sample using the first video understanding model includes: By fusing the first visual feature sample and the first audio feature sample, a multimodal feature sample is obtained; Select multiple target network layers of different depths from multiple network layers of the first video understanding model; The text feature samples and the multimodal feature samples are input into the input layer of the first video understanding model, and the multimodal feature samples are respectively input into each of the target network layers, so that the output layer of the first video understanding model outputs the first thinking process sample; The generation of the first predicted text sample based on the first thought process sample includes: The thinking text features of the first thinking process sample, the text feature sample, and the multimodal feature sample are input into the input layer of the first video understanding model, and the multimodal feature samples are respectively input into each of the target network layers, so that the output layer of the first video understanding model outputs the first predicted text sample.

5. The method according to claim 4, characterized in that, The step of inputting the multimodal feature samples into each of the target network layers includes: For each of the target network layers, the following processing is performed: Use the output of the network layer preceding the target network layer as the query feature; The multimodal feature samples input into the target network layer are subjected to feature mapping to obtain key features and value features; Attention features are obtained by performing attention encoding based on the query features, key features, and value features. The attention features are then fused with the output of the previous network layer to obtain the output of the target network layer.

6. The method according to claim 1, characterized in that, Real text includes at least one real text fragment; The step of determining a first reward value based on the plurality of predicted text fragment samples and at least one real text fragment corresponding to the plurality of predicted text fragment samples includes: Based on the first time interval corresponding to each predicted text segment sample and the second time interval corresponding to each real text segment, the temporal intersection-over-union ratio corresponding to each predicted text segment sample is determined; Based on the temporal intersection-union ratio corresponding to each predicted text fragment sample, at least one real text fragment that has a matching relationship with each predicted text fragment sample is determined; Obtain multiple predicted role information from the first predicted text sample and multiple real role information from the real text; establish a role mapping relationship based on the multiple predicted role information and the multiple real role information. The first reward value is calculated based on the role mapping relationship, multiple predicted text fragment samples, and at least one real text fragment that matches each predicted text fragment sample.

7. The method according to claim 6, characterized in that, Each of the predicted role information and each of the real role information includes an object name, a name type corresponding to the object name, and a singular or plural identifier indicating whether the object name is singular or plural; The role mapping relationship includes a first mapping relationship and a second mapping relationship; The step of establishing a role mapping relationship based on the multiple predicted role information and the multiple real role information includes: Based on the name type, the predicted character information and the real character information, where the name type is a character name, are matched by object name to obtain the first mapping relationship; The predicted role information that has not established the first mapping relationship is combined with the real role information in pairs to obtain at least one candidate mapping pair, and candidate mapping pairs with semantic conflicts are eliminated. The semantic conflict refers to the semantic contradiction between the object name corresponding to the predicted role information and the object name corresponding to the real role information in the candidate mapping pair. For the candidate mapping pairs obtained after removing semantic conflicts, a mapping score is determined based on the predicted text fragment corresponding to the predicted role information and the real text fragment corresponding to the real role information within the candidate mapping pair. The second mapping relationship is obtained based on the candidate mapping pairs that satisfy the first preset condition and the singular / plural identifiers in the predicted role information and the singular / plural identifiers in the real role information satisfy the second preset condition.

8. The method according to claim 7, characterized in that, The step of determining the mapping score corresponding to the candidate mapping pair based on the predicted text fragment corresponding to the predicted role information and the real text fragment corresponding to the real role information within the candidate mapping pair includes: Calculate the text similarity between the predicted text fragment corresponding to the predicted role information and the real text fragment corresponding to the real role information; Obtain the temporal intersection-over-union ratio (IoU) between the predicted text fragments corresponding to the predicted character information and the real text fragments corresponding to the real character information; By fusing the text similarity and the temporal intersection-union ratio, the mapping score corresponding to the candidate mapping pair is obtained.

9. The method according to claim 7, characterized in that, The process of obtaining the second mapping relationship based on the candidate mapping pairs that satisfy a first preset condition and whose singular / plural identifiers in the predicted role information and the singular / plural identifiers in the real role information satisfy a second preset condition includes: Candidate mapping pairs whose mapping scores are greater than a preset matching score threshold are considered as candidate mapping pairs that satisfy the first preset condition. Candidate mapping pairs that do not trigger the preset referential conflict condition are taken as candidate mapping pairs that satisfy the second preset condition. The referential conflict condition refers to the situation where, between the predicted role information and the real role information of the candidate mapping pair, one of the singular / plural identifiers is plural, and the other party's name type is a role name type or the singular / plural identifier is singular. The second mapping relationship is determined based on candidate mapping pairs that satisfy the first preset condition and the second preset condition.

10. The method according to claim 6, characterized in that, The step of determining the temporal intersection-over-union ratio (TIU) for each predicted text segment sample based on the first time interval corresponding to each predicted text segment sample and the second time interval corresponding to each real text segment includes: For each of the predicted text fragment samples, the following processing is performed: The first time interval corresponding to each predicted text segment sample is combined with the second time interval corresponding to each real text segment to construct at least one time series evaluation pair; For each of the time series evaluation pairs, calculate the length of the time intersection and the length of the time union between the first time interval and the second time interval; For each time series evaluation pair, the ratio between the time intersection length and the time union length is calculated as the time series intersection-union ratio corresponding to the time series evaluation pair.

11. The method according to claim 6, characterized in that, The step of determining at least one real text segment that has a matching relationship with each predicted text segment sample based on the temporal intersection-union ratio corresponding to each predicted text segment sample includes: The process is traversed in descending order of the temporal intersection-union ratio (CIU). If no matching relationship is established between the predicted text fragment sample and the real text fragment corresponding to the CIU, then a matching relationship is established between the predicted text fragment sample and the real text fragment corresponding to the CIU. The predicted text fragment sample corresponding to the CIU is determined as a predicted text fragment sample with an established matching relationship, and the real text fragment corresponding to the CIU is determined as a real text fragment with an established matching relationship. After the traversal is completed, for the predicted text fragment samples that have not established a matching relationship, real text fragments that meet the preset temporal overlap conditions with the predicted text fragment samples that have not established a matching relationship are selected from the real text fragments that have established a matching relationship, and a matching relationship is established between the predicted text fragment samples that have not established a matching relationship and the selected real text fragments. For the real text segments for which no matching relationship has been established, predictive text segment samples that satisfy the temporal overlap condition with the real text segments for which no matching relationship has been established are selected from the predicted text segment samples for which a matching relationship has been established, and a matching relationship is established between the real text segments for which no matching relationship has been established and the selected predicted text segment samples.

12. The method according to claim 6, characterized in that, The calculation of the first reward value based on the role mapping relationship, multiple predicted text fragment samples, and at least one real text fragment that matches each predicted text fragment sample includes: For the predicted text fragment samples and the real text fragments that have a matching relationship, based on the role mapping relationship, the semantic similarity between the predicted text fragment samples and the matched real text fragments is calculated in multiple dimensions, and a first evaluation value is determined based on the semantic similarity in multiple dimensions. For each of the preset temporal intersection-union (CIU) thresholds, if the CIU between the predicted text fragment sample and the matched real text fragment is greater than or equal to the CIU threshold, and the first evaluation value is greater than or equal to a preset first evaluation value threshold, then the predicted text fragment sample is determined as a hit sample; the ratio between the number of hit samples corresponding to each CIU threshold and the total number of the multiple predicted text fragment samples is calculated as a second evaluation value; The third evaluation value is obtained by averaging the second evaluation values ​​corresponding to the multiple time series intersection-union ratio thresholds. The first reward value is obtained by combining the first evaluation value, the second evaluation value, and the third evaluation value.

13. The method according to claim 12, characterized in that, Both the predicted text fragment samples and the real text fragments include at least the role field, dialogue field, action field, and environment field; The step of calculating the semantic similarity across multiple dimensions between the predicted text fragment sample and the matched real text fragment based on the role mapping relationship includes: Based on the role mapping relationship, the role field of the predicted text fragment sample is compared with the role field of the real text fragment, and the role semantic similarity is determined based on the comparison result; Calculate the dialogue semantic similarity between the dialogue fields of the predicted text fragment sample and the dialogue fields of the real text fragment; Using a pre-set language model, the action semantic similarity between the action field of the predicted text fragment sample and the action field of the real text fragment is determined; The language model is used to determine the environmental semantic similarity between the environmental field of the predicted text fragment sample and the environmental field of the real text fragment; The determination of the first evaluation value based on the semantic similarity of the multiple dimensions includes: The first evaluation value is obtained by integrating the semantic similarity of the characters, the semantic similarity of the dialogue, the semantic similarity of the actions, and the semantic similarity of the environment.

14. The method according to claim 1, characterized in that, Before updating the parameters of the first video understanding model based on the first reward value, the method further includes: The second video understanding model generates a second thought process sample corresponding to the video sample, and generates a second predicted text sample based on the second thought process sample, wherein the second video understanding model has the same network architecture as the first video understanding model; Determine the difference between the second predicted text sample and the real text, and determine the question text sample and the answer text sample corresponding to the question text sample based on the difference, wherein the real text includes the at least one real text fragment; The video samples, the question text samples, and the answer text samples are combined into a question-answering task dataset.

15. The method according to claim 14, characterized in that, The prompt word samples are also used to instruct the first video understanding model to perform a video content question-and-answer task for the video samples; The step of updating the parameters of the first video understanding model based on the first reward value includes: The video samples, question text samples, and prompt word samples in the question-answering task dataset are input into the first video understanding model for processing to obtain a third thinking process sample, and a predicted answer text sample is generated based on the third thinking process sample. Based on the predicted answer text sample and the answer text sample, a second reward value is determined; The first reward value and the second reward value are combined to obtain the third reward value; The parameters of the first video understanding model are updated based on the third reward value.

16. A method for processing video data, characterized in that, The method includes: The method acquires a first visual feature and a first audio feature of the video data to be processed, and acquires the text features of preset prompt words, wherein the prompt words are used to instruct the video understanding model to perform a video content transcription task for the video data, the sequence length of the first visual feature is the same as the sequence length of the first audio feature, and the video understanding model is trained by the method described in any one of claims 1 to 15. The video understanding model generates a thought process based on the first visual feature, the first audio feature, and the text feature, and generates predicted text based on the thought process. The first visual feature and the first audio feature are injected into network layers of different depths in the video understanding model. The thought process includes a summary of the video data and role association information. The predicted text includes structured transcribed content of multiple events.

17. A training device for a video understanding model, characterized in that, The device includes: The data processing module is used to acquire first visual feature samples and first audio feature samples of video samples, and to acquire text feature samples of preset prompt words, wherein the prompt word samples are used to instruct the first video understanding model to perform a video content transcription task for the video samples, and the sequence length of the first visual feature samples is the same as the sequence length of the first audio feature samples. The training module is used to generate a first thought process sample based on the first visual feature sample, the first audio feature sample and the text feature sample through the first video understanding model, wherein the first visual feature sample and the first audio feature sample are injected into network layers of different depths of the first video understanding model. The training module is further configured to generate a first predicted text sample based on the first thought process sample using the first video understanding model, wherein the first predicted text sample includes structured transcribed content of multiple events, and the first thought process sample includes a content summary and role association information of the video sample; The training module is further configured to divide the first predicted text sample into multiple predicted text segment samples, determine a first reward value based on the multiple predicted text segment samples and at least one real text segment corresponding to the multiple predicted text segment samples, and update the parameters of the first video understanding model based on the first reward value.

18. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method of any one of claims 1 to 15, or implements the method of claim 16.

19. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method according to any one of claims 1 to 15, or the method according to claim 16.

20. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method of any one of claims 1 to 15, or the method of claim 16.

Images