Frame parallel multi-modal state-level communication method, device and equipment for long video understanding and storage medium

Abstract

The application provides a frame parallel multimodal state level communication method, device and equipment for long video understanding and a storage medium, and relates to the technical field of artificial intelligence and computer vision. The method comprises the following steps: dividing an input video into a plurality of frame subsequences; based on a plurality of provider large language models, encoding the plurality of frame subsequences in parallel when sharing the same text context or task instruction to generate multimodal intermediate state representations; extracting visual key-value states therefrom, and performing distinguishing, screening and normalization processing to form a multimodal state key-value pair set; injecting the visual key-value states into a text condition key-value cache of a carrier large language model; based on the key-value cache after completing state injection, performing inference calculation by the carrier large language model to generate video understanding related output results. The application can guarantee communication efficiency while retaining and fusing key visual and timing information in the video more fully, and improve the overall inference performance in the long video understanding task.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and computer vision technology, and in particular to a frame-parallel multimodal state-level communication method, apparatus, device and storage medium for long video understanding. Background Technology

[0002] With the development of artificial intelligence technology, large language model systems (MLMS) are increasingly being applied to complex perception and reasoning tasks. MMLMS typically consist of multiple models with a certain degree of autonomous processing capability or large language models, collaborating to accomplish tasks that a single model cannot handle. Currently, MMLMS are widely used in natural language processing, planning and decision-making, and multimodal understanding, with their collaboration primarily relying on information exchange and coordination between large language models.

[0003] In existing large language model systems, communication between large language models is typically based on text information. Each large language model converts its own perceptions or intermediate inference results into natural language and then transmits them to other large language models via text messages. However, in video understanding tasks, video data itself possesses high dimensionality, continuous temporal structure, and rich spatial semantic information. Compressing such visual information into text descriptions inevitably leads to information loss, especially in the difficulty of fully expressing fine-grained visual features and temporal relationships. Furthermore, video understanding systems are usually limited by computational resources and context length, requiring frame sampling strategies to select a limited number of frames from long videos for processing. If the number of sampled frames is small or the temporal distribution is uneven, key visual evidence may be missed. In large language model scenarios, if multiple large language models use the same or overlapping frame sampling methods, it not only wastes computational resources but also fails to effectively improve overall video coverage. Recent research on visual and multimodal models shows that, compared to text-based intermediate representations, high-dimensional continuous state vectors within the model can more completely preserve visual information. These continuous state vectors typically contain rich spatial structure, semantic relationships, and temporal cues, making them a high-information-density representation for visual reasoning tasks. Directly utilizing and transmitting such continuous state representations helps reduce the loss caused by information compression and provides a more stable and accurate visual basis for subsequent reasoning processes.

[0004] Based on the above technical background, it is necessary to propose a new multi-language model collaborative communication method to overcome the shortcomings of existing text-based video understanding methods in terms of information retention and collaborative efficiency. Summary of the Invention

[0005] This application provides a frame-parallel multimodal state-level communication method, apparatus, device, and storage medium for long video understanding. It addresses the problems of visual information loss, insufficient temporal coverage, and limited collaborative reasoning efficiency caused by the reliance on text-based communication in existing large language model video understanding systems. By constructing a frame-parallel task division mechanism, multiple large language models process non-overlapping time segments in the video, thereby improving the system's overall temporal coverage of long videos. A multimodal state-level communication method is introduced, utilizing continuous vectorized states within the models as the information exchange carrier between the multiple language models to avoid semantic loss and instability caused by compressing high-dimensional visual information into text representations. Simultaneously, a cross-modal state injection mechanism based on key-value caching achieves effective fusion of visual information from different video segments in the carrier model without increasing cache length or computational complexity. Through these technical solutions, this application aims to improve the collaborative reasoning efficiency, information utilization, and stability of reasoning results in large language model systems for long video understanding tasks, thus providing an efficient and reliable technical implementation path for complex video understanding and multimodal reasoning applications.

[0006] In a first aspect, this application provides a frame-parallel multimodal state-level communication method for long video understanding, including: Divide the input video into multiple non-overlapping frame subsequences or time segments; Based on multiple provider large language models and carrier large language models, under the condition of sharing the same text context or task instructions, multiple frame sub-sequences are encoded in parallel to generate multimodal intermediate state representations related to their respective video segments. Visual key-value states related to visual content are extracted from each of the multimodal intermediate state representations, and the visual key-value states are distinguished, filtered and normalized to form a set of multimodal state key-value pairs. The visual key-value states are injected into the text conditional key-value cache of the carrier large language model. The injection operation is achieved through state-level mapping, modulation, or fusion operations while keeping the total length of the key-value cache of the carrier large language model unchanged. Based on the key-value cache after the state injection is completed, the carrier's large language model performs inference calculations to generate video understanding-related output results.

[0007] In one possible design, the input video is divided into multiple non-overlapping frame subsequences or time segments, including: The input video is represented as: (1) in, This indicates that the input video is available. Indicates the first Frame video data, This represents the frame index of the video data, where T represents the total number of frames in the video data. The input video is divided into multiple non-overlapping subsets of frames using the following formula: (2) And meet the following conditions: (3) in, and They represent the first The and the first A subset of frames and Indicates the frame subset index. Indicates the number of frame subsets.

[0008] In one possible design, both the provider's large language model and the carrier's large language model receive the same text prompt words. , No. Each provider's large language model processes its corresponding frame subset. A sequence of visual symbols; where the first... Input of a large language model provided by a provider Represented as: (4) After encoding, all large language models have the same text symbol sequence, but their respective processed visual symbol sequences are different. The relationship is expressed as follows: (5) in, This represents the i-th text symbol obtained after encoding processing by the first provider's large language model. This represents the i-th text symbol obtained after encoding processing by the M-th provider's large language model. This represents the j-th visual symbol generated by the m-th provider's large language model based on the corresponding frame subset. This represents the j-th visual symbol generated by the n-th provider's large language model based on the corresponding frame subset. and Indicates the frame subset index.

[0009] In one possible design, communication between the multiple provider large language models and the carrier large language model is based on the model's internal key-value cache as the carrier; for the first... For a given large language model, at any network layer, its key-value cache is defined as follows: (6) in, This represents the set of key states in the key-value cache of the m-th provider's large language model at any network layer. This represents the key vector corresponding to the i-th position in the key state set. This represents the sequence index in the key-value cache. This represents the key-value cache length of the m-th provider's large language model in the network layer. This represents the set of value states of the m-th provider's large language model in the key-value cache of the network layer. This represents the value vector corresponding to the i-th position in the set of value states; All providers' language models receive the same text prompts. Its text-related key-value states reside in a shared semantic space, specifically represented as follows: (7) in, This represents the key states related to the text prompt words in the first provider's large language model. This represents the value state related to the text prompt word in the first provider's large language model. This represents the key state related to the text prompt word in the Mth provider's large language model. This represents the value state related to the text prompt word in the Mth provider's large language model; The visually relevant key-value states are specifically represented as follows: (8) in, This represents the key state related to visual content in the m-th provider's large language model. This represents the value state related to visual content in the m-th provider's large language model. This represents the key state related to visual content in the nth provider's large language model. This represents the value state related to visual content in the nth provider's large language model.

[0010] In one possible design, visual key-value states related to visual content are extracted from each of the multimodal intermediate state representations, including: For the The provider's large language model generates the following visual key-value states: (10) in, This represents the multimodal encoding function of the m-th provider's large language model, used to generate key states and corresponding value states related to visual content based on the visual input of the text prompt word X and the corresponding frame subset; All visual key-value states generated by the provider models are collected to form a visual state set, represented as: (11) in, Represents a set of visual states.

[0011] In one possible design, the injection operation is implemented through state-level mapping, modulation, or fusion operations, and its update process is represented as follows: (12) in, This represents the key state related to the text prompt word in the carrier large language model after the injection operation. This represents the value state related to text prompt words in the carrier large language model after the injection operation is updated. This refers to the injection operator used to perform state-level mapping, modulation, or fusion on a set of visual states. This represents the key state related to the text prompt words in the large language model of the carrier before the injection operation. This represents the value state related to the text prompt words in the large language model of the carrier before the injection operation.

[0012] In one possible design, the overall state aggregation corresponding to the update process is represented as: (9) in, This represents the key state representation obtained after aggregating the overall state. This represents the value state representation obtained after aggregating the overall state. This represents the key state generated by the m-th provider's large language model. This represents the value state generated by the m-th provider's large language model. This represents an operator used to keep the key-value cache length constant while performing state fusion.

[0013] Secondly, this application provides a frame-parallel multimodal state-level communication device for long video understanding, the device comprising: The frame parallel allocation module is configured to divide the input video into multiple non-overlapping frame subsequences or time segments; The large language model processing module is configured to perform parallel encoding processing on multiple frame sub-sequences based on multiple provider large language models and carrier large language models, under the condition of sharing the same text context or task instructions, so as to generate multimodal intermediate state representations related to their respective video segments; The multimodal state key-value pair preparation module is configured to extract visual key-value states related to visual content from each of the multimodal intermediate state representations, and to distinguish, filter and normalize the visual key-value states to form a multimodal state key-value pair set. The multimodal state injection module is configured to inject the visual key-value state into the text conditional key-value cache of a carrier large language model. The injection operation is implemented through state-level mapping, modulation or fusion operations while keeping the total length of the key-value cache of the carrier large language model unchanged. The reasoning generation module is configured to perform reasoning calculations based on the key-value cache after the state injection is completed, and generate video understanding-related output results by the carrier's large language model.

[0014] Thirdly, embodiments of this application provide an electronic device, including: at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, causing the at least one processor to perform the frame-parallel multimodal state-level communication method for long video understanding as described in the first aspect and various possible designs of the first aspect.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the frame-parallel multimodal state-level communication method for long video understanding as described in the first aspect and various possible designs of the first aspect.

[0016] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the frame-parallel multimodal state-level communication method for long video understanding as described in the first aspect and various possible designs of the first aspect.

[0017] The frame-parallel multimodal state-level communication method, apparatus, device, and storage medium for long video understanding provided in this application have at least the following beneficial effects: 1. Improved the collaborative reasoning efficiency and performance of multiple language models in long video understanding. This application constructs a frame-parallel multi-model collaborative mechanism, enabling multiple provider models to process non-overlapping time segments in the video, and the carrier model to perform unified reasoning, effectively expanding the system's video coverage in the time dimension.

[0018] 2. This application avoids the loss of visual information caused by text-based communication and improves inference stability. It employs a key-value caching-based vector-level state communication method to directly transmit continuous multimodal state information within the model, avoiding the semantic loss caused by compressing high-dimensional visual features into text representations.

[0019] 3. Achieving multi-source visual information fusion without increasing cache length, demonstrating good versatility. This application utilizes a cross-modal state preparation and injection mechanism to inject visual key-value states from multiple video segments into the text key-value cache of the carrier model. This achieves cross-time segment and cross-model visual information fusion while maintaining the key-value cache length, effectively controlling storage overhead and computational complexity. It is applicable to various language model systems of different scales and various video understanding and multimodal reasoning application scenarios. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] Figure 1 The flowchart of a frame-parallel multimodal state-level communication method for long video understanding provided in this application embodiment. Figure 1 ; Figure 2 The flowchart of a frame-parallel multimodal state-level communication method for long video understanding provided in this application embodiment. Figure 2 ; Figure 3 This is a structural diagram of a frame-parallel multimodal state-level communication device for long video understanding provided in an embodiment of this application.

[0022] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0023] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0024] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0025] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0026] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0027] Currently, most multi-LLM video understanding systems employ text-to-text communication, converting information obtained from the video by various language models into natural language text for interaction. However, this approach requires compressing high-dimensional, temporally structured video signals into discrete language representations, inevitably leading to the loss of fine-grained visual semantics and temporal information. In long video understanding scenarios, due to the typically sparse frame sampling, the aforementioned text-based communication method is further prone to problems such as incomplete perceptual information and semantic drift, thus limiting the effectiveness of multi-LLM collaborative reasoning. Existing research on visual and multimodal models has shown that, compared to text representations, continuous vectorized state representations can better preserve spatial structure and semantic information in visual reasoning tasks. The visual evidence contained in high-dimensional visual states has higher information density and stability for complex reasoning tasks. Furthermore, distributed sampling across different time segments helps improve the system's overall semantic coverage of long videos.

[0028] To address the aforementioned issues, this application provides a frame-parallel multimodal state-level communication method for long video understanding. Multiple large language models perform parallel sampling and processing of different, non-overlapping frame sequences from the same video. Each language model generates corresponding multimodal state representations, and collaborative reasoning is performed through a Transformer-based key-value state sharing communication mechanism. This method achieves efficient state-level information exchange between multiple language models without relying on intermediate text representations, thereby ensuring communication efficiency while more fully preserving and integrating key visual and temporal information from the video, thus improving the overall reasoning performance in long video understanding tasks.

[0029] The core objective of this application is to enable efficient and stable collaborative reasoning across multiple language model systems in long-form video understanding tasks. To this end, this application constructs a frame-level parallel sampling strategy with clearly defined roles, allowing multiple large language models to process non-overlapping time segments within the video, thereby expanding the overall coverage of temporal information and avoiding computational redundancy caused by repeated sampling. Simultaneously, this application designs a multimodal state-level communication mechanism that preserves rich visual semantics, replacing traditional text-based intermediary information interaction. This communication method directly utilizes the continuous vectorized states within the models as communication carriers, transmitting high-information-density visual and semantic representations between multiple language models, avoiding discretization errors and semantic loss introduced during state transitions. By combining the frame-parallel sampling mechanism with the state-level communication mechanism, this application achieves effective integration of dispersed visual evidence across multiple language models without significantly increasing communication overhead and storage burden, enabling the system to obtain more comprehensive and consistent reasoning results in complex, long-temporal video understanding tasks.

[0030] This frame-parallel multimodal state-level communication method for long video understanding includes at least one provider large language model and one carrier large language model. The provider large language model and the carrier large language model operate within the same shared text context X, but receive different video visual inputs, enabling collaborative reasoning between multiple models. Figure 1 As shown, the long video to be processed is divided into different video visual inputs, such as video segment 1, video segment 2, and video segment 3. Simultaneously, a prompt word [user instruction] is introduced, and the content corresponding to this prompt word is the shared text context X. Each video visual input and prompt word is distributed to its corresponding model: video segment 1 is input to provider model 1, video segment 2 to carrier model, and video segment 3 to provider model 2. The shared text context X corresponding to the [user instruction] is simultaneously input to provider model 1, carrier model, and provider model 2. Each model processes its received video visual input and shared text context X to generate corresponding key-value states: the key-value states output by different models related to video visuals collectively constitute heterogeneous visual key-value states; while the key-value states generated by each model based on the shared text context X form shared text key-value states. The heterogeneous visual key-value states and shared text key-value states are then sent to the state preparation stage, where these key-value states are uniformly organized and preprocessed. After preprocessing, these key-value states enter the cross-modal key-value state injection module. This module integrates the key-value states across modalities through a sharing mechanism. After integration, injected key-value states are generated and transmitted to the carrier model. Finally, the carrier model performs inference calculations based on these injected key-value states and outputs the corresponding inference results, specifically [generating a video description or answer].

[0031] In one implementation, a carrier large language model is used to perform overall reasoning based on a shared textual context, while multiple provider large language models process different segments of the video in parallel, thereby providing supplementary visual state information to the carrier model. This approach allows for the introduction of visual evidence from different video segments while maintaining consistency in textual conditions.

[0032] Specifically, such as Figure 1 As shown, this method can be implemented through the following steps S10-S50.

[0033] S10: Divide the input video into multiple non-overlapping frame subsequences or time segments.

[0034] Based on the continuity of video data in the time dimension, this embodiment divides long videos into multiple non-overlapping frame subsequences or time segments through a preset sampling strategy. This achieves structured segmentation of long video content in time, providing a foundation for subsequent multi-model parallel processing, improving time coverage efficiency, and avoiding redundant calculations.

[0035] In some embodiments, in video understanding tasks, different video frames typically contain complementary visual information. To achieve efficient reasoning under limited context and computational resources, this application employs a frame-parallel video processing mechanism, in which multiple provider models process non-overlapping frame sequences in the video, while a carrier model uniformly integrates the multi-source visual information.

[0036] Let the input video be represented as: (1) in, This indicates that the input video is available. Indicates the first Frame video data, This represents the frame index of the video data, where T represents the total number of frames in the video data.

[0037] Divide the input video into A set of non-overlapping frames: (2) And meet the following conditions: (3) in, and They represent the first The and the first A subset of frames and Indicates the frame subset index. Indicates the number of frame subsets.

[0038] Each frame subset It corresponds to a video time segment or a set of sampled frames, and is processed by a provider model with the same structure.

[0039] S20: Based on multiple provider large language models and carrier large language models, under the condition of sharing the same text context or task instructions, multiple frame sub-sequences are encoded in parallel to generate multimodal intermediate state representations related to their respective video segments.

[0040] This embodiment utilizes multiple large language models provided by the provider to encode each video segment in parallel under the premise of sharing the same text context, generating multimodal intermediate states containing visual and semantic information. This improves the system throughput through parallel processing, while ensuring that each model works under the same semantic conditions, laying a consistent information foundation for subsequent state-level communication.

[0041] In some embodiments, both the Fangda language model and the carrier language model receive the same text prompts during processing. However, it receives different video visual inputs. Input of a provider model Represented as: (4) After word segmentation and encoding, the text symbol sequences for each model remain consistent, while the visual symbol sequences differ due to the different subsets of frames processed. (5) in, This represents the i-th text symbol obtained after encoding processing by the first provider's large language model. This represents the i-th text symbol obtained after encoding processing by the M-th provider's large language model. This represents the j-th visual symbol generated by the m-th provider's large language model based on the corresponding frame subset. This represents the j-th visual symbol generated by the n-th provider's large language model based on the corresponding frame subset. and Indicates the frame subset index.

[0042] This creates a multi-model reasoning scenario where shared textual states and heterogeneous visual states coexist.

[0043] S30: Extract visual key-value states related to visual content from each multimodal intermediate state representation, and distinguish, filter and normalize the visual key-value states to form a set of multimodal state key-value pairs.

[0044] This embodiment extracts key-value states related to visual content from the multimodal intermediate states generated by each model, and constructs a standardized set of states that can be used for cross-model interaction through filtering and normalization operations. This can eliminate irrelevant or low-quality information, improve the quality and consistency of state representation, provide structured and lightweight visual state input for cross-model fusion, and enable efficient collaboration between multiple models.

[0045] In one specific embodiment of step S30, the multimodal state key-value pair preparation module receives multimodal intermediate state representations generated by various provider large language models, distinguishes key and value state components strongly correlated with the visual modality based on the output of the attention layer in the model, filters the visual key-value states based on a preset information entropy threshold or attention weight, and retains key visual states with information content higher than the threshold, and further normalizes the filtered visual key-value states through layer normalization operations or linear projection matrices to align them with the text key-value cache space of the carrier large language model in terms of dimension and numerical distribution, and organizes the processed visual key-value states into a structured multimodal state key-value pair set according to the source model identifier and time segment order for subsequent cross-model injection.

[0046] S40: Inject the visual key-value state into the text conditional key-value cache of a carrier large language model. The injection operation is achieved through state-level mapping, modulation or fusion operations while keeping the total length of the key-value cache of the carrier large language model unchanged.

[0047] This embodiment injects the visual key-value states of multiple provider models into the text condition key-value cache of the carrier model through operations such as mapping, modulation, or fusion, while keeping the total length of the cache unchanged. In this way, without changing the original reasoning structure of the model, it can effectively fuse multi-source and cross-modal visual information, enhance the visual reasoning ability of the carrier model, and control the computation and storage overhead.

[0048] In some embodiments, to achieve efficient collaboration among multiple models, a vector-level state-based communication method is adopted, in which the key-value cache inside the model is used as the communication carrier between multiple models to transmit intermediate state representations.

[0049] For the For each model, at any network layer, its key-value cache is defined as: (6) in, This represents the set of key states in the key-value cache of the m-th model at any network layer. This represents the key vector corresponding to the i-th position in the key state set. This represents the sequence index in the key-value cache. This represents the key-value cache length of the m-th model in this network layer. This represents the set of value states of the m-th model in the key-value cache of this network layer. This represents the value vector corresponding to the i-th position in the set of value states.

[0050] Because all models receive the same text prompts Their text-related key-value states reside in a shared semantic space: (7) in, This represents the key states related to the text prompt words in the first model. This represents the value state related to the text prompt word in the first model. This represents the key states related to the text prompt word in the Mth model. This represents the value state related to the text prompt word in the Mth model.

[0051] The differences mainly stem from the influence of different visual conditions.

[0052] In contrast, visually related key-value states are heterogeneous because they originate from different video segments: (8) in, This represents the key state related to visual content in the m-th model. This represents the value state related to visual content in the m-th model. This represents the key states related to visual content in the nth model. This represents the value state related to visual content in the nth model.

[0053] The structural differences between the shared text state and the heterogeneous visual state provide a foundation for cross-model communication based on key-value states.

[0054] In some embodiments, in order to take into account the semantic differences between text key-value states and visual key-value states, and to avoid key-value cache bloat and additional computational overhead caused by direct concatenation, a unified cross-modal state processing framework is adopted, including a state preparation phase and a state injection phase.

[0055] The overall state aggregation process is represented as: (9) in, This represents the key state representation obtained after aggregating the overall state. This represents the value state representation obtained after aggregating the overall state. This represents the key state generated by the m-th model. This represents the state of the value generated by the m-th model. This represents an operator used to keep the key-value cache length constant while performing state fusion.

[0056] During the state preparation phase, the first A provider model in sharing text prompts Under the condition, its corresponding frame subset Encode and generate key-value states related to the visual content: (10) in, This represents the multimodal coding function of the m-th model, used to generate key and value states related to visual content under the conditions of text prompt word X and corresponding frame subset.

[0057] All visual key-value states generated by the provider models are collected to form a visual state set: (11) in, Represents a set of visual states.

[0058] During the state injection phase, the set of visual states is injected into the text key-value cache of the carrier model, and its update process is represented as follows: (12) in, This represents the key state related to the text prompt word in the carrier large language model after the injection operation. This represents the value state related to text prompt words in the carrier large language model after the injection operation is updated. This represents an injection operator used to perform state-level mapping, modulation, or fusion of text-related key-value states based on a set of visual states. This represents the key state related to the text prompt words in the large language model of the carrier before the injection operation. This represents the value state related to the text prompt words in the large language model of the carrier before the injection operation.

[0059] Through the aforementioned multimodal state-level injection operation, the multi-language model system achieves efficient utilization and integration of video visual information by introducing a collaborative mechanism between provider models and carrier models under shared text context conditions. This method first divides the input video into multiple non-overlapping frame subsets, with multiple provider models processing different video segments in parallel. This expands the coverage of long videos in the temporal dimension and forms a reasoning scenario where shared text states and heterogeneous visual states coexist. Based on this, this embodiment uses the model's internal key-value cache as a vector-level state communication carrier. Leveraging the sharing of text key-value states between different models and the differences in visual key-value states between different video segments, cross-model information complementarity is achieved. Through cross-modal state preparation and injection mechanisms, visual key-value states generated by multiple provider models are fused into the carrier model's text key-value cache without increasing the key-value cache length. This allows the carrier model to introduce visual evidence from multiple video segments while maintaining the stability of its original reasoning structure. This application achieves effective integration and collaborative reasoning of multi-source visual information under limited context and computational resources, improves the overall understanding of long video content by multi-model systems, and provides an efficient and stable technical solution for complex video understanding tasks.

[0060] S50: Based on the key-value cache after the state injection is completed, the carrier's large language model performs inference calculations to generate video understanding-related output results.

[0061] In this embodiment, the carrier model performs attention calculation and inference generation based on the key-value cache that has been injected with multi-source visual states, and outputs video understanding-related results. In this way, it can integrate visual evidence from multiple time segments, complete the understanding task in a consistent semantic context, and improve the overall accuracy and robustness of long video understanding.

[0062] In one specific embodiment of step S50, the key-value cache after multimodal state injection is used for the final inference computation of the carrier large language model. This carrier large language model receives a unified text instruction or task description as an input sequence, and this text instruction is consistent with the text context shared by the various provider large language models. During inference, the model uses the prefixed text sequence as a query vector and performs cross-modal attention computation on the cache that has fused multi-source visual key-value states. The key-value cache contains both textual conditional key-value states generated by the carrier model itself and visual key-value states injected from multiple provider large language models, corresponding to different time segments.

[0063] During each Transformer decoding operation, the model uses an attention mechanism to calculate the similarity between the query at the current text position and all keys in the cache, and then weights and aggregates the corresponding values ​​based on the calculation results. Since the visual key-value states coexist with the text key-value states in the cache through an injection mechanism, the model can simultaneously consider textual semantic information and visual evidence from multiple video segments when calculating attention weights. This design allows the model to implicitly fuse cross-modal information during inference without explicit alignment or modality transformation.

[0064] Specifically, in the calculation of the attention head, the query vector is multiplied by the key vector in the cache and scaled, and the attention weight distribution is obtained through the softmax function. Since the injection of visual key-value states does not change the total length of the cache, the space complexity of the attention calculation remains unchanged, thus maintaining computational efficiency while introducing multi-source visual information. The weighted and summed context vector is further processed by a feedforward neural network and normalized through residual connections to form the output representation of the current layer.

[0065] In the multi-round autoregressive generation process, the model uses the tokens generated in the previous time step as input for the next time step and dynamically updates its key-value cache. As generation progresses, the model continuously utilizes the injected visual state information to ensure that the output sequence maintains consistency with the visual content of multiple video segments in different generation steps. For example, in the video description generation task, the model can describe the objects, actions, and their temporal relationships in the video sequentially based on the visual states injected at different time segments; in the video question answering task, the model can synthesize the visual evidence from each segment to reason about the question and output an answer based on the fusion of information from multiple segments.

[0066] Finally, the word sequence output by the model is converted into natural language text by the decoder, forming the structured output required for video understanding tasks. The output can be a summary of the overall content of a long video, a multi-evidence answer to a specific question, or a decision suggestion derived from the video content. The entire reasoning process is completed within a unified textual semantic framework, fully leveraging the information coverage advantage of parallel processing of multiple video segments while avoiding semantic loss or inconsistency that may be caused by multiple rounds of text interaction through a state-level injection mechanism. This results in more accurate and stable reasoning performance in long video understanding tasks.

[0067] Furthermore, this inference generation method is independent of specific task formats or output structures, and can be flexibly adapted to various video understanding task paradigms such as open generation, multiple choice, and structured prediction, demonstrating the versatility and scalability of this communication mechanism in multimodal inference scenarios. By combining frame-parallel processing, state-level communication, and injection-based inference, this method can still achieve efficient and reliable understanding of long video content under resource-constrained conditions.

[0068] To further illustrate the feasibility and progressiveness of the method in this application, a detailed explanation will be provided below using an application scenario.

[0069] The method described in this application is applicable to high-efficiency, multi-model collaborative video inference system deployment scenarios, such as long video understanding systems with multiple language models that need to simultaneously process long temporal video inputs and strictly control inference latency and computational resource consumption. These systems typically operate under fixed computational resource budgets and limited context caching, making it difficult for a single model to cover the complete video content while ensuring response speed.

[0070] In the aforementioned deployment scenario, the method of this application introduces multi-source visual state information generated from processing different video segments by multiple models without increasing the key-value cache length, and injects it into the inference process of the carrier model. This enables the system to achieve visual information fusion across time segments while maintaining the stability of the original inference structure. Therefore, the system significantly reduces overall inference latency and improves the comprehensive understanding of long video content without significantly increasing storage and computational overhead, thus meeting the deployment requirements of multi-language model video understanding systems with clear constraints on inference efficiency and computational resource consumption.

[0071] Table 1 shows the performance of the two communication methods on MVLU (dev).

[0072] (Acc represents the task prediction accuracy, and Time represents the average inference time per sample, in seconds.) As shown in Table 1, the experimental results demonstrate that the key-value state communication method proposed in this application exhibits significant advantages over traditional text communication methods in terms of prediction accuracy and inference speed. By directly transmitting multimodal state information without increasing the key-value cache length, this communication method can more effectively preserve key visual semantics in the video, reduce information loss during communication, and thus improve the overall prediction performance of the model in various video understanding tasks. Simultaneously, by avoiding the additional computational overhead caused by multiple rounds of text generation and parsing, the key-value state communication method significantly reduces the average inference time per sample during inference, improving the overall efficiency of multi-model collaborative inference. The above experimental results show that the communication mechanism proposed in this application, while improving the video understanding capabilities of multiple language models, possesses higher computational efficiency and is suitable for deployment scenarios of multi-model video inference systems with requirements for both performance and latency.

[0073] Based on the methods described in the above embodiments, the main advantages of the method of this application are reflected in the following aspects: (1) A frame-parallel multi-agent division of labor mechanism is adopted to improve the information coverage efficiency of long videos. By dividing the input video into multiple non-overlapping frame subsequences and processing them in parallel by different agents, this application achieves wide coverage sampling of long videos in the time dimension without increasing the computational burden of a single model, reducing the resource waste caused by repeated frame processing and improving the ability of the multi-agent system to acquire scattered visual evidence.

[0074] (2) Introducing a multimodal state-level communication method to avoid semantic loss caused by intermediate text representation. This application uses continuously vectorized multimodal states as the communication carrier between agents to directly transmit internal state information related to visual content, avoiding the discretization error and semantic loss problem introduced by compressing high-dimensional visual signals into natural language text, thereby improving the accuracy and stability of information transmission in the process of multi-agent collaborative reasoning.

[0075] (3) Based on the cross-modal injection mechanism of key-value state, multi-source visual information is fused while maintaining the stability of the cache size. By injecting visual key-value states from multiple agents into the text condition key-value cache of the carrier agent, this application achieves cross-time segment and cross-agent visual information fusion without increasing the length of the key-value cache, effectively controlling storage overhead and computational complexity, and is suitable for resource-constrained or real-time inference scenarios.

[0076] (4) The state injection process has little interference with the original reasoning structure, ensuring the consistency and robustness of reasoning. The multimodal state injection acts on the reasoning process in a state-level manner, without changing the original input sequence structure and attention addressing relationship. While introducing supplementary visual information, it maintains the continuity and stability of the carrier model's reasoning process, reducing the risk of semantic drift during collaborative reasoning.

[0077] (5) It has good versatility and scalability, and is suitable for a variety of multimodal reasoning tasks. This application does not rely on a specific model structure or a fixed combination of modalities, and can be flexibly extended to multi-agent systems of different scales as well as a variety of video understanding and multimodal reasoning tasks, and has high engineering practical value and application prospects.

[0078] In summary, the method proposed in this application effectively addresses the technical problems of visual semantic loss, insufficient temporal coverage, and limited collaboration efficiency caused by the reliance on intermediate text representations in existing large language model video understanding systems by introducing a frame-parallel multi-language model collaboration mechanism and a key-value communication method based on multimodal states. This method achieves efficient integration of scattered visual evidence in long videos without significantly increasing computational complexity and storage overhead, thus improving the overall inference performance and stability of large language model systems in complex video understanding tasks. The technical solution has a clear structure and well-defined implementation path, exhibiting good versatility and scalability. It is applicable to various multimodal reasoning and video understanding application scenarios, possessing high engineering practical value and promising prospects for widespread adoption.

[0079] This application also provides a frame-parallel multimodal state-level communication device for long video understanding, used to implement the methods described in any of the above embodiments, such as... Figure 3 As shown, the frame-parallel multimodal state-level communication device for long video understanding includes a frame-parallel allocation module 301, a multi-language model processing module 302, a multimodal state key-value pair preparation module 303, a multimodal state injection module 304, and an inference generation module 305.

[0080] The frame parallel allocation module 301 receives input video data and processes the video according to a preset frame sampling strategy and time division rules, dividing the video into multiple non-overlapping frame sub-sequences or time segments. This module enables the reasonable splitting of long videos along the time dimension, providing a foundation for subsequent multi-model parallel processing.

[0081] The large language model processing module 302 is configured with multiple large language models working in parallel. Under the premise of sharing the same text context or task instructions, each large language model receives the frame sub-sequence allocated by the frame parallel allocation module and encodes the corresponding video segments to generate multimodal intermediate state representations related to their respective video segments.

[0082] The multimodal state key-value pair preparation module 303 is used to extract key-value state information related to visual content from the multimodal intermediate states generated by various language models, and to distinguish, filter and normalize the visual key-value states to form a set of multimodal state key-value pairs that can be used for cross-model interaction.

[0083] The multimodal state injection module 304 is used to inject visual key-value states from multiple large language models into the textual conditional key-value states of the carrier large language model without increasing the length of the target model's key-value cache. This module integrates visual information across large language models and time segments through state-level mapping, modulation, or fusion operations, thereby introducing supplementary visual evidence while maintaining the stability of the original inference structure.

[0084] The inference generation module 305, based on the key-value cache after the state injection is completed, performs inference calculations on the input task and generates the final output result. The output result can be a text description, question answer, or decision result related to video understanding, which reflects the comprehensive understanding ability after collaborative inference by multiple language models.

[0085] This application provides an electronic device. The electronic device may include a processor and a memory, wherein the processor and the memory can communicate; exemplarily, the processor and the memory communicate via a communication bus.

[0086] The processor executes computer execution instructions stored in memory, causing the processor to perform the scheme in the above embodiments. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0087] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0088] The electronic device provided in this application embodiment can be the terminal device described in the above embodiments.

[0089] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer performs the technical solution of the frame-parallel multimodal state-level communication method for long video understanding described in the above embodiments.

[0090] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the technical solution of the frame-parallel multimodal state-level communication method for long video understanding described in the above embodiments.

[0091] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0092] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0093] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0094] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0095] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0096] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0097] Buses can be Industry Standard Architecture (ISA) buses, Peripheral Component Interconnect (PCI) buses, or Extended Industry Standard Architecture (EISA) buses, etc. Buses can be categorized into address buses, data buses, control buses, etc.

[0098] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0099] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic control unit or main control device.

[0100] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A frame-parallel multimodal state-level communication method for long video understanding, characterized in that, The method includes: Divide the input video into multiple non-overlapping frame subsequences or time segments; Based on multiple provider large language models and carrier large language models, under the condition of sharing the same text context or task instructions, multiple frame sub-sequences are encoded in parallel to generate multimodal intermediate state representations related to their respective video segments. Visual key-value states related to visual content are extracted from each of the multimodal intermediate state representations, and the visual key-value states are distinguished, filtered and normalized to form a set of multimodal state key-value pairs. The visual key-value states are injected into the text conditional key-value cache of the carrier large language model. The injection operation is achieved through state-level mapping, modulation, or fusion operations while keeping the total length of the key-value cache of the carrier large language model unchanged. Based on the key-value cache after the state injection is completed, the carrier's large language model performs inference calculations to generate video understanding-related output results.

2. The frame-parallel multimodal state-level communication method for long video understanding according to claim 1, characterized in that, The input video is divided into multiple non-overlapping frame subsequences or time segments, including: The input video is represented as: （1） in, This indicates that the input video is available. Indicates the first Frame video data, This represents the frame index of the video data, where T represents the total number of frames in the video data. The input video is divided into multiple non-overlapping subsets of frames using the following formula: （2） And meet the following conditions: （3） in, and They represent the first The and the first A subset of frames and Indicates the frame subset index. Indicates the number of frame subsets.

3. The frame-parallel multimodal state-level communication method for long video understanding according to claim 1, characterized in that, Both the provider's large language model and the carrier's large language model receive the same text prompt words. , No. Each provider's large language model processes its corresponding frame subset. A sequence of visual symbols; where the first... The input of a large language model provided by the provider Represented as: （4） After encoding, all large language models have the same text symbol sequence, but their respective processed visual symbol sequences are different. The relationship is expressed as follows: （5） in, This represents the i-th text symbol obtained after encoding processing by the first provider's large language model. This represents the i-th text symbol obtained after encoding processing by the M-th provider's large language model. This represents the j-th visual symbol generated by the m-th provider's large language model based on the corresponding frame subset. This represents the j-th visual symbol generated by the n-th provider's large language model based on the corresponding frame subset. and This represents the frame subset index, where m ≠ n.

4. The frame-parallel multimodal state-level communication method for long video understanding according to claim 1, characterized in that, The communication between the multiple provider large language models and the carrier large language model is based on the key-value cache within the model as the carrier; for the first For a given large language model, at any network layer, its key-value cache is defined as follows: （6） in, This represents the set of key states in the key-value cache of the m-th provider's large language model at any network layer. This represents the key vector corresponding to the i-th position in the key state set. express, This represents the key-value cache length of the m-th provider's large language model at this network layer. This represents the set of value states in the key-value cache of the m-th provider's large language model at this network layer. This represents the value vector corresponding to the i-th position in the set of value states; All providers' language models receive the same text prompts. Its text-related key-value states reside in a shared semantic space, specifically represented as follows: （7） in, This represents the key states related to the text prompt words in the first provider's large language model. This represents the value state related to the text prompt word in the first provider's large language model. This represents the key state related to the text prompt word in the Mth provider's large language model. This represents the value state related to the text prompt word in the Mth provider's large language model; The visually relevant key-value states are specifically represented as follows: （8） in, This represents the key state related to visual content in the m-th provider's large language model. This represents the value state related to visual content in the m-th provider's large language model. This represents the key state related to visual content in the nth provider's large language model. This represents the value state related to visual content in the nth provider's large language model.

5. The frame-parallel multimodal state-level communication method for long video understanding according to claim 4, characterized in that, Extracting visual key-value states related to visual content from each of the multimodal intermediate state representations, including: For the The provider's large language model generates the following visual key-value states: （10） in, This represents the multimodal coding function of the m-th provider's large language model, which is used to generate key states and corresponding value states related to visual content when receiving visual input of text prompt word X and the corresponding m-th frame subset; All visual key-value states generated by the provider models are collected to form a visual state set, represented as: （11） in, Represents a set of visual states.

6. The frame-parallel multimodal state-level communication method for long video understanding according to claim 5, characterized in that, The injection operation is implemented through state-level mapping, modulation, or fusion operations, and its update process is represented as follows: （12） in, This represents the text-related key state of the carrier large language model after the injection operation. This represents the text-related value state of the carrier large language model after the injection operation. This refers to the injection operator used to perform state-level mapping, modulation, or fusion. This indicates the key text state of the large language model before the injection operation. This represents the text-related value state of the carrier large language model before the injection operation.

7. The frame-parallel multimodal state-level communication method for long video understanding according to claim 6, characterized in that, The overall state aggregation corresponding to the update process is represented as follows: （9） in, This represents the global bond-state representation obtained after state fusion. This represents the overall value state representation obtained after state fusion. This represents the key state generated by the m-th provider's large language model. This represents the value state generated by the m-th provider's large language model. This represents an operator used to keep the key-value cache length constant while performing state fusion.

8. A frame-parallel multimodal state-level communication device for long video understanding, characterized in that, The device includes: The frame parallel allocation module is configured to divide the input video into multiple non-overlapping frame subsequences or time segments; The large language model processing module is configured to perform parallel encoding processing on multiple frame sub-sequences based on multiple provider large language models and carrier large language models, under the condition of sharing the same text context or task instructions, so as to generate multimodal intermediate state representations related to their respective video segments; The multimodal state key-value pair preparation module is configured to extract visual key-value states related to visual content from each of the multimodal intermediate state representations, and to distinguish, filter and normalize the visual key-value states to form a multimodal state key-value pair set. The multimodal state injection module is configured to inject the visual key-value state into the text conditional key-value cache of a carrier large language model. The injection operation is implemented through state-level mapping, modulation or fusion operations while keeping the total length of the key-value cache of the carrier large language model unchanged. The reasoning generation module is configured to perform reasoning calculations based on the key-value cache after the state injection is completed, and generate video understanding-related output results by the carrier's large language model.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the frame-parallel multimodal state-level communication method for long video understanding as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the frame-parallel multimodal state-level communication method for long video understanding as described in any one of claims 1-7.

Images