Methods for enhancing the interactive capabilities of large multimodal models with streaming video content

Abstract

The present application proposes a method for enhancing the interaction capability of large multi-modal models with streaming video content, called StreamChat. To address the delay problem of existing models in streaming interaction, StreamChat dynamically updates the visual context to ensure that the model interacts with the latest video content, effectively reducing the delay. The method uses a flexible and efficient cross-attention architecture to optimize the processing of dynamic streaming input while maintaining inference efficiency. The introduction of the parallel 3D-RoPE mechanism and the dense instruction dataset enhances the model's ability to encode relative temporal information between visual and text tokens, improving the continuity of streaming interaction. Experimental results show that StreamChat performs competitively in image and video benchmark tests and outperforms existing advanced models in streaming interaction scenarios. This invention provides an efficient and powerful solution for real-time interaction with streaming video content, significantly improving the interaction experience and reducing inherent delays in dynamic environments.

Description

Technical Field

[0001] This invention relates to large multimodal models (LMMs) and streaming video technology, and in particular to a method for enhancing the interactivity of large multimodal models and streaming video content. Background Technology

[0002] The rise of large language models (LLMs) [4,12,21,35,39,48] and large multimodal models (LMMs) [10,24,46,50] has unlocked numerous application scenarios, including visual instruction following [28-31] and long video understanding [58,67]. In particular, cutting-edge models like GPT-4o

[37] and Gemini

[44] have demonstrated remarkable capabilities in interacting with streaming video, generating considerable interest in the field.

[0003] Large multimodal models (LMMs) have garnered significant attention for their strong zero-shot capabilities across a variety of tasks, including image captioning

[27] and visual question answering (VQA) [16,18,36]. In particular, Flamingo [2] demonstrated visual context learning by training on a large number of interleaved image-text datasets. GPT-4V [1] demonstrated emerging image understanding capabilities, providing coherent responses to user queries. In the open-source space, LLaVA replicated some of the capabilities of GPT-4V by fine-tuning on generated instruction-following data. Subsequent works, including LLaVA-1.5

[28] , Qwen-VL [5], and CogVLM

[51] , aimed to enhance model capabilities through architectural improvements, improved training methods, and higher-quality training datasets. Recently, open models such as InternVL2

[10] , LLaVA-OneVision

[24] and Qwen2-V L

[50] have shown better performance than state-of-the-art closed models such as GPT-4o

[37] or Gemini-1.5-Pro

[44] , paving the way for LMM research. Streaming video models. The emergence of streaming video models began with OpenAI’s GPT-4o

[37] , which showed remarkable capabilities in real-time interaction with streaming video, attracting great interest in the field. In its subsequent development, several works have aimed to enhance the ability of large multimodal models to process streaming video content. Notable approaches include VideoLLM-online[7], Flash-VStream

[65] , VITA

[15] and VideoLLM-MoD

[53] , which focus on improving the fluency or responsiveness of models during streaming video processing. However, existing models often rely on fixed video content when the question is posed to answer the question.

[0004] While some open approaches [7,53,56,65] have recently emerged to enhance streaming video processing, they remain insufficient in terms of interactive fluency and perceptual capabilities. To effectively interact with streaming video, LMMs must not only accurately identify the visual content of each frame but also track dynamic changes within the streaming video, utilizing the latest visual information to answer questions such as… Figure 1 As shown. Although LMMs have made significant progress in video understanding [11,45,57,64,68], existing models often overlook the importance of capturing dynamic changes, which negatively impacts the interactive experience. Specifically, current methods typically rely on video information at the time the question is posed; however, streaming content can change significantly during decoding, making it impossible for the model to understand these updates. For example, suppose a question is posed at time t, and the model needs t' seconds to answer it. Existing methods only use video content from the time interval 0 to t to answer the question, making it impossible for the model to understand any changes that occur between t and t+t'. This delay is particularly detrimental in highly dynamic video environments or when the question answer is long, leading to a poor user experience. Figure 2 This issue is explained in (at the top).

[0005] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The main objective of this invention is to overcome the deficiencies in the aforementioned background technology and provide a method for enhancing the interaction capabilities of large-scale multimodal models with streaming video content.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for enhancing the interactivity of large multimodal models with streaming video content includes:

[0009] S1. Cross-attention processing: Using the cross-attention mechanism, the input text is converted into an encoding form that the model can understand through the text encoder, and the video frames are converted into an encoding form that the model can understand through the visual encoder. This allows the model to dynamically integrate visual and text information during the decoding process, thereby enhancing the model's understanding of dynamic content.

[0010] S2. Parallel 3D-RoPE Processing: Corresponding to the three dimensions of time, height, and width in the video, by performing time encoding, height encoding, and width encoding on each visual and text token in parallel, information of time dimension, spatial height dimension, and spatial width dimension is added to each token to encode the relative time position of the token in the video, thereby enhancing the continuity of streaming media interaction;

[0011] S3. Model Training: The model is trained using a dense instruction dataset containing multiple triples, where each triple includes a time interval, an instruction, and an answer. The words in each instruction-answer pair are heuristically timestamped. An attention mask is applied to ensure temporal consistency between training and inference. This attention mask is generated based on timestamp information, ensuring that each text token only focuses on video information before its corresponding timestamp, shielding future information. Since visual and text tokens are organized in parallel, visual and text tokens with the same timestamp share the same temporal context in RoPE to enhance continuity during streaming interactions.

[0012] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for enhancing the interactivity of large multimodal models with streaming video content.

[0013] A computer program product includes a computer program that, when executed by a processor, implements the method for enhancing the interactivity of large multimodal models with streaming video content.

[0014] The present invention has the following beneficial effects:

[0015] This invention proposes a method called StreamChat to enhance the interaction capabilities of large multimodal models (LMMs) with streaming video content. This novel method significantly improves the interaction capabilities of LMMs with streaming video content. In streaming interaction scenarios, traditional methods rely solely on visual information at the time of question posing, leading to significant latency as the model fails to capture dynamic changes in video content in a timely manner. StreamChat effectively solves this problem by dynamically updating the visual context at each step of the decoding process, ensuring the model always utilizes the latest video content. This invention employs a flexible and efficient cross-attention architecture, which not only optimizes the model's ability to handle dynamic streaming input but also maintains inference efficiency. By introducing a parallel 3D-RoPE mechanism, StreamChat encodes the relative temporal information of visual and text tokens. This mechanism provides rich positional information for each token through three dimensions: time, height, and width, enhancing the model's understanding of video content and the continuity of interaction. Furthermore, StreamChat simulates the actual conditions of streaming interaction by constructing a new dense instruction dataset, enabling the model to better adapt to the dynamic changes of streaming video content during training. During training, attention masks are used to ensure that each text token can only focus on video information before its corresponding timestamp, effectively simulating streaming media interaction conditions and improving the model's real-time response capability. Experimental results show that StreamChat of this invention exhibits competitive performance on image and video benchmarks and surpasses existing state-of-the-art models in streaming media interaction scenarios. The StreamChat-7B model, in particular, demonstrates superior performance compared to the LLaVA-Video-72B model in evaluations, proving its excellence in streaming media interaction scenarios. In summary, StreamChat of this invention features innovative architectural design and training methods, which make the model more efficient and accurate in processing streaming video content, significantly improving the streaming media interaction experience, reducing inherent latency in dynamic environments, and providing new possibilities for the application of large-scale multimodal models in the field of real-time interaction.

[0016] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0017] Figure 1 An example of the application of the StreamChat of the present invention in streaming video is shown.

[0018] Figure 2 This paper illustrates a comparison of the context usage during the decoding process between the StreamChat of this invention and existing models.

[0019] Figure 3 The architecture of StreamChat of the present invention is shown.

[0020] Figure 4 The parallel 3D-RoPE mechanism of StreamChat of the present invention is shown.

[0021] Figure 5 The present invention’s StreamChat is shown in comparison with leading video LMMs in streaming media evaluation.

[0022] Figure 6 A qualitative evaluation of the StreamChat of the present invention on streaming video is shown. Detailed Implementation

[0023] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0024] Recent advancements in large multimodal models (LMMs) [11,45,57,64,68] have significantly improved their video understanding capabilities. However, in interactive streaming scenarios, LMMs must also accurately capture the dynamic changes in streaming video content, which is neglected by existing models. To address this gap, this invention proposes a method called StreamChat to enhance the interaction capabilities of large multimodal models (LMMs) with streaming video content. StreamChat is a novel type of LMM that can smoothly interact with streaming video and track the latest changes in the video to refine its answers. StreamChat effectively enhances the interaction capabilities of large multimodal models (LMMs) with streaming video content. In interactive streaming scenarios, existing methods rely solely on the visual information available when a question is posed, leading to significant delays in the model when it is unaware of subsequent changes in the streaming video. StreamChat addresses this limitation by updating the visual context at each decoding step, ensuring that the model uses the latest video content throughout the decoding process. Furthermore, a flexible and efficient cross-attention-based architecture is introduced to handle dynamic streaming input while maintaining inference efficiency for interactive streaming. Furthermore, a new dense instruction dataset was constructed to facilitate the training of the streaming interaction model, supplemented by a parallel 3D-RoPE mechanism to encode the relative temporal information of visual and text tokens. Experimental results show that StreamChat achieves competitive performance on established image and video benchmarks and demonstrates superior capabilities compared to state-of-the-art video LMMs in streaming interaction scenarios.

[0025] This invention provides a method for enhancing the interaction capabilities of large-scale multimodal models with streaming video content, comprising the following steps:

[0026] Step S1. Cross-attention processing: Using the cross-attention mechanism, the input text is converted into an encoding form that the model can understand through the text encoder, and the video frames are converted into an encoding form that the model can understand through the visual encoder. This allows the model to dynamically integrate visual and text information during the decoding process, thereby enhancing the model's understanding of dynamic content.

[0027] Step S2. Parallel 3D-RoPE Processing: Corresponding to the three dimensions of time, height, and width, time encoding, height encoding, and width encoding are performed on each visual and text token in parallel. This adds information in the time dimension, spatial height dimension, and spatial width dimension to each token to encode the token's relative temporal position in the video, enhancing the continuity of streaming media interaction. Here, 3D-RoPE (Relative Position Embedding) refers to extending the traditional 1D-RoPE to three-dimensional space and arranging visual and text tokens in parallel, so that the RoPE embedding vector contains three independent dimensions: time, height, and width, corresponding to the token's position on the video timeline, its position in the vertical direction of the video frame, and its position in the horizontal direction of the video frame, respectively.

[0028] Step S3. Model Training: The model is trained using a dense instruction dataset containing multiple triples, where each triple includes a time interval, an instruction, and an answer. The words in each instruction-answer pair are heuristically timestamped. An attention mask is applied to ensure temporal consistency between training and inference. This attention mask is generated based on timestamp information, ensuring that each text token only focuses on video information before its corresponding timestamp, shielding future information. Since visual and text tokens are organized in parallel, visual and text tokens with the same timestamp share the same temporal context in RoPE, thereby enhancing continuity during streaming interactions.

[0029] The method described in this invention, called StreamChat, significantly improves the real-time interaction capabilities of large-scale multimodal models with streaming video content. Addressing the latency issues of existing models in streaming media interaction, StreamChat dynamically updates the visual context, ensuring the model interacts with the latest video content, effectively reducing latency. This method employs a flexible and efficient cross-attention architecture, optimizing the processing of dynamic streaming media input while maintaining inference efficiency. The introduced parallel 3D-RoPE mechanism and dense instruction dataset enhance the model's ability to encode relative temporal information of visual and text tokens, improving the continuity of streaming media interaction. Experimental results show that StreamChat performs competitively in image and video benchmarks and surpasses existing state-of-the-art models in streaming media interaction scenarios. This invention provides an efficient and powerful solution for real-time interaction with streaming video content, significantly improving the interactive experience and reducing inherent latency in dynamic environments.

[0030] like Figure 3 As shown, in a preferred embodiment, step S1 integrates visual and text tokens through a cross-attention mechanism, specifically including: given an input streaming video, extracting visual tokens for each sampled frame using a pre-trained visual model; inserting a cross-attention block into a large language model (LLM) architecture, bridging the visual and text tokens using a cross-attention mechanism, where the text token serves as the query (Q) and the visual token serves as the key (K) and value (V); dynamically updating the visual tokens to reflect changes in the streaming video; wherein the cross-attention block shares parameters with the self-attention block of the LLM to improve the convergence speed of training.

[0031] like Figure 3As shown, in a preferred embodiment, step S1 further utilizes a visual feedforward network (V-FFN) expert to enhance visual representations throughout the forward pass of the large language model (LLM). Specifically, this includes: updating the visual tokens using the V-FFN expert during the LLM's forward pass, and inputting the updated visual tokens into subsequent cross-attention blocks; whereby the V-FFN expert allows the visual representations to better align with the hidden states of the LLM, improving final performance; further processing of textual information through the LLM's self-attention blocks and the feedforward network (FFN); repeating the above processing N times to achieve dynamic updating of the visual tokens during the model's forward pass. Preferably, the V-FFN expert is initialized from the LLM's feedforward network (FFN) to inherit the LLM's pre-trained knowledge. Preferably, a linear gate mechanism is used to scale the outputs of the cross-attention and V-FFN blocks to mitigate the vanishing gradient problem and stabilize the training process.

[0032] In a preferred embodiment, step S2, which implements parallel 3D-RoPE processing to simulate the positional information of streaming video and text, specifically includes: dividing the RoPE embedding vector into three dimensions, corresponding to time, height, and width information respectively; for text tokens, the values ​​of the three dimensions are the same, jointly identifying the token's position in the time series; for visual tokens, the three dimensions respectively identify the token's timestamp, vertical position, and horizontal position in the video frame; wherein, at any given timestamp, the corresponding text token and visual token share the same contextual information through the three dimensions, achieving time alignment.

[0033] In a preferred embodiment, step S3, preparing a dense instruction dataset for model training, specifically includes: creating a new video instruction adjustment dataset from an existing dense description dataset, the dataset containing captions paired with timestamp intervals; and generating a series of instruction-answer pairs based on the captions of the video clips using a large language model (LLM).

[0034] In a preferred embodiment, step S3 further optimizes the construction process of the dense instruction dataset, specifically including: analyzing the initially generated instruction-answer pairs through clustering methods to reduce the similarity between instructions and enhance the diversity of instruction data; manually reviewing and using the remaining examples as context examples for subsequent data generation; and collecting examples from multiple datasets representing different environments to cover a wide range of streaming media interaction scenarios.

[0035] In a preferred embodiment, step S3, the data arrangement process for model training includes: converting the coarse timestamp annotation of the initial instruction data into fine-grained timestamp annotation, and simulating the time dynamics during streaming media interaction by assigning a timestamp to each word in the instruction; and using a heuristic method to assign a timestamp to each word in the instruction data so that the model can simulate real-time interaction during the decoding process.

[0036] In a preferred embodiment, step S3, applying the attention mask, includes: using the attention mask to prevent timestamps from being... <t>Token Follow Timestamp <t>Subsequent video frames are used to ensure the model's temporal consistency during training and inference; an attention masking mechanism allows the model to focus only on the relevant visual information available in each decoding step, thus maintaining the temporal integrity of streaming interactions.

[0037] In a preferred embodiment, in step S3, the model's inference process includes: continuously reading the video stream using a separate thread and storing the extracted visual tokens in a first-in-first-out (FIFO) queue; when the LLM needs to decode to generate a response, retrieving the latest visual token from the FIFO queue and integrating this current information into decoding the next token; ensuring that the model's response is guided by the latest video stream context, enhancing the relevance of the model's output, and supporting seamless streaming interaction with dynamic video content.

[0038] In a preferred embodiment, step S3, prior to model training, further includes the following pre-training steps: implementing a two-stage pre-training process, gradually unfreezing pre-training parameters to achieve more effective pre-training; using a combination of multiple data sources in the two stages of pre-training, including video description datasets and caption datasets; in the first stage, training only the adapter to achieve alignment between model components; in the second stage, unfreezing and further training the visual encoder and the visual feedforward network (V-FFN) expert to achieve deep alignment of the model; for the caption dataset, training is performed at a lower frame rate, while for other video data, training is performed using uniformly sampled frames; through the above pre-training process, a large number of samples are used to enhance the model's ability to understand streaming video content.

[0039] In some embodiments, the StreamChat proposed in this invention enables the LLM to dynamically interact with streaming video content. Its core idea is to provide the LLM with the latest video information at each decoding step, allowing it to better capture video dynamics and adjust its response accordingly. Mechanistically, StreamChat enhances the model's ability to interact with streaming video data, ensuring a more time-aligned response, such as... Figure 1 As shown. In Figure 1 In the example, the question is posed at the 11th second. As the model outputs its text stream, it continuously follows the dynamic content of the streaming video and answers the question using the latest video content. Figure 2 As shown at the bottom, for each text token, Figure 2 The black and blue arrows in the diagram represent the start and end of the visual context used, respectively. While existing models (top) use a fixed visual context during decoding, StreamChat (bottom) aligns the video and text streams temporally and dynamically updates its visual context based on the streaming video. To effectively handle the dynamic visual input of streaming video, this invention designs a flexible and efficient architecture based on a cross-attention mechanism [3,12,49] to connect the visual inputs in LLM and StreamChat. The cross-attention design facilitates handling variable-length inputs in streaming scenarios and is more efficient when processing a large number of visual tokens. To facilitate the training of the streaming interaction model, this invention introduces a dense instruction dataset to train StreamChat. Existing video instruction tuning datasets [17,22,25,43,68] primarily focus on offline video understanding, i.e., the model can perceive the complete video before answering questions. This differs from streaming interaction scenarios, where video content dynamically changes during the answering process. To bridge this gap, this invention creates a new dense instruction dataset based on existing dense description datasets. A dense instruction dataset consists of several (time interval, instruction, answer) triples, where each word in the instruction-answer pair is heuristically timestamped. During training, this invention uses an attention mask to ensure that each text token only focuses on video information preceding its corresponding timestamp. This approach effectively simulates the conditions of streaming interaction throughout the training process. Importantly, this invention does not directly input the absolute timestamp of each token into the model, as these timestamps are unavailable during inference. Instead, this invention proposes a parallel 3D-RoPE mechanism that enables each token to be aware of its relative temporal position in the video. This invention uses three components of RoPE to represent time, height, and width. Unlike existing methods that interleave video and text [42,50], this invention organizes them in parallel to ensure that visual and text tokens with the same timestamp share the same temporal context in RoPE, which enhances continuity during streaming interaction.

[0040] Extensive experiments demonstrate that StreamChat not only achieves competitive performance on established image and video benchmarks but also significantly improves its capabilities in interactive streaming scenarios. Specifically, this invention creates a benchmark designed to evaluate the performance of LMMs in interactive streaming scenarios. It proves that the StreamChat-7B model of this invention outperforms the state-of-the-art LLaVA-Video-72B model.

[0041] The following describes specific embodiments of the present invention.

[0042] StreamChat architecture

[0043] To support streaming video content, this invention designs a flexible and efficient architecture capable of processing dynamic video input through a cross-attention mechanism. Furthermore, this invention introduces a Visual Feedforward Network (V-FFN) expert to enhance visual representations during the forward process of the Large Language Model (LLM). This invention also proposes a parallel 3D-RoPE mechanism to better encode temporal information in interactive streaming scenarios. The architecture is as follows: Figure 3 As shown, a cross-attention block is used to bridge the visual and text tokens, and a V-FFN block is used to update the visual token throughout the LLM forward pass. The outputs of these two blocks are scaled using a linear gating mechanism.

[0044] Cross attention

[0045] This invention constructs a cross-attention-based architecture to bridge visual and text tokens. Given an input streaming video, this invention uses a pre-trained visual model to extract visual tokens for each sampled frame. To integrate these visual tokens with the LLM, this invention inserts several cross-attention blocks into the LLM architecture, where text tokens serve as queries and visual tokens serve as keys and values. During the interaction, the visual tokens are dynamically updated, and the cross-attention design facilitates handling these dynamic inputs. Furthermore, compared to self-attention-based architectures (e.g., LLaVA

[30] ), cross-attention is much more efficient when there are far more visual tokens than text tokens, especially in streaming interactions with high frame rates (FPS) for inference. In fact, the cross-attention blocks of this invention share parameters with the self-attention blocks of the LLM to improve the convergence speed during training.

[0046] This invention further utilizes V-FFN experts to enhance visual representations throughout the forward pass of the LLM. Specifically, after each cross-attention block, the visual token is updated using V-FFN experts, and the updated token is input into subsequent cross-attention blocks. Unlike previous cross-attention-based models [3,8,12,60], which use the same visual representation for all cross-attention blocks, the V-FFN experts of this invention allow the visual representation to be better aligned with the hidden states of the LLM and improve the final performance. In fact, these V-FFN experts are initialized from the FFN of the LLM, rather than trained from scratch to inherit the pre-trained knowledge of the LLM. Previous cross-attention-based models [3,12] typically use tanh gating mechanisms to ensure that the language model produces the same results as the original LLM in the early stages and is stable during training. However, the tanh function suffers from the vanishing gradient problem, resulting in poor performance. Instead, this invention introduces a linear gating to scale the outputs of the cross-attention and V-FFN blocks to a relatively small range in the initial training phase, following CaiT

[47] . Linear gating mechanisms alleviate gradient problems and stabilize the training process.

[0047] Parallel 3D-RoPE

[0048] To better simulate the positional information of streaming video and text, this invention proposes a parallel 3D-RoPE, which extends the traditional 1D-RoPE to 3D space and arranges visual and text tokens in parallel. Specifically, the RoPE embedding is divided into three components. For text tokens, these components are identical, representing the temporal position of each token. For visual tokens, these three components represent the temporal, height, and width positions of each token, respectively. Unlike previous methods that arranged visual and text tokens in an interleaved manner

[50] , they are arranged in parallel, such as... Figure 4 As shown. For visual and text tokens with the same timestamp, they share the same temporal position. Given a text token and a visual token at the same timestamp, the same time index is applied to them. The inventors intuitively recognized that in streaming settings, text and visual tokens at a particular timestamp occur simultaneously and therefore should share the same temporal position, rather than staggered. Parallel permutations are crucial for high FPS inference in streaming settings, and traditional permutations can result in significant temporal position gaps between two adjacent text tokens, while the method of this invention ensures their continuity.

[0049] Dense instruction data

[0050] Existing video instruction tuning datasets [17,22,25,43,68] have made significant progress in offline video understanding, i.e., the model can see the entire video before answering a question. However, these datasets are not suitable for training streaming interaction models where the input is a streaming video and each text token can only see a portion of the video. For example, a text token with timestamp t can only perceive video frames before timestamp t. To address this issue, a new video instruction tuning dataset was created from an existing dense description dataset, where captions are paired with timestamp intervals. Given a video with its dense captions, the present invention prompts an LLM (e.g., Gemini-1.5-Pro

[44] ) to select the start time of a video segment and then generate an instruction-answer pair based on the captions of that segment. The LLM is guided to focus on the streaming interaction scenario and generate relevant instructions. To enhance the diversity of the instruction data, 5k pairs were initially generated and clustered to eliminate highly similar instructions. The remaining examples were manually reviewed and used as contextual examples for subsequent data generation. Ultimately, a total of 51k examples were collected from two dense captioning datasets, Ego4D

[17] and Vript

[59] , one representing an egocentric environment and the other representing a natural environment.

[0051] Training and reasoning

[0052] Data Arrangement. Given that the initial instruction data has coarse timestamp annotations, i.e., in the form of (time interval, instruction, answer), a heuristic method is used to assign a timestamp to each word in the instruction data. For example, consider a triple with a time interval of 5-10 seconds, where the instruction is "What is the person in the video doing now?" and the answer is "The person is cooking right now." To generate fine-grained timestamp annotations, this coarse triple is transformed into a sequence of words including time indicators. The transformation result is as follows:

[0053] instruction: <5> What is the person in the video doing now?

[0054] Answer: <5> The <6> person <7> is <8> cooking <9> right <10> now.

[0055] here, <t>This represents the second t. The intuition behind this design is that the instructions are entered by the user on the fly, and the answer is decoded by the model token by token. For this example, assume the model decodes one token per second. Note that... <t>The indicators are not directly input into the model, but are for reference only.

[0056] Attention mask. To ensure the timestamp is... <t>The token will not pay attention to what happens. <t>In subsequent video frames, attention masks are used to prevent this attention. This mechanism is crucial for maintaining the temporal integrity of streaming interactions, allowing the model to focus only on the relevant visual information available at each decoding step.

[0057] Inference. During inference, StreamChat employs a parallel approach to ensure efficient processing of streaming video content. Specifically, a separate thread continuously reads the video stream and stores the extracted visual tokens in a first-in, first-out (FIFO) queue. When the LLM needs to decode to generate a response, it retrieves the latest video token from the FIFO queue. The model then integrates this current information into decoding the next token, ensuring its response is guided by the latest video stream context. This design not only enhances the relevance of the model's output but also supports seamless streaming interaction with dynamic video content, enabling users to interact with it effectively.

[0058] Experimental setup

[0059] This section outlines the experimental setup used in the study. The model of the present invention was constructed using a SigLIP visual encoder

[63] and PaliGemma weights [6] and a 7B / 14B Qwen 2.5LLM

[45] . A multilayer perceptron (MLP) adapter

[30] was used to align the hidden dimensions of the visual and language components.

[0060] Pre-training

[0061] A two-stage pre-training process was implemented, with pre-training parameters gradually unfrozen to achieve more efficient pre-training. In both stages, a combination of the ReCap data from LLaVA-Next

[29] , a portion of the InternVL pre-training data

[10] , MMC4

[71] , and the dense captioning dataset[17,22,43,59,70] was used. In the first stage, only the MLP adapter was trained for alignment. The MLP was trained for 5000 steps with a maximum learning rate of 5×10^-4 and a batch size of 512. In the second stage, the visual encoder and visual feedforward network (V-FFN) experts were further unfrozen to achieve deeper alignment. The MLP was trained for 5000 steps with a maximum learning rate of 2×10^-5 and a batch size of 512. For the dense captioning dataset, training was performed at 1 frame per second (FPS) and a maximum of 40 frames. For other video data, training was performed by uniformly sampling 40 frames. In total, 5.1 million samples were used for pre-training.

[0062] Instruction adjustment

[0063] Two versions of the instruction tuning dataset were constructed: one based on Eagle-1.8M

[40] and the other based on LLaVAOneVision

[24] . In addition, the dense instruction dataset of this invention and LLaVA-Video

[68] were also incorporated into the instruction tuning. For ablation studies, the Eagle-1.8M version was used by default. All parameters were unfrozen and trained for 1 epoch on the dataset combination. The maximum learning rate was 2×10^-5 and the batch size was 768. For the dense instruction data, training was performed using 1 FPS and a maximum of 32 frames. For other video instruction data, training was performed by uniformly sampling 32 frames. In total, 2.9 million samples were used for instruction tuning.

[0064] Streaming evaluation

[0065] To evaluate the streaming interaction capabilities of large multimodal models (LMMs), a streaming evaluation benchmark was constructed from existing dense caption datasets. Based on the dense captions of the video, this invention prompts Gemini-1.5-Pro ​​to generate instruction-answer pairs with specific timestamps. Samples irrelevant to the streaming scenario were removed, and each remaining sample was manually reviewed and refined to ensure that the instructions and answers were aligned with the video content and timestamps. Finally, 100 evaluation samples were collected, 80 from Vript

[59] and 20 from Ego4D

[17] . Following [30,31], Gemini-1.5-Pro ​​was used as the judge for performance evaluation. Given a video and its corresponding instructions, two candidate models were inferred to predict their respective answers. The correct answer was then provided to the judge along with the outputs of the two models. The judge was required to evaluate the two answers based on consistency, usefulness, relevance, and accuracy. This invention prompts the judge to determine which model's answer is better, or whether the two are equal in quality, and requires it to provide a detailed explanation of the reasons for its judgment. The StreamChat model of this invention was used as one of the candidate models, and the overall win rate was calculated compared with other models.

[0066] Quantitative results

[0067] exist Figure 5 The comparison of StreamChat with other video LLMs in streaming evaluation is shown in the figure. The frame rate (FPS) is set to 5. 32 frames are used for StreamChat and LLaVAVideo

[68] models, while 16 frames are used for VILA

[26] because using 32 frames would be out of its context. The results of this invention show that the StreamChat model of this invention performs well in streaming interaction capabilities compared to the LLaVA-Video and VILA models. It is noteworthy that the StreamChat-7B model of this invention produces the same or better answers in 77% of the evaluation cases compared to VILA-1.5-40B, despite using a much smaller LLM. While the LLaVA-Video models perform well in offline video understanding, StreamChat-7B outperforms them in streaming interaction scenarios, highlighting the importance of capturing video dynamics during streaming inference. In addition, the StreamChat-14B of this invention is observed to exhibit better overall performance than StreamChat-7B, indicating that scaling up the underlying LLM can also improve streaming interaction performance. The StreamChat-7B / -14B was used as one of the candidate models in the evaluation, and the win / loss ratios against the VILA or LLaVA-Video models were reported. Figure 5 As shown, the StreamChat model demonstrates stronger streaming media interaction capabilities, even outperforming the LLaVA-Video-72B, which uses a larger underlying LMM.

[0068] Qualitative results

[0069] This invention provides a qualitative assessment of StreamChat's capabilities in streaming video, such as... Figure 6 As shown. In the example, a question is asked at a specific timestamp. While previous methods only used the visual context when the question was asked to answer it, StreamChat can dynamically update its visual context and adjust its answer according to the streaming video. This demonstrates that StreamChat is better able to capture dynamic video content and provide more accurate answers. In contrast, VILA

[26] and LLaVA-Video

[68] struggle to maintain time alignment with the streaming video and exhibit actual errors (highlighted in red).

[0070] Benchmark results

[0071] The performance of the StreamChat model was evaluated on popular image [9,13,19,20,23,32,33,41,54,62] and video benchmarks [14,25,34,38,52,55,61,69] using the LMMEval library

[66] . Note that to maintain StreamChat's efficiency in streaming interactions, multiple visual encoders [40,46] or image tiling techniques [10,29] were not used, which may compromise performance in benchmarks requiring high-resolution input. StreamChat's performance on image benchmarks is shown in Table 1.

[0072] Table 1

[0073]

[0074] Table 1 shows a comparison of StreamChat with existing leading LMMs in image benchmarks. StreamChat achieved competitive performance on these benchmarks while using only 256 visual tokens.

[0075] StreamChat demonstrates superior results compared to Cambrian-1, which uses multiple visual encoders, and LLaVA-NeXT, which uses image tiling. Notably, the StreamChat-7B of this invention scores 48.1 on the MMMU benchmark, surpassing LLaVA-NeXT-8B and Cambrian-1-8B by 6.4 and 5.4 points respectively. Furthermore, StreamChat outperforms LLaVA-NeXT and Cambrian-1 in TextVQA, despite using significantly fewer visual tokens. Overall, StreamChat achieves competitive performance on image benchmarks while ensuring computational efficiency.

[0076] Table 2 shows StreamChat's performance on video benchmarks.

[0077] Table 2

[0078]

[0079] Table 2 shows a comparison of StreamChat with existing leading LMMs in video benchmarks. StreamChat achieved competitive performance on these benchmarks, even outperforming models using larger underlying LMMs. StreamChat's cross-attention-based architecture is efficient when processing large numbers of video frames.

[0080] The model of this invention achieved a score of 58.6 / 62.8 on the VideoMME benchmark

[14] , which is 10.7 / 11.5 points higher than VideoLLaMA2-7B. In addition, StreamChat-14B showed superior performance compared to VILA-40B and VideoLLaMA2-72B, which use a larger underlying LLM. Importantly, the model of this invention remains efficient even when processing more frames during inference because the cross-attention-based architecture of this invention alleviates the heavy computation of self-attention between frames.

[0081] Ablation Research

[0082] Table 3 presents the ablation study results of this invention, in which the architecture design and the proposed dense instruction dataset were ablated.

[0083] Table 3

[0084]

[0085] Table 3 presents the ablation study results. StreamEval represents the streaming evaluation of this invention, using the final solution of this invention (last row) as one of the candidate models, and reports the performance of other models relative to the final solution of this invention.

[0086] Performance was compared across four image benchmarks, four video benchmarks, and the streaming evaluation of this invention. In the streaming evaluation, the StreamChat solution of this invention (last line) was used as one of the candidate models, and the performance of other models relative to StreamChat was reported. Experimental results of this invention demonstrate that the introduced architectural enhancements lead to an overall performance improvement. Specifically, the StreamChat model outperformed the version without a Visual Feedforward Network (V-FFN) expert on 8 out of 9 benchmarks. Furthermore, the use of tanh gating was observed to help with rapid convergence in the early stages of training; however, it ultimately resulted in worse performance compared to linear gating. Linear gating improved performance on 6 out of 9 benchmarks. Additionally, training instability without reusing LLM parameters was observed, which also led to poor final performance. The final solution of this invention significantly outperformed the model without parameter reuse on all evaluated benchmarks. Compared to the model trained on dense instruction data, the final solution of this invention performed comparably on existing image and video benchmarks. However, in the streaming evaluation, training using the dense instruction dataset of this invention significantly enhanced interactivity. The final solution of this invention yielded the same or better results in 75% of the evaluated cases, indicating that simply relying on existing image or video instructions to adjust the dataset is insufficient for effective streaming interaction.

[0087] In summary, StreamChat is an innovative approach that enhances the real-time interaction capabilities of large multimodal models (LMMs) with streaming video content. StreamChat is built on a flexible and efficient cross-attention-based architecture with visual feedforward network (V-FFN) experts. Unlike existing models that typically rely on fixed video content at the time the question is posed to answer it, this work emphasizes dynamically updating the visual context during decoding, thereby significantly enhancing the interactive experience and reducing the latency inherent in highly dynamic environments. By continuously updating the visual context at each decoding step, StreamChat effectively captures the dynamic changes in streaming video content, achieving time-aligned responses. A dense instruction dataset is also introduced to facilitate the training of the streaming interaction model, along with a parallel 3D-RoPE mechanism for better arrangement of streaming video and text. Extensive evaluations on established image and video benchmarks, as well as a new streaming benchmark, demonstrate that StreamChat not only achieves competitive performance on existing benchmarks but also excels in streaming interaction scenarios.

[0088] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0089] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0090] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0091] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk drive or magnetic tape drive. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0092] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0093] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0094] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0095] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0097] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0098] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0099] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0100] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

[0101] References

[0102] [1]Josh Achiam,Steven Adler,Sandhini Agarwal,Lama Ahmad,Ilge Akkaya,Florencia Leoni Aleman,Diogo Almeida,Janko Altenschmidt,Sam Altman,ShyamalAnadkat,et al.Gpt-4technical report.arXiv preprint arXiv:2303.08774,2023.

[0103] [2]Jean-Baptiste Alayrac,Jeff Donahue,Pauline Luc,Antoine Miech,IainBarr,Yana Hasson,Karel Lenc,Arthur Mensch,Katherine Millican,MalcolmReynolds,et al.Flamingo:a visual language model for few-shotlearning.Advances in Neural Information Processing Systems,35:23716–23736,2022.

[0104] [3]Jean-Baptiste Alayrac,Jeff Donahue,Pauline Luc,Antoine Miech,IainBarr,Yana Hasson,Karel Lenc,Arthur Mensch,Katherine Millican,MalcolmReynolds,et al.Flamingo:a visual language model for few-shotlearning.Advances in neural information processing systems,35:23716–23736,2022.

[0105] [4] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXivpreprint arXiv:2309.16609,2023.

[0106] [5] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966,2023.

[0107] [6]Lucas Beyer, Andreas Steiner, Andre′Susano Pinto, AlexanderKolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, MichaelTschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm fortransfer.arXiv preprint arXiv:2407.07726, 2024.

[0108] [7] Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videolm-online: Online video large language model for streaming video. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 18407–18418, 2024.

[0109] [8] Kaibing Chen, Dong Shen, Hanwen Zhong, Huasong Zhong, Kui Xia, Di Xu, Wei Yuan, Yifei Hu, Bin Wen, Tianke Zhang, et al. Evlm: An efficient vision-language model for visual understanding. arXiv preprint arXiv:2407.14177,2024.

[0110] [9] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330,2024.

[0111]

[10] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024.

[0112]

[11] Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, GuanzhengChen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-LLM.arXiv preprint arXiv:2406.07476, 2024.

[12] Abhimanyu Dubey,Abhinav Jauhri,Abhinav Pandey,Abhishek Kadian,Ahmad Al-Dahle,Aiesha Letman,Akhil Mathur,AlanSchelten,Amy Yang,Angela Fan,et al.The llama 3 herd of models.arXiv preprintarXiv:2407.21783,2024.

[0113]

[13] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394,2023.

[14] Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal LLM in video analysis. arXiv preprint arXiv:2405.21075,2024.

[0114]

[15] Chaoyou Fu,Haojia Lin,Zuwei Long,Yunhang Shen,Meng Zhao,YifanZhang,Xiong Wang,Di Yin,Long Ma,Xiawu Zheng,et al.Vita:Towards open-source interactive omni multimodal llm.arXiv preprint arXiv:2408.05211,2024.

[0115]

[16] Yash Goyal,Tejas Khot,Douglas Summers-Stay,Dhruv Batra,and DeviParikh.Making the v in vqa matter:Elevating the role of image understandingin visual question answering.In Proceedings of the IEEE conference oncomputer vision and pattern recognition,pages 6904–6913,2017.

[0116]

[17] Kristen Grauman,Andrew Westbury,Eugene Byrne,Zachary Chavis,Antonino Furnari,Rohit Girdhar,Jackson Hamburger,Hao Jiang,Miao Liu,XingyuLiu,et al.Ego4d:Around the world in 3,000 hours of egocentric video.InProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition,pages 18995–19012,2022.

[0117]

[18] Danna Gurari,Qing Li,Abigale J Stangl,Anhong Guo,Chi Lin,KristenGrauman,Jiebo Luo,and Jeffrey P Bigham.Vizwiz grand challenge:Answeringvisual questions from blind people.In Proceedings of the IEEE conference oncomputer vision and pattern recognition,pages 3608–3617,2018.

[0118]

[19] Drew AHudson and Christopher D Manning.Gqa:Anew dataset for real-world visual reasoning and compositional question answering.In CVPR,2019.

[0119]

[20] Aniruddha Kembhavi,Mike Salvato,Eric Kolve,Minjoon Seo,HannanehHajishirzi,and Ali Farhadi.Adiagram is worth a dozen images.In ComputerVision–ECCV 2016:14th European Conference,Amsterdam,The Netherlands,October11–14,2016,Proceedings,Part IV 14,pages 235–251.Springer,2016.

[0120]

[21] Jacob Devlin,Ming-Wei Chang,Kenton Lee,and KristinaToutanova.BERT:Pre-training of deep bidirectional transformers for languageunderstanding.In Proceedings of NAACL-HLT,page 2.Minneapolis,Minnesota,2019.

[0121]

[22] Ranjay Krishna,Kenji Hata,Frederic Ren,Li Fei-Fei,and Juan CarlosNiebles.Dense-captioning events in videos.In Proceedings of the IEEEinternational conference on computer vision,pages 706–715,2017.

[0122]

[23] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal LLM with generative comprehension, 2023.

[0123]

[24] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyi Li. Llava-onevision: Easy visualtask transfer. arXiv preprint arXiv:2408.03326,2024.

[0124]

[25] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206, 2024.

[0125]

[26] Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 26689–26699, 2024.

[0126]

[27] Tsung-Yi Lin,Michael Maire,Serge J.Belongie,James Hays,PietroPerona,Deva Ramanan,Piotr Dollár,and C.Lawrence Zitnick.Microsoft coco:Commonobjects in context.In ECCV,2014.

[0127]

[28] Haotian Liu,Chunyuan Li,Yuheng Li,and Yong Jae Lee.Improvedbaselines with visual instruction tuning.arXiv preprint arXiv:2310.03744,2023.

[0128]

[29] Haotian Liu,Chunyuan Li,Yuheng Li,Bo Li,Yuanhan Zhang,Sheng Shen,and Yong Jae Lee.Llava-next:Improved reasoning,ocr,and world knowledge,2024.

[0129]

[30] Haotian Liu,Chunyuan Li,Qingyang Wu,and Yong Jae Lee.Visualinstruction tuning.Advances in neural information processing systems,36,2024.

[0130]

[31] Jihao Liu,Xin Huang,Jinliang Zheng,Boxiao Liu,Jia Wang,OsamuYoshie,Yu Liu,and Hongsheng Li.Mm-instruct:Generated visual instructions forlarge multimodal model alignment.arXiv preprint arXiv:2406.197236,20234.

[0131]

[32] Yuan Liu,Haodong Duan,Yuanhan Zhang,Bo Li,Songyang Zhang,WangboZhao,Yike Yuan,Jiaqi Wang,Conghui He,Ziwei Liu,et al.Mmbench:Is your multi-modal model an all-around player?arXiv preprint arXiv:2307.06281,2023.

[0132]

[33] Pan Lu,Swaroop Mishra,Tony Xia,Liang Qiu,Kai-Wei Chang,Song-ChunZhu,Oyvind Tafjord,Peter Clark,and Ashwin Kalyan.Learn to explain:Multimodalreasoning via thought chains for science question answering.In The 36thConference on Neural Information Processing Systems(NeurIPS),2022.

[0133]

[34] Karttikeya Mangalam,Raiymbek Akshulakov,and JitendraMalik.Egoschema:A diagnostic benchmark for very long-form video languageunderstanding.Advances in Neural Information Processing Systems,36,2024.

[0134]

[35] Ben Mann,N Ryder,M Subbiah,J Kaplan,P Dhariwal,A Neelakantan,PShyam,G Sastry,A Askell,S Agarwal,et al.Language models are few-shotlearners.arXiv preprint arXiv:2005.14165,2020.

[0135]

[36] Kenneth Marino,Mohammad Rastegari,Ali Farhadi,and RoozbehMottaghi.Ok-vqa:A visual question answering benchmark requiring external knowledge.In Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition,pages 3195–3204,2019.

[0136]

[37] OpenAI.Hello gpt-4o.2024.

[0137]

[38] Viorica Lucas Smaira, Ankush Gupta, Adria`RecasensContinent, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Matthew Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Anthony Miech, Alex Frechette, Hanna Klimczak, Raphael Coster, Jun-lin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero, Red Lady, Andrew Zisserman, and Carreira.Perception test:A diagnostic benchmark for multimodal video models.In Advances in Neural Information Processing Systems,2023.

[0138]

[39] Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,SharanNarang,Michael Matena,Yanqi Zhou,Wei Li,and Peter J Liu.Exploring the limitsof transfer learning with a unified text-to-text transformer.Journal ofmachine learning research,21(140):1–67,2020.

[40] Min Shi,Fuxiao Liu,ShihaoWang,Shijia Liao,Subhashree Radhakrishnan,De-An Huang,Hongxu Yin,Karan Sapra,Yaser Yacoob,Humphrey Shi,et al.Eagle:Exploring the design space formultimodal LLM with mixture of encoders.arXiv preprint arXiv:2408.15998,2024.

[0139]

[41] Amanpreet Singh,Vivek Natarajan,Meet Shah,Yu Jiang,Xinlei Chen,Dhruv Batra,Devi Parikh,and Marcus Rohrbach.Towards vqa models that canread.In Proceedings of the IEEE / CVF conference on computer vision and patternrecognition,pages 8317–8326,2019.

[42] Jianlin Su.Totary position embedding,2024.

[0140]

[43] Yansong Tang,Dajun Ding,Yongming Rao,Yu Zheng,Danyang Zhang,LiliZhao,Jiwen Lu,and Jie Zhou.Coin:Alarge-scale dataset for comprehensiveinstructional video analysis.In Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition,pages 1207–1216,2019.

[0141]

[44] Gemini Team,Rohan Anil,Sebastian Borgeaud,Yonghui Wu,Jean-Baptiste Alayrac,Jiahui Yu,Radu Soricut,Johan Schalkwyk,Andrew M Dai,AnjaHauth,et al.Gemini:a family of highly capable multimodal models.arXivpreprint arXiv:2312.11805,2023.

[0142]

[45] Qwen Team.Qwen2.5:Aparty of foundation models,2024.

[0143]

[46] Shengbang Tong,Ellis Brown,Penghao Wu,Sanghyun Woo,ManojMiddepogu,Sai Charitha Akula,Jihan Yang,Shusheng Yang,Adithya Iyer,XichenPan,et al.Cambrian-1:A fully open,vision-centric exploration of multimodalLLM.arXiv preprint arXiv:2406.16860,2024.

[0144]

[47] Hugo Touvron,Matthieu Cord,Alexandre Sablayrolles,GabrielSynnaeve,and HervéJégou.Going deeper with image transformers.In Proceedingsof the IEEE / CVF international conference on computer vision,pages 32–42,2021.

[0145]

[48] Hugo Touvron,Louis Martin,Kevin Stone,Peter Albert,AmjadAlmahairi,Yasmine Babaei,Nikolay Bashlykov,Soumya Batra,Prajjwal Bhargava,Shruti Bhosale,et al.Llama2:Open foundation and fine-tuned chat models.arXivpreprint arXiv:2307.09288,2023.

[0146]

[49] AVaswani.Attention is all you need.Advances in Neural InformationProcessing Systems,2017.

[0147]

[50] Peng Wang,Shuai Bai,Sinan Tan,Shijie Wang,Zhihao Fan,Jinze Bai,Keqin Chen,Xuejing Liu,Jialin Wang,Wenbin Ge,et al.Qwen2-vl:Enhancing vision-language model’s perception of the world at any resolution.arXiv preprintarXiv:2409.12191,2024.

[0148]

[51] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079,2023.

[0149]

[52] Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: Abenchmark for long-context interleaved video-language understanding, 2024.

[0150]

[53] Shiwei Wu, Joya Chen, Kevin Qinghong Lin, Qimeng Wang, Yan Gao, Qianli Xu, Tong Xu, Yao Hu, Enhong Chen, and Mike Zheng Shou. Videolm-mod: Efficient video-language streaming with mixture-of-depths vision computation. arXiv preprint arXiv:2408.16730,2024.

[0151]

[54] x.ai.Grok-1.5 vision preview.6

[0152]

[55] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Nextphase of question-answering to explaining temporal actions. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9777–9786, 2021.

[0153]

[56] Zhifei Xie and Changqiao Wu.Mini-omni2:Towards open-source gpt-4omodel with vision, speech and duplex.arXiv preprint arXiv:2410.11190,2024.

[0154]

[57] Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and JiashiFeng. Pllava: Parameter-free llava extension from images to videos for videodense captioning. arXiv preprint arXiv:2404.16994, 2024.

[0155]

[58] Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhi-jian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188,2024.

[0156]

[59] Dongjie Yang, Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. Vript: A video is worth thousands of words. arXiv preprint arXiv:2406.06040,2024.

[0157]

[60] Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408.04840,2024.

[0158]

[61] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: Adataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9127–9134, 2019.

[0159]

[62] Xiang Yue,Yuansheng Ni,Kai Zhang,Tianyu Zheng,Ruoqi Liu,Ge Zhang,Samuel Stevens,Dongfu Jiang,Weiming Ren,andYuxuan Sun.Mmmu:Amassive multi-discipline multi-modal understanding and reasoning benchmark for expertagi.In CVPR,2024.

[0160]

[63] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and LucasBeyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 11975–11986, 2023.

[0161]

[64] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprintarXiv:2306.02858, 2023.

[0162]

[65] Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv preprint arXiv:2406.08085,2024.

[0163]

[66] Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. LMM-eval: Reality check on the evaluation of large multimodal models, 2024.

[0164]

[67] Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852,2024.

[0165]

[68] Yuanhan Zhang,Jinming Wu,Wei Li,Bo Li,Zejun Ma,Ziwei Liu,andChunyuan Li.Video instruction tuning with synthetic data.arXiv preprintarXiv:2410.02713,2024.

[0166]

[69] Junjie Zhou,Yan Shu,Bo Zhao,Boya Wu,Shitao Xiao,Xi Yang,YongpingXiong,Bo Zhang,Tiejun Huang,and Zheng Liu.Mlvu:Acomprehensive benchmark formulti-task long video understanding.arXiv preprint arXiv:2406.04264,2024.

[0167]

[70] Luowei Zhou,Nathan Louis,and Jason J Corso.Weakly-supervisedvideo object grounding from text by loss weighting and objectinteraction.arXiv preprint arXiv:1805.02834,2018.

[0168]

[71] Wanrong Zhu,Jack Hessel,Anas Awadalla,Samir Yitzhak Gadre,JesseDodge,Alex Fang,Youngjae Yu,Ludwig Schmidt,William Yang Wang,and YejinChoi.Multimodal c4:An open,billion-scale corpus of images interleaved withtext.Advances in Neural Information Processing Systems,36,2024.< / t> < / t> < / t> < / t> < / t> < / t>

Claims

1. A method for enhancing the interaction capability between large-scale multimodal models and streaming video content, characterized in that, include: S1. By utilizing the cross-attention mechanism, the input text is converted into an encoding form that the model can understand through a text encoder, and the video frames are converted into an encoding form that the model can understand through a visual encoder. This enables the model to dynamically integrate visual and textual information during the decoding process, thereby enhancing the model's understanding of dynamic content. S2. Corresponding to the three dimensions of time, height, and width in the video, by performing time encoding, height encoding, and width encoding on each visual and text token in parallel, information of time dimension, spatial height dimension, and spatial width dimension is added to each token to encode the relative time position of the token in the video, thereby enhancing the continuity of streaming media interaction; S3. Training is performed using a dense instruction dataset containing multiple triples, where each triple includes a time interval, an instruction, and an answer, with each instruction-answer pair's words heuristically timestamped; an attention mask is applied to ensure temporal consistency of the model during training and inference, wherein the attention mask is generated based on timestamp information, so that each text token can only focus on video information before its corresponding timestamp, masking future information; wherein, since visual and text tokens are organized in parallel, visual and text tokens at the same timestamp share the same temporal context in RoPE to enhance continuity during streaming interaction.

2. The method as described in claim 1, characterized in that, In step S1, the integration of visual and text tokens is achieved using a cross-attention mechanism, specifically including: Given an input streaming video, use a pre-trained visual model to extract the visual token for each sampled frame. Cross-attention blocks are inserted into the architecture of a large language model LLM, bridging visual and text tokens using a cross-attention mechanism, where text tokens serve as queries (Q) and visual tokens serve as keys (K) and values ​​(V). Dynamically update the visual token to reflect changes in the streaming video; The cross-attention block shares parameters with the self-attention block of LLM to improve the convergence speed of training.

3. The method as described in claim 2, characterized in that, In step S1, the visual representation is further enhanced by utilizing the Visual Feedforward Network (V-FFN) expert throughout the forward pass of the large-scale language model LLM, specifically including: During the forward pass of the LLM, the visual token is updated using a visual feedforward network (V-FFN) expert, and the updated visual token is then fed into the subsequent cross-attention block. The V-FFN expert allows the visual representation to be better aligned with the hidden state of the LLM, thereby improving the final performance. The text information is further processed using self-attention blocks of LLM and feedforward network FFN; Repeat the above process N times to achieve dynamic updates of the visual token during the forward pass of the model.

4. The method as described in claim 3, characterized in that, V-FFN experts initialize from the feedforward network FFN of LLM to inherit the pre-trained knowledge of LLM.

5. The method as described in claim 3 or 4, characterized in that, The outputs of cross-attention and V-FFN blocks are scaled using a linear gating mechanism to mitigate the vanishing gradient problem and stabilize the training process.

6. The method according to any one of claims 1 to 5, characterized in that, In step S2, parallel 3D-RoPE processing is implemented to simulate the location information of streaming video and text, specifically including: The RoPE embedding vector is divided into three dimensions, corresponding to time, height, and width information, respectively. For a text token, the values ​​of the three dimensions are the same, collectively identifying the token's position in the time series; For visual tokens, the three dimensions respectively identify the token's timestamp, vertical position, and horizontal position in the video frame; In this context, at any given timestamp, the corresponding text token and visual token share the same context information across three dimensions, achieving time alignment.

7. The method according to any one of claims 1 to 6, characterized in that, In step S3, a dense instruction dataset is prepared for model training, specifically including: Create a new video instruction adjustment dataset from an existing dense description dataset, which contains captions paired with timestamp intervals; A series of instruction-answer pairs are generated based on the subtitles of a video clip using a large language model (LLM).

8. The method as described in claim 7, characterized in that, In step S3, the construction process of the dense instruction dataset is further optimized, specifically including: Clustering methods are used to analyze the initially generated instruction-answer pairs to reduce the similarity between instructions and enhance the diversity of instruction data. Manually review and use the remaining examples as context examples for subsequent data generation; Examples are collected from multiple datasets representing different environments to cover a wide range of streaming interaction scenarios.

9. The method according to any one of claims 1 to 8, characterized in that, In step S3, the data arrangement process for model training includes: The coarse timestamp annotations of the initial instruction data are converted into fine-grained timestamp annotations by assigning a timestamp to each word in the instruction to simulate the time dynamics during streaming interaction; A heuristic approach is used to assign a timestamp to each word in the instruction data, enabling the model to simulate real-time interaction during the decoding process.

10. The method according to any one of claims 1 to 9, characterized in that, In step S3, the process of applying the attention mask includes: Use attention masks to prevent timestamps from being... <t>Token Follow Timestamp <t> Subsequent video frames are used to ensure the model's temporal consistency during training and inference.< / t> < / t> The attention masking mechanism allows the model to focus only on the relevant visual information available in each decoding step, thus maintaining the temporal integrity of streaming interactions.

11. The method according to any one of claims 1 to 10, characterized in that, In step S3, the model's reasoning process includes: A separate thread continuously reads the video stream and stores the extracted visual tokens in a first-in-first-out (FIFO) queue. When the LLM needs to decode to generate a response, it retrieves the latest visual token from the FIFO queue and integrates the current information into the decoding of the next token.

12. The method according to any one of claims 1 to 11, characterized in that, Step S3 includes the following pre-training steps before model training: Implement a two-stage pre-training process, gradually unfreezing the pre-training parameters to achieve more effective pre-training; In the two stages of pre-training, a combination of multiple data sources is used, including video description datasets and caption datasets; In the first phase, only the adapter is trained to achieve alignment between model components; In the second stage, the visual encoder and the visual feedforward network V-FFN expert are unfrozen and further trained to achieve deep alignment of the model. For the subtitle dataset, a lower frame rate was used for training, while for other video data, uniformly sampled frames were used for training. Through the pre-training process described above, a large number of samples are used to enhance the model's ability to understand streaming video content.

13. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for enhancing the interaction capabilities of large multimodal models with streaming video content as described in any one of claims 1 to 12.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for enhancing the interaction capabilities of large multimodal models with streaming video content as described in any one of claims 1 to 12.

Images