A saliency-guided multimodal large model video anomaly detection method

Abstract

This invention relates to the field of video anomaly detection technology and discloses a saliency-guided multimodal large-scale model video anomaly detection method, comprising: adaptive video quality assessment and enhancement of the video frame sequence to be detected; inputting the video frame sequence into a pre-trained video encoder; offline determining a target intermediate layer from multiple intermediate layers of the video encoder based on pre-calculated hierarchical saliency scores; calculating the temporal saliency function value of each video frame in the video based on the features output by the target intermediate layer; embedding the features of the keyframes and text prompts together into a multimodal large-scale language model for multimodal inference; and outputting the anomaly detection result and natural language explanation. This invention can achieve highly robust video anomaly detection with cross-scene generalization ability and interpretability.

Description

Technical Field

[0001] This invention relates to the field of video anomaly detection technology, and more specifically to a saliency-guided multimodal large-model video anomaly detection method. Background Technology

[0002] Video anomaly detection is a crucial research area in computer vision and artificial intelligence, with widespread applications in public safety monitoring, industrial quality inspection, and behavior analysis. With the rapid development of Large Language Models (LLMs), model capabilities have expanded from processing only text to supporting multimodal inputs such as visual, text, and audio (Multimedia LLMs, MLLMs). Leveraging large-scale pre-training and cross-modal alignment mechanisms, MLLMs demonstrate excellent understanding and reasoning abilities in tasks such as image and text retrieval, video description generation, and cross-modal reasoning. Simultaneously, video anomaly detection is evolving from the traditional model of "only outputting anomaly scores" towards "interpretability and high reliability." Users expect systems to not only identify anomalous events but also provide natural language explanations that align with human understanding, such as: "Personnel entered an unauthorized area and performed abnormal operations; their behavior is inconsistent with normal patterns and is therefore deemed anomaly." This explanatory capability is invaluable for traceability, credibility, and decision support in practical applications.

[0003] Currently, video anomaly detection methods based on multimodal large models are gradually becoming a research hotspot. These methods generally combine pre-trained visual encoders and large language models to achieve a unity of video content understanding and language generation, thus alleviating the "black box" problem of traditional deep models to some extent. However, existing technologies still face the following prominent challenges in real-world applications: 1) Unstable video quality leads to insufficient detection robustness. Actual surveillance videos are often affected by factors such as low illumination, motion blur, occlusion, and compression noise, resulting in unstable image quality. Visual encoders struggle to extract reliable features from low-quality videos, making it difficult to accurately identify abnormal behavior, leading to significant false positives and false negatives, and overall low robustness; 2) Video feature encoding lacks selectivity and exhibits significant redundancy. On the one hand, most methods directly use the features from the last layer of the visual encoder for inference, ignoring the local expressions of potentially more prominent abnormal patterns in intermediate layers, resulting in insufficient feature utilization and significant redundancy; on the other hand, videos are usually sampled uniformly across all frames at a fixed frequency or input as a whole into complex temporal models without distinguishing between key frames and irrelevant frames. Given the high cost of inference for large multimodal models, indiscriminately introducing all feature layers and all frames will dilute abnormal signals and significantly increase the computational burden, making it unsuitable for real-time processing and resource-constrained scenarios. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a saliency-guided multimodal large-scale video anomaly detection method.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A saliency-guided multimodal large-model video anomaly detection method includes: Adaptive video quality assessment and enhancement are performed on the video frame sequence to be detected. This includes inputting the video frame sequence and preset text prompts into a multimodal agent. The multimodal agent decides whether to call a video enhancement tool and to call a specific type of video enhancement tool to process the video frames based on the perception results, so as to obtain an enhanced video frame sequence or retain the original video frame sequence. A sequence of video frames is input into a pre-trained video encoder, and a target intermediate layer is determined offline from multiple intermediate layers of the video encoder based on a pre-computed hierarchical saliency score; the hierarchical saliency score is calculated by combining the inter-class mean difference and intra-class dispersion difference between normal and abnormal samples in the intermediate layer. Based on the features output by the target intermediate layer, the temporal saliency function value of each video frame in the video is calculated. The temporal saliency function value integrates the first-order semantic change, the second-order semantic acceleration, and the global semantic deviation. Based on this value, the top-ranked keyframes are selected. The features of the keyframes and text prompts are embedded into a multimodal large language model for multimodal reasoning, and anomaly detection results and natural language explanations are output.

[0007] In one embodiment, the multimodal agent generates structured instructions based on the perception results of the video frame sequence. These structured instructions are used to invoke the corresponding video enhancement tools in a preset video enhancement tool library.

[0008] In one embodiment, the step of inputting the video frame sequence into the pre-trained video encoder specifically includes: Original video frame sequence or enhanced video frame sequence Input to a Transformer-based pre-trained video encoder The encoder is made by It consists of several sequentially stacked intermediate layers, each containing a self-attention operation and a feedforward transformation structure. Output features of the intermediate layer for: ; in, This represents a bullish self-attention strategy. Representative level normalization.

[0009] In one embodiment, the hierarchical significance score is calculated by combining the inter-class mean difference and intra-class dispersion difference between normal and abnormal samples in the intermediate layer, specifically including: A subset of videos is randomly selected from the training set to form the evaluation subset. And based on video-level tags The dataset is divided into a normal sample set and an abnormal sample set. For each video in the evaluation subset, the output features of the intermediate layer of the video encoder are extracted, and the inter-class mean difference and intra-class dispersion difference are calculated. Calculate the normal sample at the th The mean vector of the output features of each intermediate layer Abnormal samples in the first The mean vector of the output features of each intermediate layer Define the difference in mean between classes for: ; in, To output feature dimensions, express The value of the d-th dimension, express The value of the d-th dimension; Calculate the normal sample at the th The standard deviation vector of the output features of each intermediate layer Abnormal samples in the first The standard deviation vector of the output features of each intermediate layer Define intra-class dispersion differences for: ; right and By performing a weighted combination, we obtain the first... The initial discriminant scores of the intermediate layers : ; and These are weight parameters; The initial discrimination scores of all intermediate layers are normalized, and the mean is calculated. with standard deviation And obtain the hierarchical significance score. : ; in, To prevent the stability constant from having a denominator of zero.

[0010] In one embodiment, the offline determination of a target intermediate layer from multiple intermediate layers of the video encoder based on pre-computed hierarchical saliency scores specifically includes: Finally, the intermediate layer with the highest hierarchical significance score was selected as the target intermediate layer. The target intermediate layer obtained The determination is made offline before deployment and remains fixed during subsequent training and inference phases.

[0011] In one embodiment, the temporal saliency function value integrates first-order semantic change, second-order semantic acceleration, and global semantic deviation, specifically including: First-order semantic change ; Indicates the first Semantic feature representation of each video frame in the target intermediate layer; Denotes the F2 norm; Second-order semantic acceleration ; Global semantic deviation intermediate variables ; This represents the total number of video frames in the video frame sequence. Will , , Unified modeling as a time-series significance function : ; in, These are preset non-negative weight parameters.

[0012] In one embodiment, the step of embedding the features of the keyframe and the text prompt into a multimodal large language model for multimodal inference, and outputting anomaly detection results and natural language explanations, specifically includes: The multimodal large language model consists of a feature projection layer and a large language model; the features output from all video frames and selected keyframes after passing through the target intermediate layer are mapped to the input dimension of the multimodal large language model through a linear projection layer: ; in, Indicates feature splicing, This represents the video feature matrix composed of the features of keyframes. Represents the complete video feature matrix; and These are the learnable projective weights and bias vectors, respectively. Embedding the projected video, Indicates the length of the video keyframe sequence. Indicates the total length of the video frame sequence. Dimensions representing visual features The dimension representing the multimodal embedding space; The text prompts are concatenated and then input into a multimodal large language model, which is then trained and optimized using cross-entropy loss for autoregressive language modeling. ; in, Represents all learnable parameters of a multimodal large language model. Represents the t-th real token. This represents the historical output of the multimodal large language model when predicting the t-th token. It is the result of splicing the projected video embedding and the text prompt embedding. This represents the total length of the tokens generated by the multimodal large language model.

[0013] Compared with the prior art, the beneficial technical effects of the present invention are: 1. Robust Intelligent Perception: This invention constructs a video processing tool library and uses an intelligent agent combined with text prompts to automatically decide whether to perform enhancement operations such as denoising, deblurring, and image stabilization, so as to adaptively improve the quality of the input video and enhance the stability and robustness of visual feature extraction from the source.

[0014] 2. Saliency-driven efficient focusing: This invention uses a saliency layer selection mechanism to screen the intermediate layer features with the strongest anomaly representation capabilities, and combines it with a saliency frame selection mechanism to filter irrelevant frames, achieving dual saliency focusing in space and time. This reduces feature redundancy and computational burden, while improving the sensitivity and efficiency of anomaly identification.

[0015] 3. Multimodal reasoning and interpretable output: Based on the key information after saliency screening, combined with text prompts to guide the large language model to perform multimodal interactive reasoning, it can simultaneously achieve high-precision anomaly detection and natural language interpretation generation, significantly improving the understandability, credibility and practical application value of the results.

[0016] This invention utilizes a multimodal large model to adaptively judge video quality based on text prompts and selects video enhancement tools and algorithms such as denoising, deblurring, image stabilization, and super-resolution to reduce the impact of noise, jitter, and low light on subsequent feature extraction. By performing various statistical metric analyses on the features of each layer of the pre-trained video encoder, it automatically selects the output of the most sensitive salient layer as a unified feature, effectively reducing feature redundancy and computational overhead while highlighting anomalous signals. In the time dimension, it combines information entropy frame screening and inter-frame difference measurement to adaptively select key change frames, compressing the number of input frames and increasing the information density of anomalous candidate frames. Based on this, it uses a large language model for multimodal reasoning, realizing both anomaly detection and localization, and generating natural language explanations, thereby constructing a video anomaly detection system with high robustness, cross-scene generalization ability, and interpretability. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a diagram of the overall architecture of the present invention. Detailed Implementation

[0018] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0019] This invention provides a saliency-guided multimodal large-model video anomaly detection method, comprising the following steps: S1, Adaptive video quality assessment and enhancement of the video frame sequence to be detected, including inputting the video frame sequence and preset text prompts into a multimodal agent. The multimodal agent decides whether to call a video enhancement tool and call a specific type of video enhancement tool to process the video frames based on the perception results, so as to obtain an enhanced video frame sequence or retain the original video frame sequence. S2, input the video frame sequence into the pre-trained video encoder, and offline determine a target intermediate layer from multiple intermediate layers of the video encoder based on the pre-computed hierarchical saliency score; the hierarchical saliency score is calculated by combining the inter-class mean difference and intra-class dispersion difference between normal samples and abnormal samples in the intermediate layer; S3. Based on the features output by the target intermediate layer, calculate the temporal saliency function value of each video frame in the video. The temporal saliency function value integrates the first-order semantic change, the second-order semantic acceleration, and the global semantic deviation. Based on this value, select the top-ranked keyframes. S4. The features of the keyframes and the text prompts are embedded into a multimodal large language model for multimodal reasoning, and the anomaly detection results and natural language explanations are output.

[0020] Video anomaly detection aims to automate the analysis of continuous video frame sequences, construct spatiotemporal pattern representations of normal scenes and behaviors, and identify events, targets, or behaviors that significantly deviate from the normal patterns. Anomalies typically refer to situations that exhibit significant differences from normal samples in terms of appearance features, motion patterns, spatiotemporal relationships, or semantic behavior, and can output corresponding anomaly markers and their temporal locations.

[0021] The method architecture proposed in this invention is as follows: Figure 2 As shown, it includes the following three parts: (1) video enhancement module, (2) salient layer selection module, and (3) salient frame selection module.

[0022] 1. Video enhancement module.

[0023] This module aims to perform adaptive quality diagnosis and enhancement on input video frame sequences to improve the accuracy and robustness of subsequent saliency analysis and anomaly detection. First, any video to be processed is obtained from the training dataset (such as XD-Violence), and it is uniformly sampled along the time dimension to obtain the video frame sequence. : ; Indicates the sampled first One video frame, This represents the number of sampling frames.

[0024] Then, text prompts for video quality assessment are constructed. This prompt guides the multimodal agent in making decisions regarding video quality assessment and tool selection. (Text prompt) Construction as follows: Analyze the overall visual quality of the video to determine if the following problems exist: (1) low illumination; (2) motion blur; (3) noise interference; (4) jitter; (5) insufficient sharpness; If the video quality is good, please answer: 'NO_ENHANCE'; If enhancement is required, please answer 'ENHANCE' and provide the recommended enhancement tool and parameters in JSON format.

[0025] video frame sequence With text prompts The inputs are shared to the multimodal agent, which includes a multimodal perception unit, a decision-making and reasoning unit, and a tool scheduling unit. The multimodal perception unit is used to extract visual features of video frames and parse the semantics of text prompts. The decision-making and reasoning unit generates tool usage decisions based on the visual features and text prompts. The tool scheduling unit outputs structured instructions to call the corresponding video enhancement algorithm according to the tool usage decisions.

[0026] If the video quality is good, the agent outputs "NO_ENHANCE" and directly updates the video frame sequence. The input is the subsequent saliency layer selection module. If enhancement is required, a JSON structured instruction containing the tool type and parameters is output, such as {'tool': 'enhance_brightness', 'params': {'gamma': 1.5}}. The JSON structured instruction is processed by the instruction parsing unit. By parsing the `tool` and `params` fields, the corresponding video enhancement method and its functional interface are looked up in the preset tool mapping table, and the specific enhancement algorithm is called according to the parameter configuration in `params` (such as gamma value, magnification, model weight path, etc.). Performing enhancement processing on the input video frame sequence can be represented as: ; in, It consists of several enhancement algorithms (such as denoising, image stabilization, deblurring, super-resolution, etc.); This is the enhanced video frame sequence.

[0027] 2. Significant layer selection module.

[0028] This module is used to offline identify the target intermediate layer with the strongest anomaly detection capability in the multi-layer structure of a video encoder, in order to avoid redundant computation caused by using features from all layers in the subsequent inference stage.

[0029] Original video frame sequence or enhanced video frame sequence Input to a Transformer-based pre-trained video encoder In a preferred embodiment, the video encoder can be VideoMAE or TimeSformer. The video encoder is composed of... It consists of several sequentially stacked intermediate layers, each containing a self-attention operation and a feedforward transformation structure. The output features of each intermediate layer can be denoted as: ; in, This represents a bullish self-attention strategy. Representative level normalization.

[0030] To evaluate the sensitivity of different intermediate layers to anomalous events, a subset of videos was randomly selected from the training set to form the evaluation subset. The samples were then divided into normal and abnormal sets based on video-level labels. For each video in the evaluation subset, intermediate layer features were extracted at each intermediate layer, and the following hierarchical significance indices were calculated.

[0031] (1) Inter-class mean difference: The mean difference between normal samples and abnormal samples was calculated separately on the 1st... The feature mean vector of the layer The difference in the mean of each level is defined as: ; in This is the feature dimension. This metric reflects the degree of shift between normal and abnormal videos in the overall feature representation of this layer.

[0032] (2) Intra-class dispersion difference: Calculate the dispersion difference between normal samples and abnormal samples on the 1st... The feature standard deviation vector of the layer The intra-class dispersion difference is defined as: ; This indicator is used to reflect the changes in the volatility of the characteristic distribution of abnormal samples.

[0033] To avoid instability from a single indicator, the two indicators mentioned above are weighted and combined to obtain the result. Initial discrimination scores for each intermediate layer: ; To eliminate dimensional differences between different layers, the discrimination scores for all layers were... Perform normalization. Calculate its mean. with standard deviation And obtain the normalized hierarchical significance score: ; in To prevent the stability constant from having a denominator of zero.

[0034] Finally, the intermediate layer with the highest significance score was selected as the target intermediate layer: ; The target intermediate layer obtained The features are determined offline before model deployment and remain fixed during subsequent training and inference phases. Only the output features of this layer are used for salient frame selection and anomaly inference.

[0035] 3. Significant frame selection module.

[0036] This module is designed to be based on the target intermediate layer. The output features are used to automatically filter out the key frames most relevant to the abnormal event from the video frame sequence, thereby reducing redundant information and improving the inference efficiency of downstream large models. The feature sequence of the target intermediate layer in the time dimension is as follows: ,in Indicates the first Semantic feature representation of frames at the target layer. This invention jointly models frame-level features from three complementary perspectives: semantic change intensity, semantic abruptness, and global semantic deviation, and constructs a frame-level semantic abruptness metric.

[0037] (1) First-order semantic change: This quantity reflects the continuous change of the semantic state of the video over time and can characterize the degree of semantic shift between adjacent frames.

[0038] .

[0039] (2) Second-order semantic acceleration: This quantity is used to characterize the abruptness and nonlinearity of semantic changes. It can effectively suppress the steady change segment and is highly sensitive to sudden events and abnormal behavior changes.

[0040] .

[0041] (3) Global semantic deviation: In order to distinguish between short-term fluctuations and abnormal semantic shifts, a video-level semantic center is introduced as a reference. This item measures the degree of deviation of the current frame from the overall video semantic distribution, which helps to identify frames with significant abnormal states: .

[0042] Will , , Unified modeling as a time-series significance function : ; in, These are preset non-negative weight parameters. Based on the significance score... Select the top scorers The video frames are used as a set of keyframes, and corresponding filter masks are constructed for subsequent anomaly inference.

[0043] 4. Input and optimization of large multimodal models.

[0044] First, the features output from all video frames and selected keyframes after passing through the target intermediate layer are mapped to the input dimension of the multimodal large language model through a linear projection layer: ; in, Represents the video feature matrix; and These are the learnable projection weights and bias vectors, respectively.

[0045] Video embedding after projection The text prompt ("Is there anything unusual in the video?") is concatenated and then input into a multimodal large language model, which is then trained and optimized using cross-entropy loss for autoregressive language modeling. ; in Represents all learnable parameters of the model. Represents the t-th real token. This represents the model's historical output when predicting the t-th token. This is the input information.

[0046] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0047] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0048] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0049] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0050] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A saliency-guided multimodal large-model video anomaly detection method, characterized in that, include: Adaptive video quality assessment and enhancement are performed on the video frame sequence to be detected. This includes inputting the video frame sequence and preset text prompts into a multimodal agent. The multimodal agent decides whether to call a video enhancement tool and to call a specific type of video enhancement tool to process the video frames based on the perception results, so as to obtain an enhanced video frame sequence or retain the original video frame sequence. A sequence of video frames is input into a pre-trained video encoder, and a target intermediate layer is determined offline from multiple intermediate layers of the video encoder based on a pre-computed hierarchical saliency score. The hierarchical significance score is calculated by combining the inter-class mean difference and intra-class dispersion difference between normal and abnormal samples in the intermediate layer; Based on the features output by the target intermediate layer, the temporal saliency function value of each video frame in the video is calculated. The temporal saliency function value integrates the first-order semantic change, the second-order semantic acceleration, and the global semantic deviation. Based on this value, the top-ranked keyframes are selected. The features of the keyframes and text prompts are embedded into a multimodal large language model for multimodal reasoning, and anomaly detection results and natural language explanations are output.

2. The saliency-guided multimodal large-scale video anomaly detection method according to claim 1, characterized in that, The multimodal agent generates structured instructions based on the perception results of the video frame sequence. These structured instructions are used to call the corresponding video enhancement tools in the preset video enhancement tool library.

3. The saliency-guided multimodal large-model video anomaly detection method according to claim 1, characterized in that, The step of inputting the video frame sequence into the pre-trained video encoder specifically includes: Original video frame sequence or enhanced video frame sequence Input to a Transformer-based pre-trained video encoder The encoder is made by It consists of several sequentially stacked intermediate layers, each containing a self-attention operation and a feedforward transformation structure. Output features of the intermediate layer for: ； in, This represents a bullish self-attention operation. Representative level normalization.

4. The saliency-guided multimodal large-model video anomaly detection method according to claim 1, characterized in that, The hierarchical significance score is calculated by combining the inter-class mean difference and intra-class dispersion difference between normal and abnormal samples in the intermediate layer, specifically including: A subset of videos is randomly selected from the training set to form the evaluation subset. And based on video-level tags The dataset is divided into a normal sample set and an abnormal sample set. For each video in the evaluation subset, the output features of the intermediate layer of the video encoder are extracted, and the inter-class mean difference and intra-class dispersion difference are calculated. Calculate the normal sample at the th The mean vector of the output features of each intermediate layer Abnormal samples in the first The mean vector of the output features of each intermediate layer Define the difference in mean between classes for: ； in, To output feature dimensions, express The value of the d-th dimension, express The value of the d-th dimension; Calculate the normal sample at the th The standard deviation vector of the output features of each intermediate layer Abnormal samples in the first The standard deviation vector of the output features of each intermediate layer Define intra-class dispersion differences for: ； right and By performing a weighted combination, we obtain the first... The initial discriminant score of the intermediate layer : ； and These are weight parameters; The initial discrimination scores of all intermediate layers are normalized, and the mean is calculated. with standard deviation And obtain the hierarchical significance score. : ； in, To prevent the stability constant from having a denominator of zero.

5. The saliency-guided multimodal large-scale video anomaly detection method according to claim 1, characterized in that, The offline determination of a target intermediate layer from multiple intermediate layers of the video encoder based on pre-calculated hierarchical saliency scores specifically includes: Finally, the intermediate layer with the highest hierarchical significance score was selected as the target intermediate layer. The target intermediate layer obtained The determination is made offline before deployment and remains fixed during subsequent training and inference phases.

6. The saliency-guided multimodal large-model video anomaly detection method according to claim 1, characterized in that, The temporal saliency function value integrates first-order semantic change, second-order semantic acceleration, and global semantic deviation, specifically including: First-order semantic change ; Indicates the first Semantic feature representation of each video frame in the target intermediate layer; Denotes the F2 norm; Second-order semantic acceleration ; Global semantic deviation intermediate variables ; This represents the total number of video frames in the video frame sequence. Will , , Unified modeling as a time-series significance function : ； in, These are preset non-negative weight parameters.

7. The saliency-guided multimodal large-scale video anomaly detection method according to claim 1, characterized in that, The step of embedding the features of the keyframes and text prompts into a multimodal large language model for multimodal inference, and outputting anomaly detection results and natural language explanations, specifically includes: The multimodal large language model consists of a feature projection layer and a large language model; the features output from all video frames and selected keyframes after passing through the target intermediate layer are mapped to the input dimension of the multimodal large language model through a linear projection layer: ； in, Indicates feature splicing, The video feature matrix represents the composition of features from keyframes. Represents the complete video feature matrix; and These are the learnable projective weights and bias vectors, respectively. Embedding the projected video, Indicates the length of the video keyframe sequence. Indicates the total length of the video frame sequence. Dimensions representing visual features The dimension representing the multimodal embedding space; The text prompts are concatenated and then input into a multimodal large language model, which is then trained and optimized using cross-entropy loss for autoregressive language modeling. ； in, Represents all learnable parameters of a multimodal large language model. Represents the t-th real token. This represents the historical output of the multimodal large language model when predicting the t-th token. It is the result of splicing the projected video embedding and the text prompt embedding. This represents the total length of the tokens generated by the multimodal large language model.

Images