A multimodal large model video content understanding inference acceleration method and system

Abstract

The application discloses a multimodal large model video content understanding reasoning acceleration method and system, and mainly relates to the technical field of artificial intelligence reasoning acceleration. The method comprises the following steps: inputting video data and pre-processing the video data to generate a video frame sequence; performing adaptive video Token compression on the generated video frame sequence to output a compressed visual Token set; performing visual feature coding and Key-Value generation on the compressed visual Token set to obtain visual KV data; performing video KV cache partition management on the visual KV data; performing cross-modal reasoning based on a vLLM framework to generate a video content understanding result; and outputting the video content understanding result and performing post-processing and structured mapping. The application has the beneficial effect that it can realize significant reasoning acceleration and throughput improvement while maintaining the original large model accuracy.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence inference acceleration technology, specifically a method and system for accelerating inference of multimodal large model video content understanding based on vLLM. Background Technology

[0002] As multimodal large-scale models are increasingly used in scenarios such as video content understanding and intelligent analysis, multimodal models for video content understanding and inference typically need to process a large number of video frames and convert them into high-dimensional visual tokens for cross-modal inference with language models. Compared to image or text tasks, video tasks have longer sequence lengths, more feature redundancy, and more complex cross-modal computations, thus significantly increasing inference overhead and making it difficult to meet the needs of real-time or large-scale applications.

[0003] Currently, the more advanced large-model inference framework vLLM has achieved good acceleration results in text tasks through PagedAttention and efficient KVCache management, but its design is mainly geared towards text scenarios. When processing video multimodal models, the following problems still exist:

[0004] (1) The number of visual tokens generated by the video is huge, resulting in excessive KVCache usage;

[0005] (2) The mixed management of visual key-value pairs and textual key-value pairs makes it impossible to reuse visual features, resulting in duplicate calculations;

[0006] (3) Cross-modal attention has low efficiency in the case of long sequences;

[0007] (4) vLLM lacks acceleration mechanisms such as compression and pruning of video tokens.

[0008] Therefore, existing vLLMs cannot effectively utilize video memory in video scenarios, resulting in significant inference latency and insufficient throughput. There is an urgent need for an inference acceleration method that combines video characteristics and multimodal model structures to optimize vLLM algorithms and modify its underlying structure, thereby improving the computational efficiency of video multimodal tasks. Summary of the Invention

[0009] The purpose of this invention is to provide a method and system for accelerating inference of multimodal large model video content understanding based on vLLM, which can achieve significant inference acceleration and throughput improvement while maintaining the accuracy of the original large model.

[0010] To achieve the above objectives, the present invention employs the following technical solution:

[0011] On the one hand, this invention provides a method for accelerating multimodal large-model video content understanding and inference, comprising the following steps:

[0012] Step S1: Input video data and preprocess the video data to generate a video frame sequence;

[0013] Step S2: Perform adaptive video token compression on the video frame sequence generated in step S1, and output a compressed visual token set;

[0014] Step S3: Perform visual feature encoding and Key-Value generation on the compressed visual token set output in step S2 to obtain visual KV data;

[0015] Step S4: Perform video KV cache partition management on the visual KV data obtained in step S3, allocate independent storage areas for visual KV and set management strategies;

[0016] Step S5: Perform cross-modal inference based on the vLLM framework. The cross-modal inference includes, but is not limited to: generating text KV data and combining visual KV data with text KV data to generate video content understanding results.

[0017] Step S6: Output the video content understanding results and perform post-processing and structured mapping.

[0018] Preferably, in step S2, the adaptive video token compression includes the following steps:

[0019] Step S21: Extract features from each video frame to obtain the frame feature vector and PatchToken set;

[0020] Step S22: Calculate the semantic variation degree between adjacent frames, and evaluate it based on cosine distance or fused optical flow energy;

[0021] Step S23: Select keyframes and mark redundant frames based on the semantic variability and the preset threshold;

[0022] Step S24: Score the importance of each patch, taking into account feature strength, temporal variation, and attention saliency;

[0023] Step S25: Perform hierarchical temporal aggregation on similar patches in redundant frames to generate aggregated tokens;

[0024] Step S26: Dynamically allocate Token quotas based on the video content change rate to control the number of output Tokens;

[0025] Step S27: Prune and output the aggregated tokens, and associate them with metadata;

[0026] Step S28: Monitor the compression ratio and task accuracy through a quality control loop and dynamically adjust the compression parameters.

[0027] Preferably, in step S22, the semantic change degree is expressed as:

[0028] ;

[0029] in, , These are the global frame vectors for the current frame and the previous frame, respectively. Represents the dot product. Represents the vector norm;

[0030] The degree of change after merging optical flow energy is expressed as:

[0031] ;

[0032] in, These are weighting coefficients. This represents the normalized optical flow intensity.

[0033] Preferably, in step S24, the importance score for each patch is represented as follows:

[0034] ;

[0035] in, The L2 norm of the eigenvectors. Indicates within the time window The variance of the patch. Indicates saliency or attention proxy score, , , These are the learnable weight coefficients.

[0036] Preferably, step S26 includes:

[0037] Calculation time window average degree of change :

[0038] ;

[0039] Token allocation based on the degree of change :

[0040] ;

[0041] in, To minimize the number of tokens to retain, For the total token budget, The sensitivity coefficient, This is the floor function.

[0042] Preferably, step S3 includes:

[0043] Linear projection is performed on the compressed visual token to map it to the hidden layer dimension of the large language model, generating key and value vectors;

[0044] Construct the KVEntry data structure, including a unique hash identifier, key vector, value vector, video ID, segment ID, token index, and importance score;

[0045] Generate globally unique values ​​using a hash algorithm. :

[0046] ;

[0047] The generated KVEntry is transferred to the cache management module for storage and retrieval.

[0048] Preferably, in step S4, the video KV cache partition management includes:

[0049] Allocate a separate storage area for visual key-value pairs and set type labels for identification;

[0050] Perform persistence operations on visual key-value segments, maintain reference counts, and support reuse across requests;

[0051] Load or release visual key-value segments on demand, triggering management operations based on reference count and memory pressure;

[0052] When reference counting And the memory pressure exceeds the threshold When this occurs, a release or spill operation is triggered:

[0053] ;

[0054] Long videos are segmented for efficient inference with high video memory.

[0055] Preferably, in step S5, the cross-modal reasoning includes:

[0056] Generate text key-value pairs, which are dynamically generated from user-input text through a language model;

[0057] Attention is calculated by combining the visual key-value pairs with the text key-value pairs, and the pagedAttention mechanism of vLLM is used to load visual key-value blocks on demand.

[0058] Perform block-based attention calculation by dividing the visual key-value pairs into blocks and performing attention calculation for each block.

[0059] Improve GPU utilization and inference throughput through pipeline scheduling and batch processing optimization.

[0060] Preferably, in step S6, the post-processing and structured mapping includes:

[0061] Collect the token stream output by vLLM and parse it into structured text results;

[0062] The metadata of visual tokens is used to perform time and space backtracking mapping to generate timestamps and region labels;

[0063] The output results are merged and deduplicated based on attention intensity and confidence scores;

[0064] Output the final video content understanding result and selectively retain or release the visual key-value cache.

[0065] On the other hand, the present invention also provides a multimodal large model video content understanding and inference acceleration system for implementing the multimodal large model video content understanding and inference acceleration method described above, comprising:

[0066] The input module is used to input video data and preprocess the video data;

[0067] The compression module is used to perform adaptive video token compression on video frame sequences.

[0068] The encoding module is used to encode visual features and generate key-value pairs from the compressed visual token set.

[0069] The cache management module is used for partitioning and scheduling visual key-value data.

[0070] The inference module is used to perform cross-modal inference based on the vLLM framework;

[0071] The output module is used to output and post-process the video content understanding results.

[0072] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0073] 1. Significantly overcomes the bottleneck of video memory, supporting ultra-long video inference: This invention breaks the limitation of video memory capacity on video length through adaptive video token compression and KV caching partition management mechanism. On the one hand, adaptive compression technology can significantly reduce the number of visual tokens, retaining only key semantic information; on the other hand, partitioned storage and on-demand loading strategies allow inactive video segments to be cached in main memory or disk, retaining only the segments needed for current inference in video memory. This makes it possible to process hours-long or even longer videos on a single GPU card, effectively solving the problem that traditional methods cannot handle long videos due to video memory overflow (OOM).

[0074] 2. Significantly Improved Inference Throughput and Response Speed: This invention employs a block-wise attention mechanism and pipelined scheduling strategy, significantly reducing inference latency. By breaking down large-scale attention computations into smaller blocks and pipelined execution on the GPU, data transmission latency is masked, avoiding idle computation cores. Simultaneously, intelligent decision-making based on a "load and recalculation" cost model ensures the system always selects the fastest data acquisition path. In scenarios such as meeting minutes and sports commentary, it can achieve extremely high token generation rates, providing users with a near real-time question-and-answer experience.

[0075] 3. Achieve efficient cross-request data reuse and reduce computing costs: This invention designs a cache management mechanism based on reference counting and persistence (Pin), realizing "encode once, reuse many times". When faced with multi-turn dialogues or multiple users asking questions about the same popular video, the system can directly locate and reuse existing visual key-value data in video memory or storage, without repeatedly calling the expensive visual encoder for feature extraction. This not only greatly reduces the server's computing load and power consumption, but also significantly shortens the first-letter generation time (TTFT).

[0076] 4. Optimizing Information Density, Balancing Compression Rate and Understanding Accuracy: This invention proposes a dynamic quota allocation strategy based on content change rate, overcoming the information loss problem caused by traditional uniform sampling. The system can intelligently identify high-dynamic segments in the video (such as PPT page turning and human actions) and allocate more token quotas, while simultaneously performing high-intensity compression on redundant static backgrounds. This mechanism ensures that while significantly reducing the amount of data, it still retains the detailed information crucial for inference, thereby maintaining a high level of video understanding accuracy with low GPU memory usage and reducing model "illusions".

[0077] 5. Enhanced System Stability and Multimodal Concurrency: This invention physically isolates the visual key-value (KV) region from the text KV region at the storage layer, preventing massive video data from crowding out the video memory space for text generation and avoiding performance fluctuations caused by video memory fragmentation. Simultaneously, combined with vLLM's Continuous Batching technology, the system can efficiently process mixed requests containing different modalities (plain text, short videos, and long videos) in parallel, improving the hardware utilization and concurrent processing capabilities of the entire inference cluster. Attached Figure Description

[0078] Figure 1 This is a flowchart of the method of the present invention;

[0079] Figure 2 This is a system structure diagram of the present invention. Detailed Implementation

[0080] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined in this application.

[0081] In this invention, terms such as "upper," "lower," "left," "right," "front," "back," "vertical," "horizontal," "side," and "bottom" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used only to facilitate the description of the structural relationships of the various components or elements of this invention and do not specifically refer to any component or element in this invention. They should not be construed as limiting the invention.

[0082] Example:

[0083] like Figure 1 As shown, this embodiment provides a method for accelerating multimodal large-model video content understanding and inference, including the following steps:

[0084] Step S1: Input video data and preprocess the video data to generate a video frame sequence;

[0085] Step S2: Perform adaptive video token compression on the video frame sequence generated in step S1, and output a compressed visual token set;

[0086] Step S3: Perform visual feature encoding and Key-Value generation on the compressed visual token set output in step S2 to obtain visual KV data;

[0087] Step S4: Perform video KV cache partition management on the visual KV data obtained in step S3, allocate independent storage areas for visual KV and set management strategies;

[0088] Step S5: Perform cross-modal inference based on the vLLM framework. The cross-modal inference includes, but is not limited to: generating text KV data and combining visual KV data with text KV data to generate video content understanding results.

[0089] Step S6: Output the video content understanding results and perform post-processing and structured mapping.

[0090] Below, this embodiment will use a conference video summary as an application scenario to explain the above steps in detail:

[0091] This application scenario involves both hardware and software environments, specifically:

[0092] Hardware environment: Single NVIDIA H100 (80GB HBM3) or dual NVIDIA A100 (80GB NVLink);

[0093] Software environment: A deeply customized vLLM inference engine based on adaptive compression and partitioned caching;

[0094] The input data is: a 20-minute meeting recording (1080p resolution, 30fps frame rate, approximately 36,000 frames in total), with typical scenarios including: PPT slide turning, speaker actions, and group discussions.

[0095] Step S1: Extract and demultiplex frames from the target video to generate a frame sequence.

[0096] The purpose of this step is to receive the video to be inferred and perform necessary preprocessing (frame segmentation, time synchronization, segmented metadata annotation) to provide basic units for subsequent adaptive compression; specifically, this includes:

[0097] Step S11: Data acquisition and preprocessing: Receive input V (real-time stream in this embodiment), and determine the video ID and time base (frame rate fps).

[0098] Step S12: Demultiplexing and framing: Generate frame sequence For real-time streaming, incremental processing can be performed according to time windows (1-second windows in this embodiment);

[0099] Step S13: Vector indexing and storage: Record metadata for each frame: timestamp ts_t, frame_indext, resolution, source_hash.

[0100] Step S2: Meeting-Adaptive Compression addresses the characteristics of meeting scenarios, such as numerous static backgrounds (whiteboards, walls) and prolonged monotonous video (slight movements of the speaker), by performing high-intensity compression. Specifically, this includes:

[0101] Step S21 Frame Feature Extraction: The system receives the input video stream and extracts features from each video frame. Feature extraction is performed by using the intermediate layer output of a pre-trained visual encoder or a lightweight encoder to extract the PatchToken set for the frame. and the frame vector after global pooling ;

[0102] The formula is expressed as:

[0103] ;

[0104] in, For time step index, This refers to the number of patches within a single frame. Indicates the first Frame number Feature vectors of each patch;

[0105] Step S22 Change Detection: To evaluate the semantic redundancy of adjacent frames, the system calculates the semantic distance between the current frame and the previous frame. ;

[0106] Basic calculation: Calculating the degree of change based on the cosine distance of the global frame vector:

[0107] ;

[0108] in Represents the dot product. Represents the vector norm;

[0109] In this embodiment, multimodal fusion is employed: to capture minute motion changes, optical flow energy is fused. The corrected degree of change is calculated as follows:

[0110] ;

[0111] in, To balance the weighting coefficients for semantic changes and motion changes, In this embodiment, the normalized optical flow intensity is... The preferred value is The rationale is as follows: In LLM video understanding tasks, semantic changes (such as scene switching and object appearance) are more important than simple pixel-level motion (such as camera shake and lighting flicker). Therefore, semantic distance (VisualEncoder feature cosine distance) is given a higher weight (0.7), while optical flow / pixel difference (0.3) is given as an auxiliary factor to prevent misjudgment caused by sudden changes in lighting.

[0112] The system calculates the semantic distance between adjacent frames. For video conferencing, set a high redundancy threshold. (High change is only considered when the PPT is switched or when there is significant movement of people).

[0113] Step S23 Keyframe Selection: Based on the calculated The system filters keyframe sets from a sliding window or the entire video clip. ;

[0114] like Then determine the first The frame is a keyframe, which retains its complete information;

[0115] like Then determine the first If the frame is a non-critical frame (redundant frame), it is marked as a "MergeCandidate" and enters the subsequent aggregation process;

[0116] This embodiment also provides an alternative: selecting the Top-K frames with the greatest variability within a sliding window as keyframes.

[0117] Step S24 Patch Importance Scoring: In order to compress the number of tokens while retaining key information, the system calculates an importance score for each patch. The scoring system integrates feature strength, temporal variation, and attention saliency. The scoring formula in this implementation introduces an L2 norm term based on feature activation theory to preserve high-texture detail, a variance term based on video temporal redundancy theory to capture dynamic events, and a semantic term based on an attention mechanism to respond to user intent. This linear combination of orthogonal features constitutes a complete measure of video token information density. The scoring formula is expressed as follows:

[0118] ;

[0119] in:

[0120] The L2 norm of the feature vector, derived from model pruning and interpretability, indicates that in deep neural networks (such as the Transformer), the magnitude (L2 Norm) of the feature vector is generally positively correlated with the information richness contained in that region. The L2 norm (magnitude) of a feature is used to measure the amount of information contained in a token. In the formula above, Represents activation intensity;

[0121] Derived from video compression coding (such as H.264 / HEVC) and background modeling, it represents a local time window. In this embodiment, the variance of the patch at that location is... It has four functions:

[0122] (1) Motion capture: This is the core difference between video and image. For example, when a person waves their hand, the patch features of the hand area will change drastically, with extremely high variance; although the background has clear texture, it remains still, with variance close to zero.

[0123] (2) Compression redundancy: This is the key to achieving a high compression rate. For static backgrounds in conference videos, no matter how good the texture is, because it does not change, this item scores 0, thereby reducing its overall weight and making it easier to aggregate.

[0124] (3) Theoretical basis: Temporal Redundancy: According to Shannon's information theory, the amount of information is related to uncertainty (entropy). In a video stream, if a pixel or feature at a certain position remains unchanged for a period of time, the amount of new information it contains is extremely low, which is "redundant information". Conversely, a large variance means that the area has changed (movement, change of lighting), which is the unique value of video compared to static images.

[0125] (4) In practical applications, it is not necessary to repeatedly store the static background, but the area where the motion occurs must be retained. Variance is the most statistically significant indicator for measuring this dynamic change.

[0126] It is an externally introduced or model-generated semantic guidance signal, representing the saliency or attention proxy score. In this embodiment, it is obtained by co-scoring with the user query. Its function is to introduce high-level semantic judgment and prevent the loss of key semantics due to "compression for the sake of compression".

[0127] , , These are the learnable weight coefficients for the three indicators, specifically:

[0128] In this embodiment, the parameter prediction subnetwork adopts a lightweight multilayer perceptron (MLP) structure, specifically comprising an input layer, a hidden layer, and an output layer, which transforms the global feature vector into three learnable weight coefficients. The specific steps are as follows:

[0129] 1. Feature compression and nonlinear transformation:

[0130] Input the global feature mean of the current video segment (dimension) First, it is mapped to the hidden layer space (dimension e.g., through the first fully connected layer (LinearLayer)). And apply the ReLU activation function, expressed as follows:

[0131] ;

[0132] 2. Generate the original logarithmic odds (Logits):

[0133] Hidden layer output After passing through the second fully connected layer, it is mapped to a 3-dimensional original output vector. :

[0134] ;

[0135] in, These correspond to the unnormalized values ​​of the three weight coefficients, respectively.

[0136] 3. Parameter Mapping:

[0137] To ensure that the output parameters meet the requirements of nonnegativity and numerical stability, the system... Perform activation function mapping, using the Softmax function on the vector. The processing and calculation formula is as follows:

[0138] ;

[0139] In this method, the sum of the three output parameters is always equal to 1, which simplifies the denominator calculation steps of the subsequent importance scoring formula;

[0140] 4. Parameter allocation:

[0141] Finally, the generated 3D vector is sliced ​​along its last dimension:

[0142] The 0th component of the vector is assigned to (Static feature weights);

[0143] The first component of the vector is assigned to (Time-series dynamic weights);

[0144] The second component of the vector is assigned to (Semantic attention weight).

[0145] In this embodiment, "face detection" and "text region detection" are introduced as... (Attention Proxy):

[0146] For PPT areas: retain high-resolution patches to ensure clear text;

[0147] For background areas (such as the ceiling): Extremely low, directly polymerized or pruned in step S25.

[0148] Step S25: Hierarchical temporal aggregation: For consecutive frame sequences marked as similar or redundant (defined in the window) (Internal), using importance scoring Weighted fusion is performed to aggregate similar patches from multiple time steps into a single semantic token. ;

[0149] The aggregation formula is expressed as:

[0150] ;

[0151] Among them, weight The importance score is obtained by softmax normalization, ensuring that frames with high information content dominate the aggregation, and is represented as:

[0152] ;

[0153] Through this step, the system will Redundant backgrounds (such as stationary walls) and slightly moving foregrounds within a frame are compressed into a very small number of representative tokens.

[0154] Step S26 Dynamic Quota: To address the issue of information loss in highly dynamic scenarios due to a fixed compression rate, this embodiment employs a dynamic quota strategy based on the content change rate. The system sets a total Token budget. (For the currently processed video segment or scene), and dynamically allocate this budget to each time window. :

[0155] 1. Segment variation calculation: First, calculate the window average degree of change :

[0156] ;

[0157] in, The inter-frame variation degree calculated in step S22, The number of frames within the window;

[0158] 2. Quota allocation formula: Calculate the specific token quota for that window based on the proportion of segment change in the overall change. :

[0159] ;

[0160] in, This indicates the minimum number of tokens to retain, preventing static images from being over-compressed and resulting in a completely blank screen. This represents the floor function; This represents the sum of the window changes in the current video segment (normalized denominator). This represents the sensitivity coefficient (Hyper-parameter), when... The algorithm tends to allocate more tokens to fragments that change drastically, when The time is a linear distribution;

[0161] 3. Execute Retention: After determining the quota Then, the system in the window Based on the aggregated score calculated in step S25, only the top-ranked scores are retained. Aggregated Tokens ;

[0162] PPT slide turning point: Allocating high quotas (e.g., 512tokens), ensure that chart details are captured;

[0163] The speaker speaks (the screen is mostly still): a low quota is allocated (e.g., 32 tokens), and only lip movements and gestures are retained;

[0164] Optimized comparison:

[0165] Unoptimized (traditional fixed sampling): Sampling at 1fps, 256 tokens per frame, total token count equals It consumes a huge amount of video memory, and the computational load of Attention explodes quadratically;

[0166] After optimization (using step S2 of this embodiment): 20 minutes of video is compressed into approximately 8,000 to 12,000 core VisualTokens, greatly improving information density and reducing computation by approximately 96%.

[0167] Step S27 Token Pruning and Output: Organize the filtered and aggregated tokens into the format of the final input model;

[0168] 1. Collection Construction: Merge the tokens retained in all windows to form a compressed visual token set. ;

[0169] 2. Metadata Association: In order for the Large Model (LLM) to understand the spatiotemporal context of the compressed token, each output token... All carry metadata tuples:

[0170] ;

[0171] in, Indicates the time span represented by the token (e.g., [start_time, end_time]); Indicates the spatial coordinates of the token within the original image; The aggregate weight or importance score of the token can be used as a supplementary input to PositionEmbedding.

[0172] Step S28 Quality Control Loop: To prevent over-compression from causing model "illusion" or incorrect responses, the system introduces a feedback adjustment mechanism:

[0173] 1. Indicator monitoring:

[0174] Calculate the compression ratio ,in To output the number of tokens, The total number of tokens in the original video frames; monitoring task-level performance metrics (such as the accuracy of video question answering, ConfidenceScore, or the generated Perplexity).

[0175] 2. Feedback Adjustment: If the task accuracy drops beyond a preset threshold (or the confidence level of the model output is too low), parameter adjustment logic is triggered:

[0176] Relax pruning restrictions: Increase total budget Or lower the keyframe determination threshold That is: when the inter-frame distance When this is the case, it is determined to be a keyframe, in this embodiment. It is based on the cosine distance of semantic feature vectors ( Defined by ), its value range is within between;

[0177] Reprocessing: The current video segment is reprocessed using new parameters, and steps S21 to S27 are repeated. The total number of tokens (usually compressed to 10%-20% of the original size) is reduced, thereby significantly reducing the memory usage of subsequent vLLM inference.

[0178] Step S3: Visual Feature Encoding and Key-Value Generation:

[0179] The system receives the compressed visual token set output in step S2. The data is transformed into key and value vectors aligned with the hidden layer dimensions of the Large Language Model (LLM) through linear projection, and then encapsulated into structured data with unique hash fingerprints for use by the attention mechanism of vLLM. The specific process is detailed below:

[0180] Step S31: Receive compressed token and metadata: The system receives the compressed visual token set from step S2. Each Token Not only includes feature vectors It also carries crucial metadata, source_meta, including:

[0181] The time segment corresponding to this token (used to answer the question of "when" it happened).

[0182] : The spatial coordinate index of the token in the original image (used to answer the "where" question);

[0183] Step S32: Visual-Language Projection Transformation: In order for visual features to interact with the text embedding of LLM, the feature vectors in the visual space need to be transformed. Mapped to the LLM's Key / Value space;

[0184] In this embodiment, the aggregated vector from step S2 can be directly reused. As input, it avoids repeatedly running the heavy VisualEncoderBackbone (such as ViT), thus significantly reducing computational latency;

[0185] Projection formula: using a pre-trained linear projection matrix and Perform the transformation:

[0186] ;

[0187] Dimension definition:

[0188] Visual feature dimensions (e.g., CLIP's 1024 dimensions).

[0189] Learnable projection weight matrix;

[0190] The hidden layer dimension of a large language model (e.g., 4096 dimensions in Llama-3).

[0191] Generated here and It will remain fixed during the inference phase as part of the Prompt (PrefixKV);

[0192] Step S33: Construct the KVEntry data structure: In order to support efficient cache lookup and reuse in the subsequent (S4) steps, the system constructs a standardized data object KVEntry for each generated KV pair;

[0193] The data structure is defined as follows:

[0194] struct KVEntry{string kv_id; / / Unique hash identifier Vector key; / / Projected key vector Vector value; / / Projected value vector string video_id; / / Video source ID string segment_id; / / Video segment ID int token_idx; / / Position index in the sequence floatscore; / / Importance score calculated in S2};

[0195] Unique hash fingerprint generation: Generating globally unique fingerprints using a hash algorithm. Used for fast location in distributed cache:

[0196] ;

[0197] This ID ensures that even when processing requests from different users, the system can identify and reuse data as long as the same video clip and the same token location are involved.

[0198] Step S34 Cache Management Interaction: The generated KVEntry object is transferred to step S4 (Video KVCache Partition Management).

[0199] Write / query logic: The system first utilizes Send a query to S4's BlockManager;

[0200] If a hit occurs: discard the current calculation result and return the cached physical address pointer;

[0201] If a memory miss occurs: request a new memory block. and Write to VideoKVRegion and update the page table.

[0202] Step S35 Batch Processing Optimization: When handling high-concurrency requests, the system performs batch processing on the projection calculation in step S32, combining multiple requests from different video segments or different requests. The data is concatenated into a large tensor (BatchTensor), and matrix multiplication is performed in one go using the parallel computing power of the GPU. and The computation maximizes GPU throughput.

[0203] Optimized comparison:

[0204] Unoptimized: After waiting 20 minutes for the video to download completely → decoding all frames → batch encoding, users have to wait tens of seconds before they can start asking questions;

[0205] After optimization (using step S3 in this embodiment): encoding and transmission overlap, and when the video transmission ends, 95% of the VisualKV is ready, achieving "instant opening".

[0206] Step S4: Full Memory Resident and Pin Strategy:

[0207] Since the 20-minute video is compressed to only about 10kTokens after step S2, the H100's 80GB of video memory is more than enough (10kTokens of KVCache only takes up about 2-4GB):

[0208] Implementation details:

[0209] Step S41: The KVEntry written in step S3 will include tags such as type=video and segment_id to distinguish the data type and the segment to which it belongs;

[0210] Step S42 Partitioning: Set the VideoKVRegion to Non-Evictable mode;

[0211] Step S43Pin: Perform the pin() operation on all segments of this session, using reference counting. locking;

[0212] Step S44 Load Decision: The decision formula is always determined as Hit, saving all PCIe data transfer overhead;

[0213] Step S45: Release / Overflow on Demand: When And mem_pressure (video memory pressure) is greater than (Threshold)

[0214] By policy, evict or compress (page-level spillover to main memory / disk):

[0215] ;

[0216] The aforementioned memory pressure threshold That is, when the GPU memory usage exceeds Furthermore, when the reference count is 0, KVBlock's eviction / swap is triggered. In this embodiment, its value ranges from 0.7 to 0.95, with a preferred value of 0.85, based on the following: 1) Fragmentation reservation: Although vLLM uses PagedAttention to reduce fragmentation, the memory allocator still needs to reserve about 15% of buffer space to cope with sudden KernelWorkspace requests or large activation peaks; 2) Performance balance: If set to 95%, it is very easy to trigger OOM (OutOfMemory) during peak concurrency periods, causing service crashes; if set to 60%, the memory utilization is too low, wasting expensive H100 resources. 85% is the industry-recognized "safe waterline".

[0217] Step S46 Segment Management: The long video is cut into several SegmentKVBlocks, which can be loaded / released individually, enabling memory-friendly inference for ultra-long videos (avoiding excessive memory usage at once).

[0218] Step S47 Interface and Atomic Operations: KVCache provides atomic operations: pin(segment), unpin(segment), load(segment), evict(segment), and query operation: is_loaded(segment);

[0219] Step S48 Consistency and Recovery: Persistent storage saves kv_id→storage_offsetmapping, supporting recovery from external storage after a crash;

[0220] Optimized comparison:

[0221] Unoptimized: If the number of tokens exceeds 300,000, insufficient video memory will require frequent Swap (exchange between video memory and RAM), resulting in a sharp drop in generation speed (down to less than 10 tokens / s).

[0222] After optimization (using step S4 of this embodiment): 100% of the data resides in the HBM3 high-speed video memory, maximizing bandwidth utilization.

[0223] Step S5: vLLM Cross-Modal Inference and Speed ​​Burst (Flash-Decoding):

[0224] Implementation details:

[0225] Step S51 Scheduling Preparation: The scheduler collects the requests for the current batch and determines the required VideoKVsegment / block list B_needed;

[0226] Step S52 Load blocks on demand: For B_needed, call the interface in step S4 to ensure that the required blocks have been loaded into the GPU (or trigger a load); (use the storage layer interface defined in S4 to realize the real-time preparation of data);

[0227] Step S53 Blocking Attention: Although the total number of tokens is reduced, in order to maximize speed, the 10k tokens are still divided into blocks with block_size=16;

[0228] Flash-Decoding kernel: On the H100, a vLLM-optimized Flash-Decoding kernel is used. When processing LongContext, this kernel loads different blocks of KVCache in parallel for Softmax calculation, which greatly alleviates the serial bottleneck of the Attention operator.

[0229] Step S54: Dynamic generation of text KV: Text input (user query, prompt) generates a corresponding textKV and writes it to the TextKVRegion (which can be recycled);

[0230] Step S55 Sparse Search: For a specific question (such as "What was discussed in the 5th minute of the meeting?"), the scheduler only activates the Block with the corresponding timestamp to participate in the calculation, further reducing the amount of computation to almost zero;

[0231] Step S56 Decoding and Merging: The vLLM scheduler merges multiple requests with the same decode step into a batch to share KV loading and kernel calls to improve GPU utilization.

[0232] Step S57 Intermediate caching and flow control: Use stream / queue control for large batches to avoid memory jitter caused by loading too many blocks at once;

[0233] Step S58 End and release the reference count: After the inference is completed, step S4 updates the reference count and releases or retains VideoKV according to the policy.

[0234] Optimized comparison:

[0235] Unoptimized: When processing long sequences of 300,000, Attention is Complexity: The Prefill stage is extremely slow, and the Decode stage is limited by the GPU memory bandwidth, making it difficult to exceed 15-20 tokens / s.

[0236] After optimization (using step S5 of this embodiment):

[0237] Prefill (first character generation): Since the input is only 10k tokens, H100 can complete the processing in less than 200ms;

[0238] Decode (Generation Speed): Based on the short sequence in step S2 + Flash-Decoding in S5, the generation speed is easily and stably maintained at 60-80 tokens / s, which fully satisfies the real-time reading experience.

[0239] S6: Generation of Structured Meeting Summaries:

[0240] Generate a video summary based on the meeting footage, subtitles, audio, and other content.

[0241] like Figure 2 As shown, this embodiment also provides a multimodal large-model video content understanding and inference acceleration system, including:

[0242] The input module is used to input video data and preprocess the video data;

[0243] The compression module is used to perform adaptive video token compression on video frame sequences.

[0244] The encoding module is used to encode visual features and generate key-value pairs from the compressed visual token set.

[0245] The cache management module is used for partitioning and scheduling visual key-value data.

[0246] The inference module is used to perform cross-modal inference based on the vLLM framework;

[0247] The output module is used to output and post-process the video content understanding results.

[0248] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A multi-modal large model video content understanding inference acceleration method, characterized in that, Includes the following steps: Step S1: Input video data and preprocess the video data to generate a video frame sequence; Step S2: Perform adaptive video token compression on the video frame sequence generated in step S1, and output a compressed visual token set; Step S3: Perform visual feature encoding and Key-Value generation on the compressed visual token set output in step S2 to obtain visual KV data; Step S4: Perform video KV cache partition management on the visual KV data obtained in step S3, allocate independent storage areas for visual KV and set management strategies; Step S5: Perform cross-modal reasoning based on the vLLM framework. The cross-modal reasoning includes: generating textual key-value data and combining visual key-value data with textual key-value data to generate video content understanding results. Step S6: Output the video content understanding results, and perform post-processing and structured mapping; In step S2, the adaptive video token compression includes the following steps: Step S21: Extract features from each video frame to obtain the frame feature vector and PatchToken set; Step S22: Calculate the semantic variation degree between adjacent frames, and evaluate it based on cosine distance or fused optical flow energy; Step S23: Select keyframes and mark redundant frames based on the semantic variability and the preset threshold; Step S24: Score the importance of each patch, taking into account feature strength, temporal variation, and attention saliency; Step S25: Perform hierarchical temporal aggregation on similar patches in redundant frames to generate aggregated tokens; Step S26: Dynamically allocate Token quotas based on the video content change rate to control the number of output Tokens; Step S27: Prune and output the aggregated tokens, and associate them with metadata; Step S28: Monitor the compression ratio and task accuracy through a quality control loop, and dynamically adjust the compression parameters; In step S4, the video KV cache partition management includes: Allocate a separate storage area for visual key-value pairs and set type labels for identification; Perform persistence operations on visual key-value segments, maintain reference counts, and support reuse across requests; Load or release visual key-value segments on demand, triggering management operations based on reference count and memory pressure; When the reference count and the memory pressure exceeds a threshold a release or overwrite operation is triggered: ； Long videos are segmented for efficient inference with high video memory.

2. The multi-modal large model video content understanding inference acceleration method according to claim 1, characterized in that, In step S22, the semantic change degree is expressed as: ； wherein, , are global frame vectors of the current frame and the previous frame, respectively, denotes the dot product, denotes the vector norm; The degree of change after merging optical flow energy is expressed as: ； wherein, is a weighting factor, is the normalized light intensity.

3. The multi-modal large model video content understanding inference acceleration method according to claim 2, characterized in that, In step S24, the importance score for each patch is represented as follows: ； in, The L2 norm of the eigenvectors. Indicates within the time window The variance of the patch. Indicates saliency or attention proxy score, , , These are the learnable weight coefficients.

4. The multi-modal large model video content understanding inference acceleration method according to claim 2, characterized in that, Step S26 includes: Computing time windows of average variability : ； Allocating token quotas according to variability : ； wherein, is the minimum number of tokens reserved, is the total token budget, is the sensitivity coefficient, is the floor function.

5. The multi-modal large model video content understanding inference acceleration method according to claim 1, characterized in that, Step S3 includes: Linear projection is performed on the compressed visual token to map it to the hidden layer dimension of the large language model, generating key and value vectors; Construct the KVEntry data structure, including a unique hash identifier, key vector, value vector, video ID, segment ID, token index, and importance score; A globally unique identifier is generated by a hash algorithm : ； The generated KVEntry is transferred to the cache management module for storage and retrieval.

6. The multi-modal large model video content understanding inference acceleration method according to claim 1, characterized in that, In step S5, the cross-modal reasoning includes: Generate text key-value pairs, which are dynamically generated from user-input text through a language model; Attention is calculated by combining the visual key-value pairs with the text key-value pairs, and the pagedAttention mechanism of vLLM is used to load visual key-value blocks on demand. Perform block-based attention calculation by dividing the visual key-value pairs into blocks and performing attention calculation for each block. Improve GPU utilization and inference throughput through pipeline scheduling and batch processing optimization.

7. The multi-modal large model video content understanding inference acceleration method according to claim 1, characterized in that, In step S6, the post-processing and structured mapping includes: Collect the token stream output by vLLM and parse it into structured text results; The metadata of visual tokens is used to perform time and space backtracking mapping to generate timestamps and region labels; The output results are merged and deduplicated based on attention intensity and confidence scores; Output the final video content understanding result and selectively retain or release the visual key-value cache.

8. A multi-modal large model video content understanding inference acceleration system for implementing the multi-modal large model video content understanding inference acceleration method according to any one of claims 1-7, characterized in that, include: The input module is used to input video data and preprocess the video data; The compression module is used to perform adaptive video token compression on video frame sequences. The encoding module is used to encode visual features and generate key-value pairs from the compressed visual token set. The cache management module is used for partitioning and scheduling visual key-value data. The inference module is used to perform cross-modal inference based on the vLLM framework; The output module is used to output and post-process the video content understanding results.

Images