Multi-mode large model video content understanding reasoning acceleration method and system

Abstract

The invention discloses a multi-mode large model video content understanding reasoning acceleration method and system, and mainly relates to the technical field of artificial intelligence reasoning acceleration. Comprising the following steps: inputting video data and preprocessing the video data to generate a video frame sequence; performing adaptive video Token compression on the generated video frame sequence, and outputting a compressed visual Token set; performing visual feature coding and Key-Value generation on the compressed visual Token set to obtain visual KV data; performing video KV cache partition management on the visual KV data; executing cross-modal reasoning based on the vLLM framework to generate a video content understanding result; and outputting a video content understanding result, and carrying out post-processing and structured mapping. The method has the beneficial effects that the obvious reasoning acceleration and throughput improvement can be realized on the premise of keeping the precision of the original large model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence reasoning acceleration technology, in particular to a multi-modal large model video content understanding reasoning acceleration method and system based on vLLM. BACKGROUND

[0002] With the increasing application of multi-modal large models in video content understanding and intelligent analysis scenarios, multi-modal models for video content understanding reasoning usually need to process a large number of video frames and convert them into high-dimensional visual tokens for cross-modal reasoning with language models. Compared with image or text tasks, video tasks have longer sequence length, more feature redundancy, and more complex cross-modal computation, so the reasoning overhead is significantly increased, making it difficult to meet the real-time or large-scale application requirements.

[0003] The current advanced large model reasoning framework vLLM has achieved good acceleration effect in text tasks through PagedAttention and efficient KVCache management, but its design is mainly for text scenarios. When processing video multi-modal models, the following problems still exist: (1) The number of visual tokens generated by video is huge, resulting in high KVCache occupancy; (2) Visual KV and text KV are managed together, which cannot reuse visual features and causes repeated computation; (3) Cross-modal Attention runs inefficiently in long sequences; (4) vLLM lacks compression, pruning and other acceleration mechanisms for video tokens.

[0004] Therefore, the existing vLLM cannot effectively utilize video memory in the video scenario, has large reasoning delay and insufficient throughput. There is an urgent need for a reasoning acceleration method that combines video characteristics and multi-modal model structure to optimize and modify the underlying algorithm of vLLM to improve the computational efficiency of video multi-modal tasks. SUMMARY

[0005] The purpose of the present application is to provide a multi-modal large model video content understanding reasoning acceleration method and system based on vLLM, which can achieve significant reasoning acceleration and throughput improvement while maintaining the original model accuracy.

[0006] To achieve the above purpose, the present application realizes the following technical solutions: On the one hand, the present application provides a multi-modal large model video content understanding reasoning acceleration method, comprising the following steps: Step S1: input video data and pre-process the video data to generate a video frame sequence; Step S2: performing adaptive video Token compression on the video frame sequence generated in step S1, outputting a compressed visual Token set; Step S3: performing visual feature coding and Key-Value generation on the compressed visual Token set output in step S2, obtaining visual KV data; Step S4: performing video KV cache partition management on the visual KV data obtained in step S3, allocating an independent storage area for the visual KV and setting a management strategy; Step S5: performing cross-modal inference based on a vLLM framework, the cross-modal inference including but not limited to: generating text KV data, and combining the visual KV data and the text KV data to generate a video content understanding result; Step S6: outputting the video content understanding result and performing post-processing and structured mapping.

[0007] Preferably, in step S2, the adaptive video Token compression includes the following steps: Step S21: performing feature extraction on each video frame to obtain a frame feature vector and a PatchToken set; Step S22: calculating a semantic change degree between adjacent frames, and evaluating based on a cosine distance or a fused optical flow energy; Step S23: selecting a key frame and marking a redundant frame according to the semantic change degree and a preset threshold; Step S24: performing importance scoring on each Patch, and comprehensively considering feature strength, temporal change and attention saliency; Step S25: performing hierarchical temporal aggregation on similar Patches in the redundant frame to generate an aggregated Token; Step S26: dynamically allocating a Token quota based on a video content change rate to control the number of output Tokens; Step S27: performing pruning and outputting on the aggregated Token, and associating metadata; Step S28: monitoring a compression ratio and a task accuracy through a quality control loop, and dynamically adjusting a compression parameter.

[0008] Preferably, in step S22, the semantic change degree is represented as: ; Wherein, , are global frame vectors of the current frame and the previous frame respectively, denotes a dot product, denotes a vector norm; The change degree after fusing the optical flow energy is represented as: ; wherein, is a weighting coefficient, is a normalized optical flow intensity.

[0009] Preferably, in step S24, the importance score of each Patch is represented as: ; wherein, represents the L2 norm of the feature vector, represents the variance of the Patch within the time window , represents the saliency or attention agent score, , , is a learnable weight coefficient.

[0010] Preferably, step S26 includes: calculating the average change degree of the time window : ; allocating Token quota according to the change degree: ; wherein, is the minimum reserved Token number, is the total Token budget, is the sensitivity coefficient, is the floor function.

[0011] Preferably, step S3 includes: performing linear projection on the compressed visual Token to map it to the hidden layer dimension of the large language model, generating Key and Value vectors; constructing a KVEntry data structure including a unique hash identifier, a Key vector, a Value vector, a video ID, a segment ID, a Token index, and an importance score; generating a globally unique through a hash algorithm: ; transmitting the generated KVEntry to a cache management module for storage and query.

[0012] Preferably, in step S4, the video KV cache partition management includes: allocating an independent storage area for visual KV and setting a type label for identification; performing a persistent operation on the visual KV segment, maintaining a reference count, and supporting cross-request reuse; Load or release visual key-value segments on demand, triggering management operations based on reference count and memory pressure; When reference counting And the memory pressure exceeds the threshold When this occurs, a release or spill operation is triggered: ; Long videos are segmented for efficient inference with high video memory.

[0013] Preferably, in step S5, the cross-modal reasoning includes: Generate text key-value pairs, which are dynamically generated from user-input text through a language model; Attention is calculated by combining the visual key-value pairs with the text key-value pairs, and the pagedAttention mechanism of vLLM is used to load visual key-value blocks on demand. Perform block-based attention calculation by dividing the visual key-value pairs into blocks and performing attention calculation for each block. Improve GPU utilization and inference throughput through pipeline scheduling and batch processing optimization.

[0014] Preferably, in step S6, the post-processing and structured mapping includes: Collect the token stream output by vLLM and parse it into structured text results; The metadata of visual tokens is used to perform time and space backtracking mapping to generate timestamps and region labels; The output results are merged and deduplicated based on attention intensity and confidence scores; Output the final video content understanding result and selectively retain or release the visual key-value cache.

[0015] On the other hand, the present invention also provides a multimodal large model video content understanding and inference acceleration system for implementing the multimodal large model video content understanding and inference acceleration method described above, comprising: The input module is used to input video data and preprocess the video data; The compression module is used to perform adaptive video token compression on video frame sequences. The encoding module is used to encode visual features and generate key-value pairs from the compressed visual token set. The cache management module is used for partitioning and scheduling visual key-value data. The inference module is used to perform cross-modal inference based on the vLLM framework; The output module is used to output and post-process the video content understanding results.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Breakthrough in video memory bottleneck, support for ultra-long video reasoning: The application breaks the limitation of video length by the memory capacity through adaptive video Token compression and partition management mechanism of KV cache. On the one hand, the adaptive compression technology can greatly reduce the number of visual Tokens, and only the key semantic information is retained; on the other hand, the partition storage and on-demand loading strategy allows the non-active video segments to be cached to the main memory or disk, and only the segments required for current reasoning are retained in the video memory. This makes it possible to process hours or even longer videos on a single GPU card, effectively solving the problem that traditional methods cannot process long videos due to out-of-memory (OOM).

[0017] 2. Greatly improve the reasoning throughput and response speed: The application adopts a block-wise attention mechanism and a pipeline scheduling strategy, which significantly reduces the reasoning delay. By decomposing large-scale attention calculation into small blocks and executing them in pipeline on GPU, the delay of data transmission is masked, and the idle time of computing cores is avoided. At the same time, based on the intelligent decision of the "load and recalculate" cost model, the system always selects the fastest data acquisition path. In the scenarios of meeting minutes, sports commentary, etc., it can achieve a very high Token generation rate, providing users with near real-time question and answer experience.

[0018] 3. Efficient cross-request data reuse, reduce computing power cost: The application designs a cache management mechanism based on reference counting and persistence (Pin), which realizes "one encoding, multiple reuse". When facing multiple dialogues or multiple users asking questions about the same popular video, the system can directly locate and reuse the existing visual KV data in the video memory or storage, without the need to repeatedly call the expensive visual encoder for feature extraction. This not only greatly reduces the server's computing load and power consumption, but also significantly shortens the first word generation time (TTFT).

[0019] 4. Optimize information density, balance compression ratio and understanding accuracy: The application proposes a dynamic quota allocation strategy based on content change rate, which overcomes the information loss problem caused by traditional uniform sampling. The system can intelligently identify high dynamic segments in the video (such as PPT flipping, character actions) and allocate more Token quotas, while compressing the redundant static background at a high intensity. This mechanism ensures that while significantly reducing the data volume, it still retains important details for reasoning, thereby maintaining a high level of video understanding accuracy under low memory occupancy, reducing model "illusion".

[0020] 5. Enhance the stability and multi-modal concurrency capability of the system: the application avoids the occupation of video data to the video memory space for text generation by physically isolating the visual KV area and the text KV area in the storage layer, preventing performance fluctuations caused by video memory fragmentation. At the same time, combined with the continuous batching technology of vLLM, the system can efficiently and parallelly process mixed requests containing different modalities (pure text, short video, long video), improving the hardware utilization and concurrency processing capability of the entire inference cluster. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a method flowchart of the application; Figure 2 is a system structure diagram of the application. DETAILED DESCRIPTION

[0022] The application will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the application and not to limit the scope of the application. In addition, it should be understood that those skilled in the art can make various modifications or changes to the application after reading the content taught by the application, and these equivalent forms also fall within the scope defined by the application.

[0023] In the application, terms such as "up", "down", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicate the orientation or positional relationship shown in the drawings, which is only a relationship word determined for the convenience of describing the structural relationship of the components or elements of the application, and cannot be understood as a limitation of the application.

[0024] Embodiment: As shown in Figure 1 , the embodiment provides a multi-modal large model video content understanding inference acceleration method, which includes the following steps: Step S1: input video data and pre-process the video data to generate a video frame sequence; Step S2: perform adaptive video Token compression on the video frame sequence generated in step S1 to output a compressed visual Token set; Step S3: perform visual feature coding and Key-Value generation on the compressed visual Token set output in step S2 to obtain visual KV data; Step S4: perform video KV cache partition management on the visual KV data obtained in step S3, allocate an independent storage area for the visual KV and set a management strategy; Step S5: Perform cross-modal inference based on the vLLM framework, which includes but is not limited to generating text KV data and combining visual KV data with text KV data to generate video content understanding results; Step S6: Output the video content understanding results and perform post-processing and structured mapping.

[0025] In the following, the present embodiment will take a conference video summary as an application scenario, and the above steps will be described in detail: There are hardware and software environments under this application scenario, specifically: Hardware environment: single-card NVIDIA H100 (80GB HBM3) or double-card NVIDIA A100 (80GB NVLink); Software environment: deeply customized vLLM inference engine based on adaptive compression and partition caching; The input data is a 20-minute conference video (resolution 1080p, frame rate 30fps, total frame number about 36000 frames), typical scenarios include: PPT flipping, speaker action, multi-person discussion.

[0026] Step S1: Frame extraction and demultiplexing of the target video to generate frame sequence: The purpose of this step is to receive the video to be inferred and do necessary preprocessing (frame division, time synchronization, segment metadata labeling), providing basic units for subsequent adaptive compression; specifically including: Step S11: Data acquisition and preprocessing: receive input V (in this embodiment, real-time stream), determine video id and time base (frame rate fps); Step S12: Demultiplexing and frame division: generate frame sequence For real-time streams, incremental processing can be performed in time windows (1 second window in this embodiment); Step S13: Vector indexing and storage: record metadata for each frame: timestamp ts_t, frame_index, resolution, source_hash.

[0027] Step S2: Adaptive conference video Token compression (Meeting-AdaptiveCompression) is performed for the characteristics of a large number of static backgrounds (whiteboard, wall) and long-time monotone pictures (speaker micro-motion) in conference scenarios, and high-intensity compression is performed, specifically including: Step S21: Frame feature extraction: the system receives the input video stream, and for each video frame Feature extraction: extract PatchToken set of the frame using pre-trained VisualEncoder intermediate layer output or LightweightEncoder and global pooled frame vector ; The formula is: ; where, is the time step index, is the number of patches within a single frame, represents the feature vector of the Patch in the frame; Step S22 change detection: to evaluate the semantic redundancy of adjacent frames, the system calculates the semantic distance between the current frame and the previous frame ; Basic calculation: calculate the change degree based on the cosine distance of the global frame vector: ; where represents the dot product, represents the vector norm; In this embodiment, multi-modal fusion is used: to capture minor motion changes, fuse optical flow energy The corrected change degree is calculated as follows: ; where, is the weighting coefficient to balance semantic change and motion change, is the normalized optical flow intensity, in this embodiment The preferred value of is , which is based on the fact that in LLM video understanding tasks, semantic changes (such as scene switching, object appearance) are more important than pure pixel-level motion (such as camera shaking, light flickering), therefore, give higher weight (0.7) to semantic distance (VisualEncoder feature cosine distance), and give optical flow / pixel difference (0.3) as an auxiliary to prevent misjudgment due to sudden changes in light; The system calculates the semantic distance of adjacent frames , for conference videos, set a high redundancy threshold (high change only when PPT switching or large movement of people); Step S23 key frame selection: based on the calculated , the system filters the key frame set on the sliding window or the entire video segment If , then the first frame is determined as a key frame (Keyframe) and its complete information is reserved; , then the first frame is determined as a key frame (Keyframe) and its complete information is reserved; , then the first frame is determined as a key frame (Keyframe) and its complete information is reserved; , then the first frame is determined as a key frame (Keyframe) and its complete information is reserved; , then the first frame is determined as a key frame (Keyframe) and its complete information is reserved; The embodiment also provides an alternative: selecting the Top-K frames with the largest change degree in the sliding window as the key frames.

[0028] Step S24: Patch importance score: in order to compress the Token quantity while reserving the key information, the system calculates the importance score of each Patch , which comprehensively considers the feature strength, temporal change and attention saliency; in the embodiment, the score formula introduces an L2 norm item based on the feature activation theory to reserve high texture details, introduces a variance item based on the video temporal redundancy theory to capture dynamic events, and introduces a semantic item in combination with the attention mechanism to respond to the user's intention, and the linear combination of such orthogonal features constitutes a complete measurement of the information density of the video Token, and the score formula is as follows: ; , wherein: is the L2 norm of the feature vector, which is derived from model pruning and explainability, and indicates that, in a deep neural network (such as a Transformer), the length of the feature vector (L2 norm) is usually positively correlated with the information richness contained in the region, and the L2 norm (length) of the feature is used to measure the amount of information contained in a Token, and in the above formula, represents the activation strength; is derived from video compression coding (Video Coding, such as H.264 / HEVC) and background modeling (Background Modeling), and indicates the variance of the position Patch in a local time window , and in the embodiment, has four effects: (1) capturing motion: this is the core of video distinguishing from images, for example, when a person waves his hand, the Patch features of the hand region will change dramatically, and the variance is extremely large; the background, although the texture is clear, remains static, and the variance is close to zero; (2) compressing redundancy: this is the key to realizing high compression rate, for the static background in the conference video, no matter how good the texture is, because it does not change, the score of this item is 0, thereby reducing its overall weight, so that it is easy to be aggregated; (3) Theoretical basis: Temporal Redundancy: According to Shannon's information theory, the amount of information is related to uncertainty (entropy). In a video stream, if the pixels or features at a certain position remain unchanged over a period of time, the amount of new information they contain is extremely low, which is "redundant information"; on the contrary, a large variance means that the region has changed (motion, change of light), which is the unique value of video compared with static pictures; (4) In practical applications, static backgrounds do not need to be stored repeatedly, but the regions where motion occurs must be preserved, and variance is the most statistically meaningful indicator of such dynamic changes; is an externally introduced or model-generated semantic guidance signal representing saliency or attention proxy scores (Saliency / Attention Proxy), which is obtained in this embodiment by collaborative scoring based on the user Query, and its role is to introduce high-level semantic judgment to prevent key semantics from being lost due to "compression for compression"; , , are the learnable weight coefficients of the three indicators, which are specifically: In this embodiment, the parameter prediction subnetwork adopts a lightweight multi-layer perceptron (MLP) structure, specifically including an input layer, a hidden layer, and an output layer, which converts the global feature vector into three learnable weight coefficients The specific steps are as follows: 1. Feature compression and nonlinear transformation: Input the global feature mean of the current video segment (dimension ), first map it to the hidden layer space (dimension, for example ) through the first full connection layer (Linear Layer), and apply the ReLU activation function, which is expressed as: ; 2. Generate original logit: The hidden layer output is mapped to a 3-dimensional original output vector through the second full connection layer: ; Among them, respectively correspond to the non-normalized values of the three weight coefficients; 3. Parameter mapping and activation (Parameter Mapping): In order to ensure that the output parameters meet the requirements of non-negativity and numerical stability, the system normalizes the original output vector The activation function mapping is performed, and the Softmax function is used for vector Processing is performed, and the calculation formula is as follows: ; In this way, the sum of the three output parameters is equal to 1, thereby simplifying the denominator calculation step of the subsequent importance score formula; 4. Parameter allocation: Finally, the generated 3-dimensional vector is sliced (Slicing) in the last dimension: The 0th component of the vector is assigned to (static feature weight); The 1st component of the vector is assigned to (time series dynamic weight); The 2nd component of the vector is assigned to (semantics attention weight).

[0029] In this embodiment, "face detection" and "text area detection" are introduced as (note attention agent): For PPT areas: high-resolution patches are retained to ensure clear text; For background areas (such as ceilings): Extremely low, directly aggregated or pruned in step S25.

[0030] Step S25 hierarchical temporal aggregation: for continuous frame sequences marked as similar or redundant (defined within the window ), the importance score is used for weighted fusion to aggregate similar patches at multiple time steps into a single semantic Token ; The aggregation formula is represented as: ; Where the weight is obtained by Softmax normalization of the importance score, ensuring that frames with large information quantities dominate in aggregation, represented as: ; Through this step, the system compresses the redundant background (such as stationary walls) and micro-motion foreground in the frame into a small number of representative Tokens.

[0031] Step S26 dynamic quota: to solve the problem that a fixed compression rate may cause loss of information in high dynamic scenes, this embodiment adopts a dynamic quota strategy based on content change rate, and the system sets a total Token budget (dynamically allocated to each time window) : 1. Segment variation calculation: First calculate the average variation of the window : : ; wherein is the inter-frame variation calculated in step S22, is the number of frames in the window; 2. Quota allocation formula: According to the proportion of segment variation in the overall variation, calculate the specific Token quota of the window : ; wherein represents the minimum reserved Token number to prevent still pictures from being excessively compressed to complete blank; represents the floor function; represents the sum of the variation of all windows in the current video segment (normalization denominator); represents the sensitivity coefficient (Hyper-parameter), when , the algorithm tends to allocate more Tokens to the segment with severe variation, when , it is linear allocation; 3. Perform reservation: After determining the quota , the system retains only the top-ranked (Top- ) aggregated Tokens in the window according to the aggregated score calculated in step S25; PPT page turning moment: allocate a high quota (such as 512 tokens) to ensure capturing chart details; Speaker speaking (picture basically still): allocate a low quota (such as 32 tokens) to retain only lip shape and gesture features; Optimized contrast: Unoptimized (traditional fixed sampling): sampled at 1 fps, 256 tokens per frame, total Token number equal to , huge memory occupation, Attention calculation amount explodes in square level; After optimization (using step S2 of the embodiment): a 20-minute video is compressed to about 8000 to 12000 core VisualTokens, the information density is greatly improved, and the calculation amount is reduced by about 96%.

[0032] Step S27 Token pruning and output: The filtered and aggregated Tokens are arranged into the format of the final input of the large model; 1. Set construction: merge all the retained Tokens in all windows to form a compressed visual Token set ; 2. Metadata association: in order to enable the large model (LLM) to understand the spatio-temporal context of the compressed Token, each output Token carries a Metadata Tuple: ; Among them, indicates the time span (e.g. [start_time, end_time]) represented by the Token; indicates the spatial position coordinates of the Token in the original picture; indicates the aggregation weight or importance score of the Token, which can be used as a supplementary input of PositionEmbedding.

[0033] Step S28 quality control loop: in order to prevent over-compression from causing model “hallucination” or incorrect answers, the system introduces a feedback regulation mechanism: 1. Index monitoring: Calculate the compression ratio , where is the number of output Tokens, is the total number of Tokens of the original video frame; monitor the task-level performance indicators (such as the accuracy of video question answering ConfidenceScore or the generated Perplexity).

[0034] 2. Feedback regulation: if the task accuracy decreases by more than a preset threshold (or the confidence of the model output is too low), trigger the parameter adjustment logic: Loosen pruning: increase the total budget , or lower the key frame determination threshold , that is, when the inter-frame distance , it is determined as a key frame, and in the present embodiment is defined based on the cosine distance of semantic feature vectors , which takes a value in the range of ; Reprocessing: use the new parameters to reprocess the current video segment using the processing procedures of steps S21 to S27, and the total number of Tokens (which can be compressed to 10%-20% of the original size), thereby greatly reducing the video memory occupation of subsequent vLLM inference.

[0035] Step S3: visual feature encoding and KV generation: The system receives the compressed visual Token set outputted in step S2 , transforms them into Key and Value vectors aligned with the dimensions of the Large Language Model (LLM) hidden layer through linear projection, and encapsulates them as structured data with a unique hash fingerprint for the attention mechanism of the vLLM. The specific process is as follows: Step S31 receives compressed Tokens and metadata: the system receives the compressed visual Token set from step S2 Each Token contains not only a feature vector but also carries key metadata source_meta, including: : the time segment corresponding to the Token (used to answer questions about "when" something happened).

[0036] : the spatial coordinate index of the Token in the original image (used to answer questions about "where"). Step S32 visual-language projection transformation: in order to enable visual features to interact with text Embedding of the LLM, it is necessary to map the feature vector in the visual space to the Key / Value space of the LLM. In this embodiment, the aggregated vector in step S2 can be directly reused as input, avoiding the need to run the heavy VisualEncoderBackbone (such as ViT) again, thereby significantly reducing the computational delay. Projection formula: use the pre-trained linear projection matrix and for transformation: ; Dimension definition: : visual feature dimension (e.g. 1024 dimensions of CLIP); : learnable projection weight matrix; : hidden layer dimension of the large language model (e.g. 4096 dimensions of Llama-3); The and generated here will be part of the Prompt (PrefixKV) and remain fixed during the inference stage. Step S33: Construct KVEntry data structure: To support efficient cache lookup and reuse in the following step (S4), the system constructs a standardized data object KVEntry for each pair of KV generated; The data structure is defined as follows: struct KVEntry { string kv_id; / / unique hash identifier Vector key; / / projected Key vector Vector value; / / projected Value vector string video_id; / / video source ID string segment_id; / / video segment ID int token_idx; / / position index in sequence float score; / / importance score calculated in S2}; Unique Hashing: Generate a globally unique by hash algorithm, used to quickly locate in distributed cache: ; This ID ensures that even when processing requests from different users, as long as the same video segment and the same Token position are involved, the system can identify and reuse the data; Step S34: Cache management interaction: The generated KVEntry object is transmitted to step S4 (Video KVCache partition management); Write / query logic: The system first uses to query the BlockManager of S4; If hit: Discard the current calculation result directly and return the cached physical address pointer; If miss: Apply for a new video memory Block, write and to the VideoKVRegion, and update the page table.

[0037] Step S35: Batch processing optimization: When processing high-concurrency requests, the system performs batch processing on the projection calculation of step S32, splices multiple from different video segments or different requests into a large tensor (BatchTensor), and uses the parallel computing capability of the GPU to complete the matrix multiplication and operation at one time, maximizing the GPU Throughput; Optimization comparison: Unoptimized: Wait for 20 minutes for video download to complete → decode all frames → batch encoding, users need to wait for tens of seconds to start asking questions; After optimization (using step S3 in this embodiment): encoding and transmission overlap, 95% of VisualKV is ready at the end of video transmission, achieving "second opening".

[0038] Step S4: Full video memory residence and Pin strategy: Since the 20-minute video is only about 10k Tokens after compression by step S2, the 80GB memory of H100 is extremely abundant (10k Token KVCache only occupies about 2-4GB): Implementation details: Step S41: The KVEntry written in step S3 will be tagged with type=video, segment_id, etc. to distinguish the data type and the segment it belongs to; Step S42 partition: VideoKVRegion is set to Non-Evictable mode; Step S43 Pin: Perform pin() operation on all Segments of the meeting, reference count Lock; Step S44 load decision: the decision formula is constantly determined as Hit, saving all PCIe data transfer overhead; Step S45 on-demand release / overflow: when And mem_pressure (memory pressure) is greater than (threshold): Evict (evict) or compress (page-level overflow to main memory / disk) according to the strategy: ; The memory pressure threshold mentioned above That is, when the GPU memory occupancy rate exceeds And the reference count is 0, trigger the eviction / overflow (Evict / Swap) of KVBlock, in this embodiment, its value range is 0.7 to 0.95, and the preferred value is 0.85, the basis is: 1) Fragmentation reservation: although vLLM uses PagedAttention to reduce fragmentation, the memory allocator (Allocator) still needs to reserve about 15% of the buffer space to cope with sudden KernelWorkspace application or large Activation value peaks; 2) Performance balance: if set to 95%, it is easy to trigger OOM (OutOfMemory) at the peak of concurrency, causing the service to crash; if set to 60%, the memory utilization is too low, wasting expensive H100 resources, and 85% is the "safe water line" recognized by the industry; Step S46 Segment management: segment the long video into several SegmentKVBlock, which can be loaded / released individually, to realize memory-friendly inference of long video (avoid occupying too much memory at one time); Step S47 Interface and atomic operation: KVCache provides atomic operations: pin(segment), unpin(segment), load(segment), evict(segment), and query operations: is_loaded(segment); Step S48 Consistency and recovery: the persistent storage saves the kv_id→storage_offset mapping, supporting recovery from external storage after crash; Optimization comparison: Unoptimized: if the number of tokens exceeds 300,000, the memory is insufficient, and frequent Swap (memory and memory exchange) is triggered, resulting in a cliff-like drop in generation speed (reduced to less than 10 tokens / s); Optimized (using step S4 of the embodiment): 100% data resides in HBM3 high-speed memory, maximizing bandwidth utilization.

[0039] Step S5: vLLM cross-modal inference and speed explosion (Flash-Decoding): Implementation details: Step S51 scheduling preparation: the scheduler collects the requests of the current batch, determines the required VideoKVsegment / block list B_needed; Step S52 load blocks on demand: for B_needed, call step S4 interface to ensure that the required blocks are loaded to the GPU (or trigger load); (use the storage layer interface defined in S4 to realize instant data preparation); Step S53 block Attention: although the total number of tokens is reduced, to achieve extreme speed, 10k tokens are still divided into blocks of block_size=16; Flash-Decoding kernel: on H100, use the vLLM optimized Flash-Decoding kernel, which parallelly loads different segments of KVCache for Softmax calculation when processing LongContext (long context), greatly relieving the serial bottleneck of Attention operator; Step S54 dynamic generation of text KV: text input (user query, prompt) generates corresponding textKV at each step and writes it to TextKVRegion (which can be recycled); Step S55 Sparse Search: For specific questions (such as "What was discussed in the 5th minute of the meeting?"), the scheduler only activates the Block corresponding to the timestamp for calculation, further reducing the calculation amount to almost zero; Step S56 Decode Merge: vLLM scheduler merges multiple request groups of the same decode step into a batch to share KV loading and kernel calls to improve GPU utilization; Step S57 Intermediate Cache and Flow Control: Use stream / queue control for large batches to avoid excessive block loading causing memory jitter; Step S58 End and Release Count: After inference, step S4 updates the reference count and releases or retains Video KV according to the strategy.

[0040] Optimization comparison: Unoptimized: When processing 300,000 long sequences, Attention is Complexity, Prefill stage is extremely slow, Decode stage is limited by video memory bandwidth, speed is difficult to exceed 15-20 tokens / s; After optimization (using step S5 of the embodiment): Prefill (first word generation): Since the input is only 10k tokens, H100 can complete processing in less than 200ms; Decode (generation speed): Based on short sequences of step S2 + Flash-Decoding of step S5, the generation speed is easily stabilized at 60-80 tokens / s, fully meeting the real-time reading feeling.

[0041] S6: Structured meeting summary generation: Generate a video summary based on meeting pictures, subtitles, and voice content.

[0042] As Figure 2 shown, the embodiment also provides a multi-modal large model video content understanding inference acceleration system, comprising: An input module for inputting video data and pre-processing the video data; A compression module for adaptively compressing video frame sequences into video tokens; An encoding module for encoding visual features and generating KV from the compressed visual token set; A cache management module for partition management and scheduling of visual KV data; An inference module for performing cross-modal inference based on the vLLM framework; An output module for outputting and post-processing the video content understanding result.

[0043] The above describes the preferred embodiments of the present application, but the present application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A method for accelerating multimodal large-model video content understanding and reasoning, characterized in that, Includes the following steps: Step S1: Input video data and preprocess the video data to generate a video frame sequence; Step S2: Perform adaptive video token compression on the video frame sequence generated in step S1, and output a compressed visual token set; Step S3: Perform visual feature encoding and Key-Value generation on the compressed visual token set output in step S2 to obtain visual KV data; Step S4: Perform video KV cache partition management on the visual KV data obtained in step S3, allocate independent storage areas for visual KV and set management strategies; Step S5: Perform cross-modal inference based on the vLLM framework. The cross-modal inference includes, but is not limited to: generating text KV data and combining visual KV data with text KV data to generate video content understanding results. Step S6: Output the video content understanding results and perform post-processing and structured mapping.

2. The method for accelerating multimodal large-model video content understanding and reasoning according to claim 1, characterized in that, In step S2, the adaptive video token compression includes the following steps: Step S21: Extract features from each video frame to obtain the frame feature vector and PatchToken set; Step S22: Calculate the semantic variation degree between adjacent frames, and evaluate it based on cosine distance or fused optical flow energy; Step S23: Select keyframes and mark redundant frames based on the semantic variability and the preset threshold; Step S24: Score the importance of each patch, taking into account feature strength, temporal variation, and attention saliency; Step S25: Perform hierarchical temporal aggregation on similar patches in redundant frames to generate aggregated tokens; Step S26: Dynamically allocate Token quotas based on the video content change rate to control the number of output Tokens; Step S27: Prune and output the aggregated tokens, and associate them with metadata; Step S28: Monitor the compression ratio and task accuracy through a quality control loop and dynamically adjust the compression parameters.

3. The method for accelerating multimodal large-model video content understanding and reasoning according to claim 2, characterized in that, In step S22, the semantic change degree is expressed as: ； in, , These are the global frame vectors for the current frame and the previous frame, respectively. Represents the dot product. Represents the vector norm; The degree of change after merging optical flow energy is expressed as: ； in, These are weighting coefficients. This represents the normalized optical flow intensity.

4. The method for accelerating multimodal large-model video content understanding and reasoning according to claim 3, characterized in that, In step S24, the importance score for each patch is represented as follows: ； in, The L2 norm of the eigenvectors. Indicates within the time window The variance of the patch. Indicates saliency or attention proxy score, , , These are the learnable weight coefficients.

5. The method for accelerating multimodal large-model video content understanding and reasoning according to claim 3, characterized in that, Step S26 includes: Calculation time window average degree of change : ； Token allocation based on the degree of change : ； in, To minimize the number of tokens to retain, For the total token budget, The sensitivity coefficient, This is the floor function.

6. The method for accelerating multimodal large-model video content understanding and reasoning according to claim 1, characterized in that, Step S3 includes: Linear projection is performed on the compressed visual token to map it to the hidden layer dimension of the large language model, generating key and value vectors; Construct the KVEntry data structure, including a unique hash identifier, key vector, value vector, video ID, segment ID, token index, and importance score; Generate globally unique values ​​using a hash algorithm. : ； The generated KVEntry is transferred to the cache management module for storage and retrieval.

7. The method for accelerating multimodal large-model video content understanding and reasoning according to claim 1, characterized in that, In step S4, the video KV cache partition management includes: Allocate a separate storage area for visual key-value pairs and set type labels for identification; Perform persistence operations on visual key-value segments, maintain reference counts, and support reuse across requests; Load or release visual key-value segments on demand, triggering management operations based on reference count and memory pressure; When reference counting And the memory pressure exceeds the threshold When this occurs, a release or spill operation is triggered: ； Long videos are segmented for efficient inference with high video memory.

8. The method for accelerating multimodal large-model video content understanding and reasoning according to claim 1, characterized in that, In step S5, the cross-modal reasoning includes: Generate text key-value pairs, which are dynamically generated from user-input text through a language model; Attention is calculated by combining the visual key-value pairs with the text key-value pairs, and the pagedAttention mechanism of vLLM is used to load visual key-value blocks on demand. Perform block-based attention calculation by dividing the visual key-value pairs into blocks and performing attention calculation for each block. Improve GPU utilization and inference throughput through pipeline scheduling and batch processing optimization.

9. The method for accelerating multimodal large-model video content understanding and reasoning according to claim 1, characterized in that, In step S6, the post-processing and structured mapping includes: Collect the token stream output by vLLM and parse it into structured text results; The metadata of visual tokens is used to perform time and space backtracking mapping to generate timestamps and region labels; The output results are merged and deduplicated based on attention intensity and confidence scores; Output the final video content understanding result and selectively retain or release the visual key-value cache.

10. A multimodal large-model video content understanding and reasoning acceleration system, used to implement the multimodal large-model video content understanding and reasoning acceleration method as described in any one of claims 1-9, characterized in that, include: The input module is used to input video data and preprocess the video data; The compression module is used to perform adaptive video token compression on video frame sequences. The encoding module is used to encode visual features and generate key-value pairs from the compressed visual token set. The cache management module is used for partitioning and scheduling visual key-value data. The inference module is used to perform cross-modal inference based on the vLLM framework; The output module is used to output and post-process the video content understanding results.

Images