A heterogeneous encoding scheduling method and system for multi-modal large model inference
Patent Information
- Application Number
- CN202611210828.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-11
- Publication Date
- 2026-11-03
AI Technical Summary
In existing multimodal large model inference services, the static allocation of GPU resources in the encoding, pre-filling and decoding stages is unreasonable, resulting in high latency and difficulty in optimizing throughput due to visual load fluctuations. In addition, the latency of the general CPU framework fluctuates greatly and cannot effectively schedule heterogeneous resources.
By employing a heterogeneous encoding allocation method based on global caching deduplication, residual visual workload modeling, morphological clustering, and expected first word latency margin, visual input is dynamically scheduled to CPU or GPU encoding workers. By combining CPU and GPU parallel execution, fine-grained token streaming overlap and dynamic rebalancing across workers are achieved.
It significantly shortened the initial word generation latency, optimized the overall throughput, improved the system's adaptability to multimodal heterogeneous loads, and reduced unnecessary computational overhead and latency.
Smart Images
Figure CN123033095A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of visual coding technology, and in particular to a heterogeneous coding scheduling method and system for multimodal large model reasoning. Background Technology
[0002] With the widespread deployment of large-scale multimodal models in multimodal dialogue and visual understanding tasks, efficient scheduling of multimodal large model inference services has attracted much attention. The separation architecture of encoding, pre-filling, and decoding has become the mainstream solution to solve the coupling problem between visual encoding and language model inference. It decouples the inference process into three stages: visual encoding, pre-filling, and decoding, and introduces an independent GPU encoding worker pool to alleviate the direct interference of encoding on pre-filling. However, this pure GPU EPD solution still has inherent limitations: in a fixed GPU resource pool, the number of GPUs needs to be statically allocated in the encoding, pre-filling, and decoding stages. When the visual load fluctuates drastically in multimodal requests, queuing is likely to occur due to insufficient encoding GPUs, which will increase the first word latency, or pre-filling and decoding capacity will be squeezed due to reserving too many encoding GPUs. The computational cost of visual encoding is mainly determined by image resolution, patch partitioning, and the number of visual tokens. Traditional static routing strategies such as round-robin or least-request-number cannot accurately reflect the heterogeneity of requests, exacerbating GPU resource competition and long-tail latency.
[0003] Existing technologies often rely solely on the number of requests or a rough image count for scheduling decisions, neglecting the fine features of the visual input within multimodal requests and the real-time load of the downstream pre-filling stage. This leads to lightweight requests being easily blocked by heavyweight requests, making it difficult to achieve an optimal balance between first-word latency and throughput. Although some studies have proposed using general-purpose CPUs for visual encoding backoff, the lack of end-to-end path optimization in the general-purpose CPU framework results in significant latency fluctuations, making it impossible to incorporate it into unified management as a predictable online scheduling resource. There is an urgent need for a scheduling method that can accurately quantify the residual visual workload before the encoding stage and dynamically coordinate heterogeneous CPU and GPU resources based on the latency budget of the downstream language model stage, in order to achieve globally optimal allocation of computing resources in multimodal inference services. Summary of the Invention
[0004] In view of the aforementioned existing problems, the present invention is proposed.
[0005] Therefore, this invention provides a heterogeneous coding scheduling method for multimodal large model inference to solve how to deduplication through gateway-side global caching, residual visual workload modeling, morphological perception clustering, and heterogeneous coding allocation based on expected first word delay margin.
[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a heterogeneous coding scheduling method for multimodal large model inference, which includes receiving a multimodal request and extracting text, visual input, model version, preprocessing parameters and first word latency budget; A global cache key is generated based on parameters such as visual input. The global encoding cache is queried, and the visual input status is divided into three categories: completed, processing, and missed. Establish request-wait relationships for processing states, extract sets for missed states, calculate the estimated encoding cost for each missed input, calculate the residual visual workload, and perform morphological perception clustering based on the number of patches and resolution. Based on the GPU queue depth, historical latency, first-word latency budget, residual workload, and clustering results, the expected first-word latency margin is obtained. A heterogeneous hardware-aware encoding batching strategy is adopted to adaptively partition and pipeline small and large graph clusters, allocating them to CPU or GPU encoding workers. Specifically, the CPU encoding backend implements weight pre-packaging, NUMA node binding, and workspace memory reuse methods to reduce operator startup overhead and improve memory locality. CPU encoding results are delivered with low overhead through zero-copy or shared memory methods, and an encoding task anomaly isolation mechanism is established to independently isolate and quickly recover from CPU / GPU encoding failures, timeouts, or health check anomalies. Visual encoding is performed in parallel by CPU and GPU, progress is monitored by an embedded tracker, and dynamic cloning and parallel appending are based on the difference between pre-padding and encoding speeds. The complete visual embedding is combined with the text, self-attention computation is performed, and the final text response is generated by autoregression of the decoder.
[0007] As a preferred embodiment of the heterogeneous coding scheduling method for multimodal large-model inference described in this invention, the step of generating a global cache key based on parameters such as visual input, querying the global coding cache, and classifying the visual input status into three categories: completed, processing, and missed, specifically includes: Based on the visual input, model version, preprocessing parameters, and accuracy configuration, a global cache key is obtained through a deterministic identifier generation function. The global cache key is then sent to the global encoding cache module for querying. The query results returned by the global encoding cache module yield the state classification result for each visual input, which includes three categories: completed, processing, and missed.
[0008] As a preferred embodiment of the heterogeneous coding scheduling method for multimodal large model inference described in this invention, the steps of establishing request-wait relationships for processing states, extracting sets of missed states, calculating the estimated coding cost for each missed input, obtaining residual visual workload, and performing morphological perception clustering based on the number of patches and resolution specifically include: Based on the state division results, for visual inputs in the processing state, the current request is registered as a waiter to establish a waiting relationship; for visual inputs in the completed state, the corresponding visual embedding is directly read from the global encoding cache module to obtain the set of missed visual inputs. The estimated encoding cost of each missed visual input is calculated based on the set of missed visual inputs, and the residual visual workload is obtained by summarizing them. The set of missed visual inputs is then subjected to morphological perception clustering according to the number of patches and the resolution, resulting in small image clusters and large image clusters.
[0009] As a preferred embodiment of the heterogeneous encoding scheduling method for multimodal large-model inference described in this invention, the step of obtaining the expected first-word latency margin based on combining GPU queue depth, historical latency, first-word latency budget, residual workload, and clustering results, and allocating small and large graph clusters to CPU or GPU encoding workers, specifically includes: The current queue depth and historical prefilling time of the GPU prefilling worker are obtained. Combined with the first word latency budget, the residual visual workload, and the small and large graph clusters, the expected first word latency margin is obtained. Based on this, a heterogeneous hardware-aware encoding batch processing strategy is executed: small graph clusters of homogeneous form are batch aggregated on the CPU, and large graph clusters are arranged in tensor parallelism or pipeline parallelism on the GPU. The CPU encoding backend loads a static computation graph pre-packaged with weights at startup, binds the computation thread and memory allocation to the same NUMA node according to the system topology information, and reuses a fixed-size work area buffer between multiple encoding tasks to reduce the overhead of dynamic memory allocation and release. The CPU encoding results are delivered with low overhead through shared memory or device-mapped memory to avoid additional copying. At the same time, an exception isolation domain is constructed. When the CPU matrix expansion worker encounters an exception or timeout, only the corresponding encoding subtask is interrupted and rescheduled through the gateway, without affecting the GPU encoding and prefilling process. When the expected first word delay margin is lower than a preset threshold, the small image cluster is assigned to the CPU matrix expansion encoder; the large image cluster is assigned to the GPU encoder; when the expected first word delay margin is higher than or equal to the preset threshold, both the small image cluster and the large image cluster are assigned to the GPU encoder, thus obtaining the assigned subset to be encoded and the target encoder information.
[0010] As a preferred embodiment of the heterogeneous encoding scheduling method for multimodal large-model inference described in this invention, the method of executing visual encoding in parallel using CPU and GPU, monitoring progress through an embedded tracker, and dynamically cloning and parallel appending based on the difference between pre-padding and encoding speeds specifically includes: Based on the allocated subsets to be encoded and the target worker information, the CPU matrix expands the encoding worker and the GPU encoding worker to perform visual encoding in parallel. During the encoding process, the progress of visual token production for each subset is monitored by an embedded tracker. The collaborative relationship between the modules is as follows: the preprocessing module is responsible for request parsing and feature extraction, the caching module maintains the global encoding cache and state partitioning, the clustering module outputs small and large graph clusters for the encoding module to schedule, and the encoding module dispatches tasks to the CPU or GPU according to the heterogeneous hardware awareness strategy and provides real-time progress feedback through the embedded tracker. When any subset produces the first batch of visual token blocks, the visual tokens of each ready subset are concatenated in their original order to form a partial visual embedding and delivered to the GPU prefill worker to start block prefilling and reserve key-value cache slots, and to track the range of visual tokens that the GPU prefill worker has written to the key-value cache. When the pre-filling processing speed exceeds the encoding output speed, the remaining unencoded parts of the lagging subset are dynamically cloned and dispatched to idle GPU encoding workers for parallel execution. The results completed first are appended to the key-value cache and the other copy is canceled to obtain the complete visual embedding and the pre-filled results already written to the key-value cache.
[0011] As a preferred embodiment of the heterogeneous encoding scheduling method for multimodal large-model inference described in this invention, the step of combining the complete visual embedding with text, performing self-attention computation, and generating the final text response by autoregression from the decoder specifically includes: Based on the complete visual embedding and the pre-filled result written to the key-value cache, the GPU pre-fill worker combines the complete visual embedding with the text input into a multimodal cue sequence and performs self-attention calculation. The calculation result is then passed to the GPU decoding worker, which generates an output token through autoregression, forming the final text response to the user's request.
[0012] As a preferred embodiment of the heterogeneous encoding scheduling method for multimodal large-model inference described in this invention, after the GPU decoding worker generates the output token autoregressively to form the final text response to the user request, the method further includes: When any encoding worker fails or times out, the gateway puts the task back into the scheduling queue and selects a worker of the same type to retry or switches to another worker based on the failure type. If the CPU matrix expansion worker is overloaded or fails a health check, new tasks are suspended from being dispatched to it. If the GPU encoding worker is under low pressure, more tasks are assigned to the GPU encoding worker. The global encoding cache module is managed using capacity limits and least recently used strategies, and timeout and cleanup mechanisms are set for in-transit states to prevent long-term blocking of waiting tasks due to abnormal exit of the owner's task. After abnormal recovery, the CPU encoding backend quickly rebuilds the execution environment based on the bound NUMA nodes and the reused state of the work area, reducing cold start overhead and ensuring the continuous service capability of the overall system.
[0013] Secondly, the present invention provides a heterogeneous coding scheduling system for multimodal large model inference, including a preprocessing module that receives multimodal requests and extracts text, visual input, model version, preprocessing parameters and first word latency budget. The caching module generates a global cache key based on parameters such as visual input, queries the global encoding cache, and classifies the visual input status into three categories: completed, processing, and missed. The clustering module establishes request-wait relationships for processing states, extracts sets for missed states, calculates the estimated encoding cost for each missed input, calculates the residual visual workload, and performs morphological perception clustering based on the number of patches and resolution. The encoding module, based on the GPU queue depth, historical latency, first word latency budget, residual workload and clustering results, obtains the expected first word latency margin. It adopts a heterogeneous hardware-aware encoding batch processing strategy, combined with the CPU backend's weight pre-packaging, NUMA binding and workspace reuse methods, as well as the CPU encoding result low-overhead delivery and anomaly isolation mechanism, to allocate small and large graph clusters to CPU or GPU encoding workers. The append module performs visual encoding in parallel via CPU and GPU, monitors progress via an embedded tracker, and performs dynamic cloning and parallel appending based on the difference between pre-padding and encoding speeds. The execution module combines the complete visual embedding with the text, performs self-attention computation, and generates the final text response through autoregression by the decoder; The collaborative relationship among the modules is as follows: the preprocessing module provides standardized request metadata to downstream modules; the caching module provides state partitioning results to the clustering module; the clustering module provides morphology-aware cluster structures to the encoding module; the encoding module drives parallel encoding of CPU and GPU according to the heterogeneous hardware awareness strategy, and implements block pre-filling and dynamic cloning through the append module; after the multimodal cue sequence is constructed, the execution module takes over the self-attention calculation and decoding generation, forming an end-to-end closed-loop scheduling link.
[0014] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the heterogeneous coding scheduling method for multimodal large model inference as described in the first aspect of the present invention.
[0015] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the heterogeneous coding scheduling method for multimodal large model inference as described in the first aspect of the present invention.
[0016] The beneficial effects of this invention are as follows: By receiving multimodal requests and extracting text, visual input, and first-word latency budget, combined with the establishment of global cache key lookup and in-transit waiting relationships, redundant encoding of repeated visual inputs is effectively eliminated, significantly reducing ineffective computational overhead; by calculating the estimated encoding cost of missed inputs and summarizing it to obtain the residual visual workload, precise quantitative basis is provided for scheduling decisions; on this basis, by performing morphological perception clustering on missed visual inputs based on patch quantity and resolution to obtain small and large image clusters, fine-grained splitting of visual input subsets within requests is achieved, creating a prerequisite for subsequent differentiated allocation of heterogeneous resources; by obtaining the queue depth and historical latency of the GPU pre-filled worker and combining it with the first-word latency budget and residual workload, the expected first-word latency margin is obtained, and the small image... Clusters and large graph clusters are dynamically allocated to CPU matrix expansion encoders or GPU encoders, enabling the service level target of the downstream language model stage to be backpropagated to the encoding side to regulate the heterogeneous skew rate. This optimizes the overall throughput and reduces the first word latency under fixed GPU resources. Visual encoding is executed in parallel by CPU and GPU, and the progress is monitored by an embedding tracker. When the first batch of visual token blocks are produced in any subset, they are concatenated into partial visual embeddings and delivered to the GPU pre-filling encoder to start block pre-filling. At the same time, based on the difference between pre-filling and encoding speeds, lagging subsets are dynamically cloned and appended in parallel. This achieves fine-grained token streaming overlap and dynamic rebalancing across encoders and pre-filling stages, significantly shortening the first word generation latency and improving the system's adaptability to multimodal heterogeneous loads. Attached Figure Description
[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1This is a flowchart of a heterogeneous coding scheduling method for multimodal large model inference.
[0019] Figure 2 This is a schematic diagram of a heterogeneous coding scheduling system for multimodal large model inference. Detailed Implementation
[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0021] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.
[0022] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.
[0023] Reference Figures 1-2 This is one embodiment of the present invention, which provides a heterogeneous coding scheduling method for multimodal large model inference, comprising the following steps: Based on the visual input, model version, preprocessing parameters, and accuracy configuration, a global cache key is obtained through a deterministic identifier generation function. The global cache key is then sent to the global encoding cache module for querying. The query results returned by the global encoding cache module yield the state classification result for each visual input, which includes three categories: completed, processing, and missed.
[0024] Furthermore, upon receiving the parsed multimodal request, the encoding model gateway extracts the visual content or normalized content, visual encoder model version, preprocessing parameters such as cropping, scaling, and patch partitioning, as well as quantization format and other precision configurations affecting the semantics of the visual embedding for each visual input (such as an image, video frame, or document page) in the request. The gateway inputs these fields together into a deterministic identifier generation function H to calculate a globally unique cache key K for that visual input, K = H(x, m, p, q). The gateway then sends this cache key K to the global encoding cache module for querying. The global encoding cache module maintains all encoded visual embeddings and their states, returning the corresponding query results based on the cache key. Based on these results, the gateway explicitly classifies the state of each visual input into three categories: if an identical visual embedding with consistent semantics exists in the cache, it is marked as "completed"; if it does not exist in the cache but other requests have triggered the encoding of the same visual input and it has not yet been completed, it is marked as "processing"; if no relevant record exists in the cache, it is marked as "missed". This state partitioning provides direct input for subsequent steps, enabling the system to identify reusable, mergeable, and newly encoded visual inputs before scheduling, thereby avoiding ineffective encoding resource consumption.
[0025] Based on the state division results, for visual inputs in the processing state, the current request is registered as a waiter to establish a waiting relationship. For visual inputs in the completed state, the corresponding visual embedding is directly read from the global encoding cache module to obtain a set of missed visual inputs. The encoding cost estimate for each missed visual input is calculated based on the set of missed visual inputs, and the residual visual workload is obtained by summarizing them. The set of missed visual inputs is then subjected to morphological perception clustering according to the number of patches and the resolution to obtain small image clusters and large image clusters. Specifically, the expression is:
[0026] in, For residual visual workload, The index variable for visual input. For visual input set, This is the cache state for the i-th visual input. For the first An estimated encoding cost for each visual input. This indicates a cache hit.
[0027] Furthermore, using the state partitioning result obtained in the previous step as input, the gateway first processes visual inputs in the "processing" state: for these inputs, the gateway does not repeatedly initiate encoding tasks, but registers the current request as a waiter, establishing a waiting relationship with the original owner task. Once the owner task completes its encoding, the gateway directly reuses its result, thereby eliminating in-transit duplicate encoding. Next, for visual inputs in the "completed" state, the gateway directly reads the corresponding visual embedding from the global encoding cache module. These inputs are no longer included in subsequent encoding scheduling. After the above deduplication, the remaining "missed" visual inputs constitute the set to be encoded. For each missed visual input in this set, the gateway uses the number of visual tokens as a base feature, combined with image resolution, window attention shape, historical profiling data, and data transmission overhead, to calculate its estimated encoding cost u_i through methods such as table lookup, piecewise linear models, or online sliding statistics. Subsequently, the estimated encoding costs of all missed visual inputs are aggregated to obtain the residual visual workload for the request. The gateway then performs morphological clustering on the set of missed visual inputs based on the number of patches and resolution: those with fewer patches and lower resolution are grouped into small clusters, while those with more patches and higher resolution are grouped into large clusters. The small and large clusters output from this step are directly used as input for the next step of heterogeneous allocation, achieving fine-grained segmentation of the visual input subset within the request and laying the foundation for differentiated scheduling.
[0028] The current queue depth and historical pre-filling time of the GPU pre-filling worker are obtained. Combined with the first-word latency budget, the residual visual workload, and the small and large graph clusters, the expected first-word latency margin is calculated. Based on this, a heterogeneous hardware-aware encoding batch processing strategy is executed: small graph clusters of homogeneous form are batch-aggregated on the CPU, and large graph clusters are arranged in tensor parallelism or pipelined parallelism on the GPU. The CPU encoding backend loads a pre-packaged static computation graph at startup, binds computation threads and memory allocation to the same NUMA node based on system topology information, and reuses a fixed-size work area buffer between multiple encoding tasks to reduce dynamic memory allocation and deallocation. The CPU encoding results are delivered with low overhead via shared memory or device-mapped memory, avoiding additional copying. Simultaneously, an exception isolation domain is constructed. When the CPU matrix expansion worker experiences an exception or timeout, only the corresponding encoding subtask is interrupted and rescheduled through the gateway, without affecting the GPU encoding and pre-filling process. When the expected first-word latency margin is lower than a preset threshold, the small image cluster is assigned to the CPU matrix expansion encoding worker; the large image cluster is assigned to the GPU encoding worker. When the expected first-word latency margin is higher than or equal to the preset threshold, both the small and large image clusters are assigned to the GPU encoding worker, obtaining the assigned subset to be encoded and target worker information.
[0029] Furthermore, the gateway takes the small and large image clusters obtained in the previous step, the residual visual workload, and the requested first-word latency budget as input, while simultaneously obtaining the current queue depth and historical pre-filling P90 time from the GPU pre-filling worker in real time. The gateway constructs an expected first-word latency margin model: Expected first-word latency margin = First-word latency budget - (Estimated encoding completion time + Estimated pre-filling time). The estimated encoding completion time is estimated based on the residual visual workload and the service time model of the candidate worker type (CPU matrix expansion or GPU), and the estimated pre-filling time is extrapolated based on the queue depth and historical time. If the expected margin is lower than a preset threshold, it indicates that the pre-filling stage is about to become a system bottleneck. In this case, the gateway executes SLO backpropagation backoff: it forcibly allocates small graph clusters with low computational cost to the CPU matrix expansion encoder to reserve GPU resources for pre-filling and decoding, while allocating large graph clusters with high computational cost to the GPU encoder. If the expected margin is higher than or equal to the threshold, it indicates that the pre-filling pressure is relatively light, and both small and large graph clusters can be allocated to the GPU encoder to accelerate encoding. After allocation, the gateway generates information on the allocated subsets to be encoded and the target encoders, including which visual inputs each subset contains, which encoder type it is assigned to, and the corresponding network address or logical interface. This information directly drives the next step of parallel encoding execution, realizing the back-modulation of the upstream heterogeneous encoding skew rate by the latency budget of the downstream language model stage.
[0030] Based on the allocated subsets to be encoded and the target worker information, the CPU matrix expands the encoding worker and the GPU encoding worker to perform visual encoding in parallel. During the encoding process, the progress of visual token production for each subset is monitored by an embedding tracker. The collaborative relationship between the modules is as follows: the preprocessing module is responsible for request parsing and feature extraction; the caching module maintains the global encoding cache and state partitioning; the clustering module outputs small and large graph clusters for the encoding module to schedule; the encoding module dispatches tasks to the CPU or GPU according to the heterogeneous hardware awareness strategy and provides real-time progress feedback through the embedding tracker; when any subset produces... When the first batch of visual token blocks is generated, the visual tokens of each ready subset are concatenated in their original order to form a partial visual embedding and delivered to the GPU pre-fill worker to start block pre-filling and reserve key-value cache slots. The range of visual tokens that the GPU pre-fill worker has written to the key-value cache is tracked. When the pre-filling processing speed exceeds the encoding output speed, the remaining unencoded parts of the lagging subset are dynamically cloned and dispatched to idle GPU encoding workers for parallel execution. The results completed first are appended to the key-value cache and the other copy is canceled to obtain the complete visual embedding and the pre-filled results that have been written to the key-value cache.
[0031] Furthermore, the CPU matrix expansion encoding worker and the GPU encoding worker in the heterogeneous encoding resource pool execute the encoding tasks of their respective visual subsets in parallel, based on the allocation information from the previous step. The CPU matrix expansion encoding worker uses a dedicated backend for the visual Transformer, mapping dense matrices such as patch projection and QKV projection to a matrix expansion kernel (such as AMX), and controlling end-to-end latency through layout preservation, weighted packing reuse, and small-shape lightweight paths. During the encoding process, the gateway maintains an embedding tracker for each request, which continuously monitors the visual token production progress of each subset. When any subset (regardless of whether it comes from the CPU or GPU) produces its first batch of visual token blocks, the gateway immediately concatenates the ready visual tokens of each subset into a partial visual embedding in the order of the original request and delivers it to the GPU pre-filling worker, triggering block pre-filling computation, while reserving a complete key-value cache slot for this request on the GPU pre-filling worker. Subsequently, the embedding tracker continues to track the range of visual tokens written to the key-value cache by the GPU pre-filling worker. If the pre-filling of text or ready tokens is faster than the progress of the encoding worker for a certain subset, causing pre-filling to potentially idle and wait, the gateway dynamically clones the remaining unencoded portion of that lagging subset and dispatches it to an idle GPU encoding worker for parallel execution. The two encodings compete for output, and the result completed first is appended to the key-value cache, while the other unfinished task is canceled. In this way, encoding and pre-filling form a fine-grained pipeline overlap at the token granularity, ultimately generating a complete visual embedding and a pre-filled result written to the key-value cache.
[0032] Based on the complete visual embedding and the pre-filled result written to the key-value cache, the GPU pre-fill worker combines the complete visual embedding with the text input into a multimodal cue sequence and performs self-attention calculation. The calculation result is then passed to the GPU decoding worker, which generates an output token through autoregression, forming the final text response to the user's request.
[0033] Furthermore, the GPU pre-fill worker takes the complete visual embedding obtained in the previous step and the pre-filled result already written to the key-value cache as input, and combines the visual embedding with the text input in the request into a multimodal cue sequence according to the model's requirements. Subsequently, the GPU pre-fill worker performs a forward self-attention computation on this complete sequence, processing all historical context and expanding the key-value cache. After computation, the GPU pre-fill worker passes the generated context state and the updated key-value cache to the GPU decoding worker. Based on this, the GPU decoding worker generates output tokens one by one in an autoregressive manner until a complete final text response to the user's request is formed. At this point, the entire multimodal inference process ends, and the output of this step is the final text response, which is directly returned to the user.
[0034] When any encoding worker fails or times out, the gateway puts the task back into the scheduling queue and selects a worker of the same type to retry or switches to another worker based on the failure type. If the CPU matrix expansion worker is overloaded or fails a health check, new tasks are suspended from being assigned to it. If the GPU encoding worker is under low pressure, more tasks are assigned to the GPU encoding worker. The global encoding cache module is managed using capacity limits and least recently used strategies, and timeout and cleanup mechanisms are set for in-transit states to prevent long-term blocking of waiting tasks due to abnormal exit of the owner's task. After the CPU encoding backend recovers from an anomaly, it quickly rebuilds the execution environment based on the bound NUMA nodes and the reuse status of the work area, reducing cold start overhead and ensuring the continuous service capability of the overall system.
[0035] Furthermore, during continuous system operation, the gateway monitors the execution status of the encoding workers in each of the above steps in real time. If any CPU matrix expansion encoding worker or GPU encoding worker fails or times out while executing a task, the gateway puts the task back into the scheduling queue and decides whether to retry with the same type of worker or switch to a different type of worker based on the error type (such as transient network failure, resource exhaustion, or permanent error). Simultaneously, the gateway periodically collects the load and health metrics of each worker: if the CPU matrix expansion worker has excessive load or fails a health check, the gateway immediately suspends dispatching new tasks to it and redirects traffic to the GPU encoding worker; if the GPU encoding worker has low pressure and available capacity, it proactively allocates more encoding tasks to the GPU. In addition, the global encoding cache module employs capacity limits and a Least Recently Used (LRU) strategy for self-management, eviction of cold data to prevent unlimited memory growth; for in-transit status, the system has timeout and cleanup mechanisms. Once the owner's task exits abnormally or fails to complete within a predetermined time, the gateway forcibly clears the in-transit marker, waking up the waiting worker or changing it to a missed state and re-initiating scheduling, thereby preventing long-term blocking of waiting workers. These exception handling and dynamic backoff mechanisms ensure the stability and reliability of the entire heterogeneous coding scheduling system in complex production environments.
[0036] This embodiment also provides a heterogeneous coding scheduling system for multimodal large model inference, including: The preprocessing module receives multimodal requests and extracts text, visual input, model version, preprocessing parameters, and first-word latency budget. The caching module generates a global cache key based on parameters such as visual input, queries the global encoding cache, and classifies the visual input status into three categories: completed, processing, and missed. The clustering module establishes request-wait relationships for processing states, extracts sets for missed states, calculates the estimated encoding cost for each missed input, calculates the residual visual workload, and performs morphological perception clustering based on the number of patches and resolution. The encoding module, based on the GPU queue depth, historical latency, first word latency budget, residual workload and clustering results, obtains the expected first word latency margin. It adopts a heterogeneous hardware-aware encoding batch processing strategy, combined with the CPU backend's weight pre-packaging, NUMA binding and workspace reuse methods, as well as the CPU encoding result low-overhead delivery and anomaly isolation mechanism, to allocate small and large graph clusters to CPU or GPU encoding workers. The append module performs visual encoding in parallel via CPU and GPU, monitors progress via an embedded tracker, and performs dynamic cloning and parallel appending based on the difference between pre-padding and encoding speeds. The execution module combines the complete visual embedding with the text, performs self-attention computation, and generates the final text response through autoregression by the decoder; The collaborative relationship among the modules is as follows: the preprocessing module provides standardized request metadata to downstream modules; the caching module provides state partitioning results to the clustering module; the clustering module provides morphology-aware cluster structures to the encoding module; the encoding module drives parallel encoding of CPU and GPU according to the heterogeneous hardware awareness strategy, and implements block pre-filling and dynamic cloning through the append module; after the multimodal cue sequence is constructed, the execution module takes over the self-attention calculation and decoding generation, forming an end-to-end closed-loop scheduling link.
[0037] This embodiment also provides a computer device applicable to heterogeneous coding scheduling methods for multimodal large model inference, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the heterogeneous coding scheduling method for multimodal large model inference proposed in the above embodiment.
[0038] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.
[0039] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the heterogeneous coding scheduling method for multimodal large model inference as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0040] In summary, this invention effectively eliminates redundant encoding of repetitive visual inputs and significantly reduces ineffective computational overhead by receiving multimodal requests and extracting text, visual input, and first-word latency budgets, combined with the establishment of global cache key lookup and in-transit waiting relationships. By calculating the estimated encoding cost of missed inputs and summarizing it to obtain the residual visual workload, it provides a precise quantitative basis for scheduling decisions. Furthermore, by performing morphological-aware clustering on missed visual inputs based on patch count and resolution to obtain small and large image clusters, it achieves refined splitting of visual input subsets within requests, creating a prerequisite for subsequent differentiated allocation of heterogeneous resources. By obtaining the queue depth and historical latency of the GPU pre-filled worker and combining it with the first-word latency budget and residual workload, it obtains the expected first-word latency margin, and then combines the small image clusters with... Large image clusters are dynamically allocated to either the CPU matrix expansion encoder or the GPU encoder, enabling the service level target of the downstream language model stage to be backpropagated to the encoding side to regulate the heterogeneous skew rate. This optimizes the overall throughput and reduces the first word latency under fixed GPU resources. Visual encoding is executed in parallel by CPU and GPU, and the progress is monitored by an embedding tracker. When the first batch of visual token blocks are produced in any subset, they are concatenated into partial visual embeddings and delivered to the GPU pre-filling encoder to start block pre-filling. At the same time, based on the difference between pre-filling and encoding speeds, the lagging subsets are dynamically cloned and appended in parallel. This achieves fine-grained token streaming overlap and dynamic rebalancing across encoders and pre-filling stages, significantly shortening the first word generation latency and improving the system's adaptability to multimodal heterogeneous loads.
[0041] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A heterogeneous coding scheduling method for multimodal large-scale model inference, characterized in that: include, Receive multimodal requests and extract text, visual input, model version, preprocessing parameters, and first-word latency budget; A global cache key is generated based on parameters such as visual input. The global encoding cache is queried, and the visual input status is divided into three categories: completed, processing, and missed. Establish request-wait relationships for processing states, extract sets for missed states, calculate the estimated encoding cost for each missed input, calculate the residual visual workload, and perform morphological perception clustering based on the number of patches and resolution. Based on the GPU queue depth, historical latency, first-word latency budget, residual workload, and clustering results, the expected first-word latency margin is obtained. A heterogeneous hardware-aware encoding batching strategy is adopted to adaptively partition and pipeline small and large graph clusters, allocating them to CPU or GPU encoding workers. Specifically, the CPU encoding backend implements weight pre-packaging, NUMA node binding, and workspace memory reuse methods to reduce operator startup overhead and improve memory locality. CPU encoding results are delivered with low overhead through zero-copy or shared memory methods, and an encoding task anomaly isolation mechanism is established to independently isolate and quickly recover from CPU / GPU encoding failures, timeouts, or health check anomalies. Visual encoding is performed in parallel by CPU and GPU, progress is monitored by an embedded tracker, and dynamic cloning and parallel appending are based on the difference between pre-padding and encoding speeds. The complete visual embedding is combined with the text, self-attention computation is performed, and the final text response is generated by autoregression of the decoder.
2. The heterogeneous coding scheduling method for multimodal large model inference as described in claim 1, characterized in that: The process involves generating a global cache key based on parameters such as visual input, querying the global encoding cache, and classifying the visual input status into three categories: completed, processing, and missed. Specifically, this includes: Based on the visual input, model version, preprocessing parameters, and accuracy configuration, a global cache key is obtained through a deterministic identifier generation function. The global cache key is then sent to the global encoding cache module for querying. The query results returned by the global encoding cache module yield the state classification result for each visual input, which includes three categories: completed, processing, and missed.
3. The heterogeneous coding scheduling method for multimodal large model inference as described in claim 2, characterized in that: The process involves establishing request-wait relationships for processing states, extracting a set of missed states, calculating an estimated encoding cost for each missed input, obtaining residual visual workload, and performing morphological perception clustering based on the number of patches and resolution. Specifically, this includes: Based on the state division results, for visual inputs in the processing state, the current request is registered as a waiter to establish a waiting relationship; for visual inputs in the completed state, the corresponding visual embedding is directly read from the global encoding cache module to obtain the set of missed visual inputs. The estimated encoding cost of each missed visual input is calculated based on the set of missed visual inputs, and the residual visual workload is obtained by summarizing them. The set of missed visual inputs is then subjected to morphological perception clustering according to the number of patches and the resolution, resulting in small image clusters and large image clusters.
4. The heterogeneous coding scheduling method for multimodal large model inference as described in claim 3, characterized in that: The process involves combining GPU queue depth, historical latency, first-word latency budget, residual workload, and clustering results to obtain the expected first-word latency margin, and then allocating small and large graph clusters to CPU or GPU encoding workers. Specifically, this includes: The current queue depth and historical prefilling time of the GPU prefilling worker are obtained. Combined with the first word latency budget, the residual visual workload, and the small and large graph clusters, the expected first word latency margin is obtained. Based on this, a heterogeneous hardware-aware encoding batch processing strategy is executed: small graph clusters of homogeneous form are batch aggregated on the CPU, and large graph clusters are arranged in tensor parallelism or pipeline parallelism on the GPU. The CPU encoding backend loads a static computation graph pre-packaged with weights at startup, binds the computation thread and memory allocation to the same NUMA node according to the system topology information, and reuses a fixed-size work area buffer between multiple encoding tasks to reduce the overhead of dynamic memory allocation and release. The CPU encoding results are delivered with low overhead through shared memory or device-mapped memory to avoid additional copying. At the same time, an exception isolation domain is constructed. When the CPU matrix expansion worker encounters an exception or timeout, only the corresponding encoding subtask is interrupted and rescheduled through the gateway, without affecting the GPU encoding and prefilling process. When the expected first word delay margin is lower than a preset threshold, the small image cluster is assigned to the CPU matrix expansion encoder; the large image cluster is assigned to the GPU encoder; when the expected first word delay margin is higher than or equal to the preset threshold, both the small image cluster and the large image cluster are assigned to the GPU encoder, thus obtaining the assigned subset to be encoded and the target encoder information.
5. The heterogeneous coding scheduling method for multimodal large model inference as described in claim 4, characterized in that: The process of performing visual encoding in parallel using CPU and GPU, monitoring progress through an embedded tracker, and dynamically cloning and parallel appending based on the speed difference between pre-padding and encoding specifically includes: Based on the allocated subsets to be encoded and the target worker information, the CPU matrix expands the encoding worker and the GPU encoding worker to perform visual encoding in parallel. During the encoding process, the progress of visual token production for each subset is monitored by an embedded tracker. The collaborative relationship between the modules is as follows: the preprocessing module is responsible for request parsing and feature extraction, the caching module maintains the global encoding cache and state partitioning, the clustering module outputs small and large graph clusters for the encoding module to schedule, and the encoding module dispatches tasks to the CPU or GPU according to the heterogeneous hardware awareness strategy and provides real-time progress feedback through the embedded tracker. When any subset produces the first batch of visual token blocks, the visual tokens of each ready subset are concatenated in their original order to form a partial visual embedding and delivered to the GPU prefill worker to start block prefilling and reserve key-value cache slots, and to track the range of visual tokens that the GPU prefill worker has written to the key-value cache. When the pre-filling processing speed exceeds the encoding output speed, the remaining unencoded parts of the lagging subset are dynamically cloned and dispatched to idle GPU encoding workers for parallel execution. The results completed first are appended to the key-value cache and the other copy is canceled to obtain the complete visual embedding and the pre-filled results already written to the key-value cache.
6. The heterogeneous coding scheduling method for multimodal large model inference as described in claim 5, characterized in that: The process of combining the complete visual embedding with text, performing self-attention computation, and generating the final text response via autoregression by the decoder specifically includes: Based on the complete visual embedding and the pre-filled result written to the key-value cache, the GPU pre-fill worker combines the complete visual embedding with the text input into a multimodal cue sequence and performs self-attention calculation. The calculation result is then passed to the GPU decoding worker, which generates an output token through autoregression, forming the final text response to the user's request.
7. The heterogeneous coding scheduling method for multimodal large model inference as described in claim 6, characterized in that: After the GPU decoding worker autoregressively generates the output token to form the final text response to the user's request, it also includes: When any encoding worker fails or times out, the gateway puts the task back into the scheduling queue and selects a worker of the same type to retry or switches to another worker based on the failure type. If the CPU matrix expansion worker is overloaded or fails a health check, new tasks are suspended from being dispatched to it. If the GPU encoding worker is under low pressure, more tasks are assigned to the GPU encoding worker. The global encoding cache module is managed using capacity limits and least recently used strategies, and timeout and cleanup mechanisms are set for in-transit states to prevent long-term blocking of waiting tasks due to abnormal exit of the owner's task. After abnormal recovery, the CPU encoding backend quickly rebuilds the execution environment based on the bound NUMA nodes and the reused state of the work area, reducing cold start overhead and ensuring the continuous service capability of the overall system.
8. A heterogeneous coding scheduling system for multimodal large model inference, based on the heterogeneous coding scheduling method for multimodal large model inference according to any one of claims 1 to 7, characterized in that: include, The preprocessing module receives multimodal requests and extracts text, visual input, model version, preprocessing parameters, and first-word latency budget. The caching module generates a global cache key based on parameters such as visual input, queries the global encoding cache, and classifies the visual input status into three categories: completed, processing, and missed. The clustering module establishes request-wait relationships for processing states, extracts sets for missed states, calculates the estimated encoding cost for each missed input, calculates the residual visual workload, and performs morphological perception clustering based on the number of patches and resolution. The encoding module, based on the GPU queue depth, historical latency, first word latency budget, residual workload and clustering results, obtains the expected first word latency margin. It adopts a heterogeneous hardware-aware encoding batch processing strategy, combined with the CPU backend's weight pre-packaging, NUMA binding and workspace reuse methods, as well as the CPU encoding result low-overhead delivery and anomaly isolation mechanism, to allocate small and large graph clusters to CPU or GPU encoding workers. The append module performs visual encoding in parallel via CPU and GPU, monitors progress via an embedded tracker, and performs dynamic cloning and parallel appending based on the difference between pre-padding and encoding speeds. The execution module combines the complete visual embedding with the text, performs self-attention computation, and generates the final text response through autoregression by the decoder; The collaborative relationship among the modules is as follows: the preprocessing module provides standardized request metadata to downstream modules; the caching module provides state partitioning results to the clustering module; the clustering module provides morphology-aware cluster structures to the encoding module; the encoding module drives parallel encoding of CPU and GPU according to the heterogeneous hardware awareness strategy, and implements block pre-filling and dynamic cloning through the append module; after the multimodal cue sequence is constructed, the execution module takes over the self-attention calculation and decoding generation, forming an end-to-end closed-loop scheduling link.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the heterogeneous coding scheduling method for multimodal large model inference as described in any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the heterogeneous coding scheduling method for multimodal large model inference as described in any one of claims 1 to 7.