An edge computing-oriented large model dynamic scheduling and inference optimization system

Abstract

This invention belongs to the field of computer systems, specifically a dynamic scheduling and inference optimization system for large-scale models oriented towards edge computing. It includes a request scheduling and concurrency management module, a key-value (KV) cache pool management module, a dynamic batch processing and parallel inference scheduling module, an inference engine optimization module, an edge resource awareness module, and a global coordination and control module. The KV cache pool management module establishes a cross-request key-value vector caching system. The dynamic batch processing module identifies the KV sharing relationships between concurrent requests and generates a fusion attention mask matrix to execute batch parallel inference. By adopting the above technical solution, this application can break through the serial paradigm of token-by-token generation, achieve theoretically guaranteed efficient parallel inference, and improve the overall inference throughput and real-time response speed of edge devices in multi-user scenarios.

Description

Technical Field

[0001] This invention belongs to the field of computer systems, specifically a dynamic scheduling and inference optimization system for large models oriented towards edge computing. Background Technology

[0002] Large language models possess excellent semantic understanding and content generation capabilities, and are widely used in various artificial intelligence scenarios and continue to extend to high-value fields. Edge computing can achieve low-latency response and localized data processing for large models, and is a key technology for the widespread adoption of large models. However, edge devices have limited computing power and memory resources, and large models have inherent serial dependency characteristics in autoregressive inference. Efficiently realizing the inference operation of large language models on the edge side is the core technical challenge in this field.

[0003] Existing edge large language model inference optimization technologies mainly include three types of optimization schemes: model compression, operator optimization, and hardware acceleration. By reducing the amount of model computation, reducing memory overhead, and improving hardware computing power, these technologies can improve the inference performance of edge devices and enable the basic deployment and inference application of large language models on edge devices.

[0004] Existing optimization methods are all peripheral patching optimization strategies, which have not broken through the inherent architectural limitations of large models being generated serially by autoregression per token. Model compression suffers from accuracy loss and cannot dynamically adapt to tasks. Operator optimization cannot eliminate serial dependencies between tokens. Hardware acceleration cannot fully leverage the advantages of parallel computing power. There are defects such as serious idle computing power, high latency of multiple concurrent requests, and low inference efficiency. They cannot fundamentally solve the problem of insufficient computing power utilization of large edge models, which seriously restricts the efficient and large-scale application of large language models on the edge.

[0005] Therefore, this invention provides a dynamic scheduling and inference optimization system for large models oriented towards edge computing. Summary of the Invention

[0006] In order to overcome the shortcomings of the prior art, at least one technical problem raised in the background art is solved.

[0007] The technical solution adopted by this invention to solve its technical problem is as follows: A large-model dynamic scheduling and inference optimization system for edge computing, comprising a request scheduling and concurrency management module, a KV cache pool management module, a dynamic batch processing and parallel inference scheduling module, an inference engine optimization module, an edge resource awareness module, and a global coordination and control module; the output of the request scheduling and concurrency management module is connected to the input of the KV cache pool management module; the output of the KV cache pool management module is connected to the input of the dynamic batch processing and parallel inference scheduling module; the dynamic batch processing and parallel inference scheduling module is communicatively connected to the inference engine optimization module and the edge resource awareness module respectively; the global coordination and control module is signal-connected to the request scheduling and concurrency management module, the KV cache pool management module, the dynamic batch processing and parallel inference scheduling module, the inference engine optimization module, and the edge resource awareness module respectively.

[0008] The request scheduling and concurrency management module is configured to receive multiple user inference requests from outside the edge device and to perform queue management and priority sorting on the multiple user inference requests. The module is internally configured with a request parsing unit and a task dispatching unit. The request parsing unit is responsible for extracting the request identifier, input sequence length, expected service quality level and preset timeout threshold from each inference request. The task dispatching unit sorts the parsed inference requests according to a preset priority algorithm and distributes the sorted inference requests to the KV cache pool management module in order.

[0009] The KV cache pool management module is configured to establish and maintain a Key-Value vector cache system covering the historical context of all user inference requests. This module is internally configured with a multi-level cache management unit, a cache allocation unit, and a cache reclamation unit. The multi-level cache management unit divides the KV cache into three levels: hot cache, warm cache, and cold cache. The hot cache uses high-bandwidth memory to store recently generated high-frequency access KV vectors, the warm cache uses double data rate synchronous dynamic random access memory to store KV vectors with medium access frequency, and the cold cache uses non-volatile memory to store historically accumulated low-frequency access KV vectors. The cache allocation unit dynamically allocates storage space for the hot or warm cache based on the length of the input sequence of the new inference request. The cache reclamation unit adopts a reclamation strategy based on reference counting. When the reference count of a KV vector reaches zero, a data migration or eviction operation is triggered in the cold cache.

[0010] The dynamic batch processing and parallel inference scheduling module is configured to identify key-value vector sharing relationships among multiple concurrent inference requests, and dynamically incorporate inference requests with dependency decoupling conditions into the same batch processing task based on the sharing relationships. This module is internally configured with a sharing relationship analysis unit, a batch processing grouping unit, and a parallel execution scheduling unit. The sharing relationship analysis unit determines whether there is a shareable key-value vector interval between the current inference request and the historical inference request by comparing the prefix of the input sequence of each inference request with the key-value identifier of the existing key-value vector in the cache pool. The batch processing grouping unit aggregates multiple inference requests that meet the parallel conditions into a batch processing task according to the sharing relationship analysis results, and generates a corresponding fusion attention mask matrix. The parallel execution scheduling unit distributes the batch processing task to the inference engine optimization module for execution.

[0011] The inference engine optimization module is configured to perform multi-request fusion inference computation after receiving batch processing tasks. This module is internally configured with a fusion attention computation unit, a computation graph optimization unit, and a result separation unit. The fusion attention computation unit merges the attention computation of multiple inference requests into a single matrix operation based on the fusion attention mask matrix generated by the batch processing queuing unit. In the specific execution process, this unit first concatenates the Query vectors of each inference request in the batch dimension to obtain a batch Query matrix, copies and expands the shared KV vectors in the batch dimension to obtain a batch KV matrix, then performs batch matrix dot product operation to obtain a fusion attention score matrix, and finally performs mask filtering on the fusion attention score matrix through the fusion attention mask matrix to separate the independent attention output of each inference request. The computation graph optimization unit performs operator fusion optimization on the result graph output by the fusion attention computation unit, merging adjacent matrix operation nodes into a single fusion operator to reduce kernel call overhead. The result separation unit splits the fusion inference result into independent inference outputs based on the identifier of each inference request and sends them back to the corresponding requester.

[0012] The edge resource sensing module is configured to monitor the computing power resource status and memory usage status of edge devices in real time. The module is equipped with a computing power monitoring unit and a memory monitoring unit. The computing power monitoring unit collects the real-time utilization rate, current frequency and active core count of the central processing unit and neural network processing unit through hardware performance counters. The memory monitoring unit collects the total system memory capacity, allocated memory amount, available memory amount and memory bandwidth utilization rate in real time. The output end of the edge resource sensing module is connected to the global coordination and control module, and sends the real-time collected computing power data and memory data to the global coordination and control module.

[0013] The global coordination and control module is configured to aggregate status information from various functional modules and generate collaborative scheduling instructions. Internally, this module includes a status aggregation unit, a strategy calculation unit, and an instruction issuance unit. The status aggregation unit receives request queue length and priority distribution information from the request scheduling and concurrency management module, cache occupancy rates and eviction event information at all levels from the KV cache pool management module, and computing power utilization and memory bandwidth occupancy data from the edge resource awareness module. The strategy calculation unit calculates the current optimal batch processing strategy and cache eviction strategy based on the aggregated multi-dimensional status information, generating a scheduling parameter set including the upper limit of batch processing task size, KV cache migration threshold, and inference task priority adjustment coefficient. The instruction issuance unit sends the scheduling parameter set to the KV cache pool management module, the dynamic batch processing and parallel inference scheduling module, and the inference engine optimization module to perform dynamic parameter adjustment.

[0014] Preferably, in the request scheduling and concurrency management module, the priority algorithm adopts a weighted fair queue scheduling strategy. This strategy assigns an initial weight value to each user inference request. The weight value is adjusted according to the service quality level of the request. The weight value of a request with a high service quality level is multiplied by a priority coefficient. The adjusted weight value is used to determine the ordering position of the request in the waiting queue. Requests within the same service quality level are ordered according to the first-come, first-served principle.

[0015] Preferably, in the KV cache pool management module, the hot cache storage capacity of the multi-level cache management unit is set to 10% to 20% of the total cache capacity, the warm cache storage capacity is set to 30% to 50% of the total cache capacity, and the cold cache storage capacity is set to 40% to 60% of the total cache capacity. When the hot cache occupancy rate reaches a preset threshold of 80%, a cache pre-migration operation is triggered to migrate the low-frequency access KV vectors in the hot cache to the warm cache.

[0016] Preferably, in the KV cache pool management module, when the cache allocation unit allocates cache space for a new inference request, it first checks whether the length of the input sequence exceeds the single allocation limit of the hot cache. If it does, the excess part is allocated to the warm cache. If the warm cache space is insufficient, the space is released by the least frequent eviction algorithm. During the eviction process, KV vectors with a reference count of zero are migrated to the cold cache first.

[0017] Preferably, in the dynamic batch processing and parallel inference scheduling module, the comparison logic of the shared relationship analysis unit adopts a prefix matching algorithm. This algorithm matches the prefix of the input sequence of the new inference request with the complete sequence of each historical request in the cache pool one by one. When the matching length is greater than or equal to the preset sharing threshold, it is determined that there is a shareable KV vector interval. The sharing threshold is set to a length of 32 to 128 tokens.

[0018] Preferably, in the dynamic batch processing and parallel inference scheduling module, the process of generating the fusion attention mask matrix by the batch processing grouping unit includes the following steps: first, assigning a unique batch identifier and in-sequence position index to each inference request in the batch processing task; then, setting the allowed attention elements of the mask matrix to 1 and the prohibited attention elements to 0 according to the KV sharing relationship between each request; and finally, converting the fusion attention mask matrix into a sparse matrix storage format to reduce memory usage.

[0019] Preferably, in the inference engine optimization module, the batch matrix dot product operation performed by the fusion attention calculation unit adopts a block calculation strategy, dividing the batch query matrix and batch KV matrix into multiple sub-blocks along the batch dimension. Each sub-block performs the matrix dot product operation in parallel in an independent calculation thread. The block size is adaptively adjusted according to the cache capacity of the neural network processing unit of the edge device. When the cache capacity is greater than or equal to 4 megabytes, a block size of 1024 multiplied by 512 is used, and when the cache capacity is less than 4 megabytes, a block size of 512 multiplied by 256 is used.

[0020] Preferably, in the inference engine optimization module, the operator fusion optimization of the computation graph optimization unit includes fusion of batch matrix dot product operations and their subsequent normalization operations into a single fusion operator, and vectorization and merging of the head calculations of multi-head attention when the dimensionality compatibility condition is met, thereby reducing the number of kernel calls by no less than 30% after fusion.

[0021] Preferably, in the edge resource sensing module, the sampling frequency of the computing power monitoring unit is set to 10 to 50 times per second, the sampling frequency of the memory monitoring unit is set to 50 to 200 times per second, and the edge resource sensing module sends the collected data to the global coordination control module after exponential moving average filtering. The filtering coefficient is set to 0.7 to 0.9 to balance data real-time performance and noise suppression effect.

[0022] Preferably, in the global coordination and control module, the strategy calculation unit uses a model predictive control algorithm to calculate the optimal scheduling strategy. This algorithm takes maximizing system throughput as the optimization objective and the computing power utilization and memory bandwidth occupancy of edge devices as constraints. It solves the optimal scheduling parameter set at the current moment through rolling time-domain optimization, and the time-domain prediction window is set to 5 to 10 inference task cycles.

[0023] Preferably, in the global coordination and control module, the instruction issuing unit and each functional module adopt a shared memory communication mechanism. Each functional module writes its own status data to a preset shared memory address. The global coordination and control module reads the status data from the shared memory address, performs calculations, and writes the scheduling parameter set into the corresponding address area of ​​the shared memory. Each functional module reads the scheduling parameter set from the shared memory address and performs parameter updates. The shared memory adopts a lock-free circular buffer design to support high-concurrency read and write access.

[0024] Compared with the prior art, the present invention has the following beneficial effects: 1. The large-model dynamic scheduling and inference optimization system for edge computing described in this invention establishes a Key-Value vector caching system covering the historical context of all user inference requests by introducing a KV cache pool management module. The KV vectors that can be shared between inference requests are centrally stored and managed in the cache pool. This design breaks through the architectural limitation of each inference request independently maintaining a complete KV context in traditional autoregressive inference, realizes cross-request KV vector reuse, solves the problem of wasted computing power caused by repeated calculations when edge devices process multiple concurrent inference requests, and effectively improves the overall inference throughput of edge devices in multi-user scenarios.

[0025] 2. The large-scale model dynamic scheduling and inference optimization system for edge computing described in this invention identifies the KV vector sharing relationship between multiple concurrent inference requests through a dynamic batch processing and parallel inference scheduling module and performs fused attention calculation. This design merges the originally scattered multiple attention calculations into a single batch matrix operation while maintaining the correctness of autoregressive inference. This technical feature enables the multi-core computing power of edge devices to be fully utilized in a unified batch computing process, transforming the serial inference paradigm into a parallelizable batch inference paradigm. It fundamentally breaks through the computing power utilization bottleneck caused by token-by-token generation and achieves theoretically guaranteed efficient parallel inference.

[0026] 3. The large-model dynamic scheduling and inference optimization system for edge computing described in this invention significantly reduces the memory access overhead and kernel call overhead of fusion attention computation through operator fusion optimization and block computation strategy of the inference engine optimization module. This technical feature enables batch inference computation to be executed on the neural network processing unit of the edge device with near-theoretical peak efficiency, effectively shortens the end-to-end latency of a single batch inference, and improves the system's response speed in real-time inference scenarios.

[0027] 4. The large-model dynamic scheduling and inference optimization system for edge computing described in this invention monitors the computing power and memory usage status of edge devices in real time through an edge resource awareness module, and dynamically adjusts batch processing and cache management strategies based on real-time status information through a global coordination and control module. This design enables the system to adaptively adjust the scheduling parameters of inference tasks according to the actual load of edge devices, maintaining stable inference performance even in scenarios with limited computing power and memory, effectively solving the adaptability problem caused by the heterogeneity of edge device hardware resources. Attached Figure Description

[0028] The invention will now be further described with reference to the accompanying drawings.

[0029] Figure 1 This is a structural block diagram of a large-scale model dynamic scheduling and inference optimization system for edge computing in this invention; Figure 2 This is a schematic diagram illustrating the collaborative scheduling and interaction relationship between the global coordination and control module and the edge resource perception module in this invention. Detailed Implementation

[0030] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0031] Example 1: This embodiment applies to a large language model inference service deployment environment in an edge computing scenario. In this application environment, multiple edge devices simultaneously receive natural language inference requests from users, including various task types such as intelligent dialogue, text generation, and semantic analysis. The hardware resources of edge devices are limited by physical size and power consumption constraints. The number of CPU cores, neural network processing unit computing power, and system memory bandwidth of these devices are far lower than those of cloud data centers. Furthermore, the hardware configurations of different edge devices exhibit high heterogeneity. The system designed in this embodiment achieves high-throughput parallel inference services for large language models under the condition of limited computing resources on edge devices through dynamic scheduling and inference optimization mechanisms. The system includes six core functional modules: request scheduling and concurrency management module, KV cache pool management module, dynamic batch processing and parallel inference scheduling module, inference engine optimization module, edge resource awareness module, and global coordination and control module.

[0032] like Figure 1 and Figure 2As shown, the request scheduling and concurrency management module is located at the front end of the entire system, undertaking the core task of receiving and managing multiple user inference requests from external edge devices. This module internally comprises two functional components: a request parsing unit and a task dispatching unit. The input interface of the request parsing unit is connected to the network communication interface of the edge device, listening in real time and receiving inference request data packets sent from external user terminals. This parsing unit has a built-in protocol parser that unpacks each inference request data packet, extracting four key parameters: request identifier, input sequence length, expected quality of service level, and preset timeout threshold. The identifier uses a globally unique string encoding format, with a length of 32 hexadecimal characters, and is used to track the processing status of the request throughout the entire inference lifecycle. The length of the input sequence is measured in units of tokens, ranging from 1 to 32768 tokens. The expected service quality level is divided into three levels: Level 1 corresponds to real-time interaction scenarios requiring an end-to-end latency of no more than 500 milliseconds; Level 2 corresponds to standard response scenarios requiring an end-to-end latency of no more than 2 seconds; and Level 3 corresponds to background processing scenarios allowing an end-to-end latency of no more than 30 seconds. The preset timeout threshold is set in milliseconds. When the inference request is in the queue... When the waiting time in the queue exceeds the threshold, the system triggers a timeout handling process. The task assignment unit has a built-in priority sorting algorithm and a task dispatch scheduler. The priority sorting algorithm adopts a weighted fair queue scheduling strategy. This strategy assigns an initial weight value to each pending inference request. The initial weight value is normalized according to the length of the input sequence of the request. The longer the input sequence, the higher the base weight is given to balance the waiting time of long and short requests. The weight value is then adjusted according to the service quality level of the request. The weight value of a high service quality level request is multiplied by the corresponding priority coefficient. The priority coefficient for the first level of service quality is set to 3.0, the priority coefficient for the second level of service quality is set to 1.5, and the priority coefficient for the third level of service quality is set to 1.0. The adjusted weight value is used to determine the order position of the request in the waiting queue. The larger the weight value, the higher the position in the queue. Requests within the same service quality level are sorted according to the first-come, first-served principle to ensure fairness among requests of the same level. The task dispatch scheduler takes the inference requests from the waiting queue in sequence according to the sorting result and sends the requests to the KV cache pool management module through the inter-module communication interface. The number of requests distributed each time is dynamically adjusted according to the current load status of the KV cache pool management module.

[0033] The KV cache pool management module is located downstream of the request scheduling and concurrency management module. Its input receives inference requests sent from the task dispatch unit, and its output is connected to the dynamic batch processing and parallel inference scheduling module. This module is internally configured with three functional components: a multi-level cache management unit, a cache allocation unit, and a cache reclamation unit. Together, they realize the establishment and maintenance of a Key-Value vector cache system that covers the entire historical context of user inference requests. The multi-level cache management unit divides the cache space into three levels: hot cache, warm cache, and cold cache, based on the access frequency and importance of KV vectors. The hot cache uses high-bandwidth memory as the storage medium. High-bandwidth memory is a high-density memory chip that uses through-silicon via (TSV) technology and 3D stacked packaging. Its single-chip bandwidth can reach more than 1 terabyte per second and is specifically used to store recently generated high-frequency access KV vectors. The storage capacity of the hot cache is set to 15% of the total cache capacity. In this embodiment, the total cache capacity is 30% of the edge device system memory, that is, the hot cache occupies 4% of the total system memory.The warm buffer, accounting for 5% to 6% of the total cache capacity, uses Double Data Rate Synchronous Dynamic Random Access Memory (DDRSM) as its storage medium. DDRSM is a conventional capacity memory chip with a lower unit storage cost than high-bandwidth memory, making it suitable for storing key-value vectors with moderate access frequency. The warm buffer's storage capacity is set at 40% of the total cache capacity. The cold buffer uses Non-Volatile Memory (NVMemory) as its storage medium. NVOCT maintains data integrity after power failure, making it suitable for storing historically accumulated low-frequency access key-value vectors. It can quickly restore the cache state after a power outage and restart of the edge device. The storage capacity of the hot cache area is set to 45% of the total cache capacity. The multi-level cache management unit continuously monitors the occupancy rate of each cache layer. When the hot cache occupancy rate reaches a preset threshold of 80%, a cache pre-migration operation is triggered, migrating the KV vectors with the lowest access frequency in the hot cache to the warm cache to free up hot cache space. The cache allocation unit is responsible for allocating cache space for newly arrived inference requests. After receiving an inference request from the task dispatch unit, this unit first checks whether the length of the input sequence of the request exceeds the single allocation limit of the hot cache. The single allocation limit of the hot cache is set to... The KV vector storage space corresponding to 2048 tokens is allocated entirely to the hot cache if the input sequence length does not exceed the single allocation limit. If it does, the excess space is allocated to the warm cache. If the available space in the warm cache is insufficient, a least-usage-frequency eviction algorithm is triggered. This algorithm scans the historical access records of all KV vectors in the warm cache, prioritizing the eviction of KV vectors with the fewest accesses and zero reference counts. During the eviction process, the KV vectors to be evicted are migrated to the cold cache to retain data. The cache reclamation unit adopts a eviction strategy based on reference counting. The cache maintains a reference count variable for each key-value vector. When a new inference request reuses an existing key-value vector, the reference count is incremented by 1. When an inference request referencing a key-value vector completes processing and releases resources, the reference count is decremented by 1. When the reference count of a key-value vector reaches zero and the vector is located in the hot cache or warm cache, the cache reclamation unit triggers a data migration operation in the cold cache, migrating the key-value vector to permanent storage in the cold cache. When the cold cache space is insufficient, a data eviction operation is performed, evicting the earliest key-value vector to enter the cold cache according to the first-in, first-out (FIFO) principle.

[0034] The dynamic batch processing and parallel inference scheduling module is located downstream of the KV cache pool management module. Its input receives inference requests from the allocated cache space of the KV cache pool management module, and its output is connected to the inference engine optimization module and the edge resource awareness module, respectively. This module internally comprises three functional components: a shared relationship analysis unit, a batch processing queuing unit, and a parallel execution scheduling unit. The shared relationship analysis unit is responsible for identifying the KV vector sharing relationship between the currently arriving inference request and existing inference requests in the cache pool. This unit determines whether there is a shareable KV vector interval by comparing the prefix of the new inference request's input sequence with the complete sequences of each historical request in the cache pool. The algorithm employs a prefix matching algorithm. During execution, the shared relation analysis unit matches the first n terms of the input sequence of a new inference request with the complete sequences of each historical request in the cache pool. When the matching length is greater than or equal to a preset sharing threshold, a shareable key-value vector interval is determined to exist. In this embodiment, the sharing threshold is set to a length of 64 terms, which can be dynamically adjusted according to the computing power of the edge device. When the shared relation analysis unit determines that a shareable key-value vector interval exists, it generates a shared relation descriptor containing the sharing start position, sharing length, and sharing target request identifier, and sends it to the batch processing queuing unit. The batch processing queuing unit then processes the shared relation according to the specified length. The relational analysis results aggregate multiple inference requests that meet the parallel conditions into batch processing tasks. The batch processing task generation process includes the following steps: First, each inference request in the batch processing task is assigned a unique batch identifier and an intra-sequence position index. The batch identifier is an incrementing integer starting from 1, and the intra-sequence position index records the position of each term in the request sequence. Second, a fusion attention mask matrix is ​​set according to the KV sharing relationship between the requests. When there is a shareable KV vector interval between two requests, the element at the corresponding position in the mask matrix is ​​set to 1 to indicate that attention computation is allowed to be executed across requests. When there is no sharing relationship between two requests, the element at the corresponding position is set to 1. Setting it to 0 indicates that attention computation is disabled across requests. The third step converts the fused attention mask matrix into a sparse matrix storage format. The sparse matrix storage format only records the position and value of the elements with a value of 1 in the mask matrix, reducing memory usage. The parallel execution scheduling unit sends the aggregated batch processing tasks to the inference engine optimization module for execution. During the scheduling process, the parallel execution scheduling unit dynamically adjusts the size of each batch processing task based on the current computing power resource status fed back by the edge resource awareness module. When computing power resources are sufficient, the number of batch processing task requests is increased to improve throughput. When computing power resources are scarce, the number of batch processing task requests is reduced to ensure the response latency of individual requests.

[0035] The inference engine optimization module is located downstream of the dynamic batch processing and parallel inference scheduling module. Its input receives batch processing tasks from the parallel execution scheduling unit, and its output connects to the request scheduling and concurrency management module to return inference results. This module internally comprises three functional components: a fusion attention computation unit, a computation graph optimization unit, and a result separation unit. The fusion attention computation unit is responsible for performing multi-request fusion inference computation after receiving batch processing tasks. Its core function is to merge the originally scattered multiple attention computations into a single batch matrix operation to fully utilize the hardware's parallel capabilities. The specific execution process includes the following steps: First, extract the Query vector of each inference request from the batch processing task, and then... The first step involves concatenating the batch query matrix along the batch dimension. The batch query matrix's dimension is calculated by multiplying the batch size by the sequence length and the hidden dimension. The second step extracts the key-value (KV) vectors of each inference request from the batch processing task. For multiple requests with shared relationships, their shared KV vectors are copied and expanded along the batch dimension. For requests without shared relationships, the original KV vectors are retained, resulting in a batch KV matrix. The third step performs a batch matrix dot product operation, multiplying the batch query matrix and the batch KV matrix to obtain the fused attention score matrix. The fused attention score matrix's dimension is calculated by multiplying the batch size by the sequence length and the sequence length. The fourth step uses the fused attention mask matrix generated by the batch processing grouping unit. A masking filter is applied to the fused attention score matrix, setting the attention score at the position corresponding to an element of 0 in the mask matrix to zero, thereby separating the independent attention output of each inference request. The batch matrix dot product operation performed by the fused attention computation unit adopts a block computation strategy. This strategy divides the batch query matrix and batch key-value matrix into multiple sub-blocks along the batch dimension. Each sub-block performs the matrix dot product operation in parallel in an independent computation thread. The block size is adaptively adjusted according to the cache capacity of the neural network processing unit of the edge device. When the final-level cache capacity of the neural network processing unit is greater than or equal to 4 megabytes, a block size of 1024 multiplied by 512 is used; when the final-level cache capacity is less than 4 megabytes, a block size of 512 multiplied by 256 is used. The computation graph optimization unit optimizes the computation result graph output by the fusion attention computation unit by dividing the block size into chunks. This optimization has two levels: First, it merges the batch matrix dot product operation with its subsequent Softmax normalization operation into a single fusion operator, reducing the number of intermediate result writes to and reads from GPU memory. Second, it vectorizes and merges the computations of each attention head in the multi-head attention mechanism when the dimensionality compatibility condition is met, merging the independent matrix operations of multiple attention heads into a single batch matrix operation. The fusion effect of the computation graph optimization unit aims to reduce the number of kernel calls by at least 30%. The result separation unit splits the fused inference result into independent inference outputs based on the unique identifier of each inference request.This unit extracts the output fragment corresponding to each request from the fused inference results according to the batch identifier and sequence position index. It then assembles the output fragment with the request identifier into an independent result data packet and sends it back to the corresponding requester via the communication interface.

[0036] The edge resource awareness module runs independently in the system background, continuously monitoring the hardware resource usage status of edge devices and reporting to the global coordination and control module. This module internally comprises two functional components: a computing power monitoring unit and a memory monitoring unit. The computing power monitoring unit collects real-time operating parameters of the central processing unit (CPU) and neural network processing unit (NN unit) through a hardware performance counter interface. The hardware performance counter is a dedicated hardware module built into modern processor chips, capable of recording various hardware events during processor instruction execution with extremely low overhead. The parameters collected by the computing power monitoring unit include the overall utilization rate of the CPU, the overall utilization rate of the NN unit, the current operating frequency of the CPU and NN unit, the number of active cores in the CPU, and the number of active computing units in the NN unit. The computing power monitoring unit's sampling frequency is set to 20 times per second, meaning a complete status sampling is completed every 50 milliseconds. The memory monitoring unit collects real-time data within the system. The memory monitoring unit reads the total system memory capacity, allocated system memory, available system memory, and real-time bandwidth utilization of the memory controller through the memory management interface provided by the operating system. The memory monitoring unit's sampling frequency is set to 100 times per second, that is, a status sampling is completed every 10 milliseconds. This is higher than the sampling frequency of the computing power monitoring unit because changes in memory resource status are more sensitive to the impact on inference performance. Before sending the collected data to the global coordination and control module, the edge resource perception module performs preprocessing operations. The preprocessing operation uses an exponential moving average filtering algorithm to smooth the original collected data to suppress high-frequency noise components. In this embodiment, the filtering coefficient is set to 0.8, that is, 80% of the current filtered output value comes from the previous filtering result and 20% comes from the current original collected value. The data after exponential moving average filtering effectively reduces the interference of sensor noise and instantaneous fluctuations on control decisions while maintaining the real-time nature of the data.

[0037] The global coordination and control module, as the system's central scheduling unit, establishes signal connections with the request scheduling and concurrency management module, the KV cache pool management module, the dynamic batch processing and parallel inference scheduling module, the inference engine optimization module, and the edge resource awareness module. This enables centralized monitoring and coordinated scheduling of all functional modules across the system. Internally, this module comprises three functional components: a state aggregation unit, a policy calculation unit, and an instruction issuance unit. The state aggregation unit continuously receives and aggregates state information from each module through data interfaces reserved by the functional modules. The input data received by the state aggregation unit includes: from the request scheduling and concurrency management module, the current waiting queue length, the priority distribution histogram of each request in the waiting queue, and the request arrival rate within the past time window; from the KV cache pool management module, the hot cache utilization rate, warm cache utilization rate, cold cache utilization rate, the timestamp of the most recent cache eviction event, and the timestamp of the most recent cache migration event; and from the edge resource awareness module, the real-time utilization rate of the central processing unit, the real-time utilization rate of the neural network processing unit, the real-time memory bandwidth utilization rate, and the real-time available memory space. The system performs unified format conversion and time alignment on the aforementioned multi-source heterogeneous data and stores it in a local state database for use by the strategy calculation unit. The strategy calculation unit performs scheduling strategy optimization calculations based on the aggregated multi-dimensional state information. The optimization algorithm used by the strategy calculation unit is the model predictive control algorithm. This algorithm takes maximizing system throughput as the optimization objective and uses the upper limit of computing power utilization and memory bandwidth occupancy of edge devices as constraints. It solves for the optimal scheduling parameter set at the current moment through a rolling time-domain optimization method. The rolling time-domain window of the model predictive control algorithm is set to 8 inference task cycles. The duration of each inference task cycle is estimated based on historical statistical data as the average processing time required for each inference request from entry to completion. The output of the strategy calculation unit is a scheduling parameter set, which contains three core parameters: the upper limit of batch processing task size is set as the upper limit of the number of requests that can be processed in parallel. This parameter is dynamically adjusted according to computing power monitoring data to avoid computing power overload; the KV cache migration threshold is set as the condition value for triggering cache level migration operations. This parameter is dynamically adjusted according to the cache occupancy data of each layer to balance cache hit rate and storage cost.The inference task priority adjustment coefficient is set as a correction factor for real-time adjustment of the weight values ​​of each request. This parameter is dynamically calculated based on the waiting queue status and request timeout risk to optimize the overall response time. The instruction issuing unit is responsible for sending the scheduling parameter set generated by the strategy calculation unit to each execution module. Efficient data transmission is achieved between the instruction issuing unit and each functional module using a shared memory communication mechanism. The shared memory area is divided into multiple independent address segments, with each functional module corresponding to a dedicated write address segment and read address segment. The instruction issuing unit writes the scheduling parameter set into the corresponding address segment of the shared memory according to a preset memory layout format. Each functional module retrieves the latest scheduling parameters from its respective read address segment and performs parameter updates. The shared memory communication mechanism adopts a lock-free circular buffer design. The read and write pointers of the circular buffer are synchronized through atomic operations, supporting simultaneous read and write access by multiple functional modules without data contention.

[0038] The workflow of the large-model dynamic scheduling and inference optimization system for edge computing involved in this embodiment is as follows: Inference request reception and parsing: Multiple external user terminals send natural language inference requests to the system through the network interface of edge devices. The inference requests are transmitted to the request scheduling and concurrency management module in the form of data packets conforming to a preset protocol format. The network listener built into the request parsing unit continuously listens for incoming data on the network interface. After capturing the incoming request data packets, it performs protocol parsing processing. The protocol parsing processing first verifies the structural integrity of the data packet, checking whether the header field, length field, and check field of the data packet are correct. After the verification is passed, the parsing unit extracts four core parameters from the data packet payload: request identifier field, input sequence field, expected service quality level field, and preset timeout threshold field. The input sequence field exists in the form of a word sequence. The parsing unit performs lexical analysis and encoding conversion on the word sequence to generate an integer identifier sequence that the model can process. The parsing unit assembles the extracted parameter information into an internal request object and puts it at the tail of the waiting queue. At the same time, it updates the current length of the waiting queue to the number of existing requests in the waiting queue plus 1. After completing the parameter extraction, the request parsing unit immediately releases the memory occupied by the original data packet to avoid memory leaks.

[0039] Priority sorting and task distribution: The task distribution unit continuously checks if the waiting queue is empty. When the waiting queue is not empty, the priority sorting and distribution process is triggered. The task distribution unit traverses all request objects in the waiting queue, reading three fields for each request: input sequence length, expected service quality level, and timestamp of entry into the queue. For each request, firstly, a normalized base weight is calculated based on the input sequence length. The calculation formula is: divide the input sequence length by the average input sequence length of all waiting requests to obtain the normalization coefficient, multiply this coefficient by the base weight value of 1.0 to obtain the base weight, and then find the corresponding priority sorting and distribution process based on the expected service quality level. First, a coefficient is multiplied by the base weight to obtain the corrected weight. The corrected weight of a high service quality level request is significantly higher than that of a low service quality level request. Finally, the requests are sorted in descending order according to their corrected weights to generate a new sorting sequence. After sorting, the task dispatching unit retrieves the request objects from the sorting sequence one by one. After each retrieval, the current load status of the KV cache pool management module is checked. If the load has not reached the upper limit, the request object is sent to the KV cache pool management module through the inter-module communication interface. If the load has reached the upper limit, the dispatching is paused until the load decreases. When dispatching each request, the task dispatching unit also sets the start timestamp of the request for subsequent calculation of waiting delay.

[0040] KV cache space allocation and management: After receiving an inference request from the task dispatch unit, the KV cache pool management module executes the cache space allocation process. The cache allocation unit first extracts the input sequence length parameter from the inference request and determines whether the length exceeds the single allocation limit of 2048 tokens in the hot cache. If it does not exceed this limit, the KV vector storage space corresponding to the request is allocated entirely in the hot cache. The allocation size is equal to the input sequence length multiplied by the number of bytes of storage for each KV vector corresponding to each token. If the length exceeds this limit, the storage space corresponding to 2048 tokens is first allocated in the hot cache, and the remaining part is allocated in the warm cache. During the allocation process, the cache allocation unit queries the warm cache. If the available space in the warm buffer is insufficient to accommodate the excess data, a least-used-frequency eviction algorithm is triggered. The eviction algorithm scans the access count records of all key-value vectors in the warm buffer, constructs a min-heap structure with access count as the key and key-value vector identifier as the value, and then pops the top element of the heap, which is the key-value vector with the fewest accesses. It checks whether its reference count is zero. If the reference count is zero, it is moved to the cold buffer and the original storage space is released. This process is repeated until the available space in the warm buffer is sufficient to accommodate the data to be allocated. After the cache allocation unit completes the space allocation, it updates the occupancy statistics of each cache layer and returns an allocation completion confirmation message to the requester.

[0041] The KV shared relation analysis and batch processing module, along with the dynamic batch processing and parallel inference scheduling module, initiates the shared relation analysis process upon receiving a cached inference request. The shared relation analysis unit extracts the prefix of the input sequence from the received inference request. The prefix length is a preset shared threshold of 64 words. The shared relation analysis unit traverses all stored historical inference request records in the KV cache pool, performing word-by-word matching between the new request's 64-word prefix and the complete input sequence of each historical request. The matching process uses an efficient prefix tree data structure to accelerate the query. Upon successful matching, the matching length and the identifier of the matched historical request are recorded. When the matching length reaches or exceeds 64 words, it is determined that there is a shareable KV vector interval between the new request and the corresponding historical request. The shared interval length is 64 words, and the starting position of the shared interval is the first word position of the historical request's input sequence. The shared relation analysis unit assembles all successfully matched shared relation records into a shared relation descriptor list and sends it to the batch processing queue. The batch processing grouping unit aggregates multiple inference requests with shared relationships into batch processing tasks based on the list of shared relationship descriptors. The number of requests in a batch processing task ranges from 2 to the upper limit of the batch processing task size. The batch processing grouping unit assigns a unique intra-batch identifier to each request in the batch processing task, and the intra-batch identifier starts from 0 and increments. At the same time, the batch processing grouping unit generates a fusion attention mask matrix based on the shared relationships. The number of rows and columns of the mask matrix is ​​the sum of the lengths of the request sequences in the batch processing task. The elements in the matrix are set according to the following rules: when the j-th word position of the i-th request has a KV shared relationship with the l-th word position of the k-th request, the corresponding element in the matrix is ​​set to 1. When there is no shared relationship, it is set according to the standard causal attention rule, that is, the j-th word of the i-th request can only pay attention to historical words in its own sequence whose position does not exceed j and shared historical words. After the fusion attention mask matrix is ​​generated, it is sparsified, and only the position and value of the element with a value of 1 in the matrix are retained to reduce storage overhead.

[0042] The fusion inference computation and result are separated. After receiving the batch processing task from the batch processing array unit, the inference engine optimization module starts the fusion inference computation process. The fusion attention computation unit first extracts the Query vector, Key vector, and Value vector of each inference request from the batch processing task. The dimension of the Query vector of each request is the sequence length multiplied by the hidden dimension. In this embodiment, the three levels of hidden dimensions are 256, 512, and 768 respectively to adapt to language models of different sizes. The fusion attention computation unit concatenates the Query vectors of each request along the batch dimension to obtain a batch Query matrix. The dimension of the batch Query matrix is ​​the batch size multiplied by the sequence length multiplied by the hidden dimension. For multiple requests with KV sharing relationships, the fusion attention computation unit copies and expands the shared Key vector and Value vector along the batch dimension to make the KV matrix dimension participating in the fusion attention computation consistent with the Query matrix dimension. In the batch matrix dot product operation stage, the batch Query matrix and the batch KV matrix are multiplied by matrix multiplication. This operation is performed along the hidden dimension to obtain the fusion attention score matrix. The dimension of the fusion attention score matrix is ​​the batch size multiplied by the sequence length multiplied by the sequence length, where each element represents a certain... The attention score of a query vector at a certain position to a key-value vector at another position is requested. The attention calculation unit performs a block matrix operation strategy, dividing the batch query matrix and batch key-value matrix into multiple sub-blocks along the batch dimension. The query matrix dimension of each sub-block is the block size multiplied by the sequence length multiplied by the hidden dimension, and the key-value matrix dimension of each sub-block is the block size multiplied by the sequence length multiplied by the hidden dimension. The block size is dynamically determined according to the last-level cache capacity of the neural network processing unit. In this embodiment, if the last-level cache capacity is detected to be greater than or equal to 4 megabytes, the block size is set to 1024 multiplied by 512. If the capacity is less than 4 megabytes, the block size is set to 512 multiplied by 256. Each sub-block is assigned to an independent computing thread to perform matrix dot product operations. The results of multiple threads are finally converged and merged. The advantage of the block computing strategy is that the intermediate computing results of each sub-block can be fully contained in the final cache of the neural network processing unit, avoiding frequent access to external main memory with limited bandwidth, thereby significantly improving computing efficiency. After the fused attention score matrix is ​​generated, it is multiplied element-wise with the fused attention mask matrix to achieve mask filtering. The filtered result is normalized by Softmax to obtain the final attention weight distribution of each request.

[0043] The computation graph optimization unit analyzes and optimizes the computation graph generated by the above-mentioned fusion attention computation process. The computation graph optimization unit first identifies continuous matrix operation nodes and normalization operation nodes in the computation graph. When it is detected that there are no other operations between the batch matrix dot product operation node and its subsequent Softmax operation node, the two nodes are merged into a single fusion operator. When the fusion operator is executed, the matrix dot product operation is first completed to obtain the intermediate result matrix. Then, the intermediate result matrix is ​​directly passed to the Softmax function for normalization without writing to temporary video memory. The computation graph optimization unit also detects the computational dependencies between attention heads in the multi-head attention mechanism. When the dimensions of the Query, Key, and Value projection matrices of each attention head are compatible, the independent matrix operations of multiple attention heads are merged into a single batch matrix operation. After the computation graph optimization is completed, an optimized computation graph description file is generated. This file describes the execution order, memory layout, and thread configuration information of the fusion operator.

[0044] The result separation unit receives the optimized fusion inference result and performs a result splitting operation. The result separation unit extracts the corresponding output fragments from the fusion inference result according to the batch identifier of the batch processing task and the sequence position index of each request. The extraction process first divides the fusion result into independent results for each request along the batch dimension according to the batch size, and then divides the independent results of each request into outputs for each word position along the sequence dimension. The result separation unit assembles the output fragment of each request with the original request identifier of the request into an independent result data packet. The result data packet is sent back to the request scheduling and concurrency management module through the communication interface. The request scheduling and concurrency management module forwards the result to the corresponding user terminal according to the request identifier.

[0045] Resource status monitoring and collaborative scheduling: The edge resource awareness module runs independently in the system background, periodically executing the resource status monitoring process. The computing power monitoring unit initiates a collection request through the hardware performance counter interface to obtain real-time utilization data of the central processing unit and the neural network processing unit. The data returned by the hardware performance counter includes the ratio of the number of cycles of useful instructions executed by the central processing unit to the total number of cycles within the collection time window (i.e., utilization percentage), the ratio of the number of cycles of matrix operation instructions executed by the neural network processing unit to the total number of cycles within the collection time window (i.e., utilization percentage), the stable value of the processor's current operating frequency, and the number of active computing cores. The computing power monitoring unit assembles the above data into a computing power status report and sends it to the central processing unit of the edge resource awareness module. The memory monitoring unit connects through the operating system kernel. The unit queries the current memory status and obtains the total system memory capacity, allocated system memory, available system memory, and the current bandwidth usage percentage of the memory controller. The memory monitoring unit assembles the above data into a memory status report and sends it to the central processing unit of the edge resource perception module. The edge resource perception module performs exponential moving average filtering on the computing power status report and the memory status report. The processing is as follows: maintain the filtered output value of the previous moment, and calculate the current filtered output value by weighting the current raw acquisition value and the previous filtered output value according to the weight ratio. The filtering coefficient is 0.8, that is, 80% of the current filtered output value comes from the previous filtered output value and 20% comes from the current raw acquisition value. The filtered status data is sent to the status aggregation unit of the global coordination and control module through the communication interface.

[0046] The state aggregation unit of the global coordination and control module continuously receives state data from various functional modules and executes the state aggregation process. The state aggregation unit allocates an independent receiving buffer for each functional module. Newly arrived state data is first written to the receiving buffer of the corresponding functional module. When at least one complete set of state data is received from all functional modules, the state aggregation unit triggers the policy calculation process. The policy calculation unit reads the aggregated multi-dimensional state information from the local state database, including the current waiting queue length, the recent request arrival rate, hot buffer occupancy, warm buffer occupancy, cold buffer occupancy, central processing unit utilization, neural network processing unit utilization, and memory bandwidth utilization. The policy calculation unit executes the model predictive control algorithm to calculate the optimal scheduling parameters. The algorithm aims to maximize system throughput, i.e., maximize the number of inference requests completed per unit time, with constraints including central processing unit utilization not exceeding 95%, neural network processing unit utilization not exceeding 90%, and memory bandwidth utilization not exceeding 85%. The algorithm maintains a length of 8 inference requests. The task cycle uses a rolling prediction window, and the optimization problem is resolved in each control cycle. The optimization process first constructs the request arrival prediction sequence and resource availability prediction sequence from the current time to the end time of the prediction window. Then, an optimization model is constructed with maximizing the cumulative throughput within the prediction window as the objective function and the constraints as the constraint equations. Finally, a numerical optimization algorithm is called to solve the model to obtain three core scheduling parameters: the upper limit of batch processing task size, the KV cache migration threshold, and the inference task priority adjustment coefficient. The policy calculation unit writes the scheduling parameter set into the policy output section of shared memory. The instruction issuing unit synchronously notifies each functional module to read the latest scheduling parameters. After obtaining the scheduling parameters from the policy reading section of shared memory, each functional module executes the corresponding parameter update operation. The request scheduling and concurrency management module updates the sorting weight of each request according to the priority adjustment coefficient, the KV cache pool management module updates the condition value that triggers the migration operation according to the KV cache migration threshold, and the dynamic batch processing and parallel inference scheduling module adjusts the upper limit of the number of requests in each batch processing queue according to the upper limit of batch processing task size.

[0047] Example 2: This embodiment provides a variant design based on Embodiment 1, mainly optimizing and expanding the adaptive scheduling capability in scenarios with heterogeneous hardware resources of edge devices. In this variant design, the computing power monitoring unit of the edge resource perception module adds support for the graphics processing unit, expanding the support to collect the real-time utilization rate, memory bandwidth utilization rate, and number of active tensor cores of the graphics processing unit. The graphics processing unit serves as the main neural network acceleration hardware in some edge devices. This variant design enables it to adapt to edge computing devices equipped with graphics processing units. The sampling frequency of the computing power monitoring unit is adjusted to a configurable mode of 10 times per second to 50 times per second. Users can select an appropriate sampling frequency according to the specific computing power level and power consumption constraints of the edge device to balance monitoring accuracy and resource consumption.

[0048] Regarding cache management strategies, the KV cache pool management module in this variant design adds a dynamic cache capacity adjustment function. When the edge resource awareness module detects that the total system memory capacity is sufficient, it automatically expands the upper limit of the storage capacity of each cache layer. The upper limit of the hot cache layer can be expanded to 25% of the total cache capacity, and the upper limit of the warm cache layer can be expanded to 55% of the total cache capacity. When the system memory is tight, it automatically shrinks the lower limit of the storage capacity of each cache layer. The lower limit of the hot cache layer can be shrunk to 8% of the total cache capacity, and the lower limit of the warm cache layer can be shrunk to 25% of the total cache capacity. During the dynamic adjustment process, the multi-level cache management unit performs a smooth migration operation to migrate the excess data from the shrunk cache layer to other cache layers or perform an eviction operation.

[0049] Regarding batch processing strategies, this variant design adds prediction-based batch processing optimization to the dynamic batch processing and parallel inference scheduling modules. While performing prefix matching analysis, the shared relationship analysis unit uses a machine learning model to predict the possible expansion of the term range of the current request sequence. When it is predicted that multiple new requests are likely to share a relationship with historical requests in the current cache pool, the relevant historical requests are retained in the hot or warm cache in advance to improve the hit rate of subsequent batch processing. The batch processing queuing unit adds a dynamic batch processing window mechanism. The batch processing window duration can be configured from 100 milliseconds to 500 milliseconds. Within the window duration, inference requests that meet the parallel conditions are continuously collected and uniformly queued for fusion inference. This mechanism effectively improves the request aggregation degree of batch processing when the request arrival rate is low.

[0050] Example 3: This embodiment provides a variant design based on Embodiment 1, mainly focusing on optimizing the system's scalability in large-scale multi-user concurrent scenarios. In this variant design, the request scheduling and concurrency management module adds a multi-level queue scheduling architecture. Priority partition queues are added to the original single-level waiting queue, with each service quality level corresponding to an independent priority partition queue. The task dispatch unit first polls each partition queue in descending order of service quality level, and then distributes requests within the same partition queue according to the first-come, first-served principle. The multi-level queue scheduling architecture effectively reduces the probability of high-priority requests being blocked by low-priority long requests, and improves the end-to-end response speed of high service quality level requests.

[0051] Regarding the KV cache pool management module, this variant design upgrades the storage medium of the cold cache from non-volatile memory to a distributed storage node network. The distributed storage node network consists of multiple storage devices interconnected by the network. Each storage node is configured with an independent controller and management software to achieve distributed storage and redundant backup of data. The data migration operation of the cold cache distributes the KV vector and its associated metadata to multiple nodes in the distributed storage node network for storage. The failure of a single node will not cause data loss. When it is necessary to read the KV vector from the cold cache, the distributed storage controller queries the index information of multiple nodes in parallel and reads the data from the node with the fastest response.

[0052] Regarding the global coordination and control module, this variant design adds a multi-agent collaborative scheduling function. Multiple global coordination and control module instances are deployed in the system to form a coordination cluster. Each instance is responsible for managing a computing node partition of the edge device. The instances synchronize global state information through a consistency protocol. The scheduling decision of the coordination cluster is jointly determined by multiple instances based on a negotiation mechanism. When an instance fails, other instances automatically take over the computing node partition it is responsible for to achieve fault tolerance. The multi-agent collaborative scheduling function enables the computing load of the system to be dynamically balanced among multiple computing nodes, effectively supporting the system expansion requirements in large-scale multi-user concurrent inference scenarios.

[0053] Comparative example: To objectively evaluate the technical effects of the various embodiments of the present invention, a comparative analysis of the prior art is provided below. This comparative analysis adopts a traditional edge computing large model inference architecture. In this architecture, each inference request independently maintains a complete KV context cache, and there is no cross-request KV vector sharing mechanism. Each inference request enters the inference engine sequentially according to the arrival order to perform serial inference. The inference engine performs a standard autoregressive decoding process. The generation of each token must wait for the calculation of the previous token to be completed. The system is not configured with dynamic batch processing and parallel inference scheduling functions, nor is it configured with edge resource awareness and global coordination control functions. There is a lack of real-time feedback of state information and a collaborative scheduling mechanism between functional modules.

[0054] Regarding system throughput, the comparative example shows that because each inference request independently maintains a complete KV context and does not support batch processing parallelism, multiple concurrent requests can only be processed serially. Experimental data shows that when processing inference requests with an average input length of 512 words and an average output length of 256 words on an edge device with an 8-core central processing unit plus a neural network processing unit, the single device processing capacity of the comparative example is only 3 to 4 requests per second. Embodiment 1 of the present invention improves the single device processing capacity to 18 to 22 requests per second through a KV cache sharing mechanism and dynamic batch processing parallel inference, which is more than 5 times higher than the comparative example.

[0055] Regarding computing power utilization, the autoregressive serial inference characteristic of the comparative model results in the central processing unit and neural network processing unit being idle and waiting for a large amount of time during the inference process. The actual test data shows that the average utilization rate of the central processing unit in the comparative model is only 15% to 20% and the average utilization rate of the neural network processing unit is only 25% to 35% when processing multiple concurrent requests. Embodiment 1 of the present invention combines the attention operations of multiple requests into batch matrix operations by fusing attention calculation, which significantly improves the utilization efficiency of computing power resources. The actual test data shows that under the same hardware conditions, the average utilization rate of the central processing unit in Embodiment 1 of the present invention is increased to 55% to 65% and the average utilization rate of the neural network processing unit is increased to 70% to 80%.

[0056] Regarding memory bandwidth efficiency, the comparative example shows that storing the complete KV vector independently for each request leads to a linear increase in memory usage with the number of concurrent requests. When processing 8 concurrent requests simultaneously, the KV cache memory usage of the comparative example reaches 4.2 gigabytes, and memory bandwidth becomes a bottleneck limiting inference performance. Embodiment 1 of the present invention achieves a significant reduction in memory usage through a cross-request KV vector sharing mechanism. Under the same conditions, the KV cache memory usage is only 1.8 gigabytes, which is more than 55% lower than that of the comparative example.

[0057] Regarding inference latency, the average end-to-end latency of requests in high-concurrency scenarios increases sharply due to the serial processing mechanism and lack of resource-aware scheduling. Actual test data shows that the 95th percentile latency of the comparison model exceeds 8 seconds in a scenario with 16 concurrent requests. Embodiment 1 of the present invention controls the 95th percentile latency to within 2.5 seconds in the same scenario through real-time resource monitoring and dynamic scheduling of the global coordination control module.

[0058] Table 1: Comparison of Embodiments of the Invention with Comparative Examples Analysis and summary of the above comparative data show that the embodiments of the present invention are significantly superior to the comparative embodiments of the prior art in terms of core performance indicators such as system throughput, computing power utilization, memory efficiency, and inference latency. Embodiment 1 of the present invention realizes cross-request KV vector sharing and dynamic batch processing parallel inference mechanism through KV cache pool management module, fundamentally breaking through the serial paradigm limitation of traditional autoregressive inference. Embodiment 2 further improves the adaptive scheduling capability of the system by expanding graphics processing unit support, increasing dynamic cache capacity adjustment and predictive batch processing optimization. Embodiment 3 significantly enhances the scalability of the system in large-scale multi-user concurrent scenarios through multi-level queue scheduling architecture, distributed cold cache storage and multi-agent collaborative scheduling function. The model predictive control algorithm and shared memory communication mechanism of the global coordination control module ensure that each embodiment can maintain a stable and efficient operating state and performance under different load conditions.

[0059] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A dynamic scheduling and inference optimization system for large models oriented to edge computing, characterized in that, It includes a request scheduling and concurrency management module, a key-value cache pool management module, a dynamic batch processing and parallel inference scheduling module, an inference engine optimization module, an edge resource awareness module, and a global coordination and control module; The output of the request scheduling and concurrency management module is connected to the input of the KV cache pool management module; The output of the KV cache pool management module is connected to the input of the dynamic batch processing and parallel inference scheduling module. The dynamic batch processing and parallel inference scheduling module is communicatively connected to the inference engine optimization module and the edge resource awareness module, respectively. The global coordination and control module is connected to the request scheduling and concurrency management module, the KV cache pool management module, the dynamic batch processing and parallel inference scheduling module, the inference engine optimization module, and the edge resource awareness module.

2. The large-scale model dynamic scheduling and inference optimization system for edge computing according to claim 1, characterized in that, The request scheduling and concurrency management module is internally configured with a request parsing unit and a task dispatching unit. The request parsing unit extracts the request identifier, input sequence length, expected service quality level and preset timeout threshold from each inference request. The task dispatching unit sorts the parsed inference requests according to a preset priority algorithm and distributes the sorted inference requests in order.

3. The large-scale model dynamic scheduling and inference optimization system for edge computing according to claim 1, characterized in that, The KV cache pool management module is internally configured with a multi-level cache management unit, a cache allocation unit, and a cache reclamation unit. The multi-level cache management unit divides the KV cache into three levels: hot cache area, warm cache area, and cold cache area. The hot cache area uses high-bandwidth memory for storage, the warm cache area uses double data rate synchronous dynamic random access memory for storage, and the cold cache area uses non-volatile memory for storage.

4. The large-scale model dynamic scheduling and inference optimization system for edge computing according to claim 1, characterized in that, The dynamic batch processing and parallel inference scheduling module is internally configured with a shared relationship analysis unit, a batch processing grouping unit, and a parallel execution scheduling unit. The shared relationship analysis unit determines whether there is a shareable KV vector interval between the current inference request and the historical inference request by comparing the prefix of the input sequence of each inference request with the key value identifier of the existing KV vector in the cache pool. The batch processing grouping unit aggregates multiple inference requests that meet the parallel conditions into a batch processing task and generates a fusion attention mask matrix.

5. The large-scale model dynamic scheduling and inference optimization system for edge computing according to claim 1, characterized in that, The inference engine optimization module is internally configured with a fusion attention calculation unit, a computation graph optimization unit, and a result separation unit. The fusion attention calculation unit concatenates the query vectors of each inference request in the batch dimension to obtain a batch query matrix, copies and expands the shared KV vectors in the batch dimension to obtain a batch KV matrix, then performs batch matrix dot product operation to obtain a fusion attention score matrix, and finally performs mask filtering on the fusion attention score matrix through a fusion attention mask matrix.

6. The large-scale model dynamic scheduling and inference optimization system for edge computing according to claim 1, characterized in that, The edge resource perception module is internally configured with a computing power monitoring unit and a memory monitoring unit. The computing power monitoring unit collects the real-time utilization rate, current frequency and number of active cores of the central processing unit and the neural network processing unit through a hardware performance counter. The memory monitoring unit collects the total system memory capacity, allocated memory, available memory and memory bandwidth utilization rate in real time.

7. The large-scale model dynamic scheduling and inference optimization system for edge computing according to claim 1, characterized in that, The global coordination and control module is internally configured with a state aggregation unit, a strategy calculation unit, and an instruction issuing unit. The state aggregation unit receives state information from each functional module, and the strategy calculation unit calculates the current optimal batch processing strategy and cache eviction strategy based on the aggregated multi-dimensional state information, generating a scheduling parameter set including the upper limit of batch processing task size, KV cache migration threshold, and inference task priority adjustment coefficient.

8. A dynamic scheduling and inference optimization system for large models oriented to edge computing according to claim 2, characterized in that, The priority algorithm adopts a weighted fair queue scheduling strategy, which assigns an initial weight value to each user's inference request. The weight value is adjusted according to the service quality level of the request. The weight value of a request with a high service quality level is multiplied by a priority coefficient. The adjusted weight value is used to determine the order position of the request in the waiting queue.

9. A dynamic scheduling and inference optimization system for large models oriented to edge computing according to claim 3, characterized in that, The hot cache storage capacity is set to 10% to 20% of the total cache capacity, the warm cache storage capacity is set to 30% to 50% of the total cache capacity, and the cold cache storage capacity is set to 40% to 60% of the total cache capacity. When the hot cache occupancy rate reaches a preset threshold of 80%, a cache pre-migration operation is triggered.

10. A large-scale model dynamic scheduling and inference optimization system for edge computing according to claim 5, characterized in that, The batch matrix dot product operation performed by the fusion attention computing unit adopts a block computing strategy, which divides the batch query matrix and batch KV matrix into multiple sub-blocks along the batch dimension. Each sub-block performs the matrix dot product operation in parallel in an independent computing thread. The block size is adaptively adjusted according to the cache capacity of the neural network processing unit of the edge device.

Images