A heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks

Abstract

The application discloses a heterogeneous intelligent computing power optimization management scheduling system for accelerating large model inference tasks, and relates to the technical field of computing power optimization management scheduling.The application reconstructs computing power management paradigm through heterogeneous operator-level scheduling, cooperatively acts of dynamic size block division and gradient mutation detection, converts the memory fragmentation problem in the long sequence scene into a controllable block migration task, and breaks through the performance bottleneck of the traditional memory exchange mechanism;the operator-level scheduling strategy based on the hardware capability fingerprint library makes the calculation-intensive tasks such as position coding accurately match the vector instruction set hardware, and eliminates the resource mismatch loss caused by the black box scheduling;the integer routing and counting sorting algorithm reconstructs the expert selection process, realizes near-lossless inference in the instruction set limited node, and activates the potential value of the old computing power pool.

Description

Technical Field

[0001] This invention relates to the field of computing power optimization management and scheduling technology, and in particular to a heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks. Background Technology

[0002] With the accelerated construction of new power systems, grid fault assessment requires real-time processing of long time-series data generated by millions of sensors, equivalent to more than 128K tokens. Such tasks rely on large spatiotemporal models for millisecond-level anomaly detection. The computational efficiency of traditional GPU clusters for long sequence position encoding cannot meet the current requirements, and NPU arrays have difficulty maintaining stable throughput due to sudden faults. In addition, the computational load explodes exponentially when a fault occurs, and static resource allocation strategies can lead to a backlog of critical tasks.

[0003] Current mainstream solutions, such as AIAK's dynamic batch processing and TACO-LLM's sequence parallelism, have significantly improved performance in short text scenarios. However, when vLLM's PagedAttention mechanism processes long sequences, memory fragmentation causes a sharp drop in effective utilization. Alternatively, heterogeneous schedulers are used, but they rely on preset hardware labels and cannot perceive the computationally intensive characteristics of position encoding in the spatiotemporal model. Under sudden traffic surges, K8s elastic scaling requires minute-level response, which is much higher than the 200ms fault assessment window of the power grid.

[0004] Some manufacturers use model distillation to compress sequence length, but this results in a loss of fault identification accuracy. Other solutions reduce computational load by using FP8 precision, but older NPUs cannot benefit from this due to missing instruction sets. The key issue is that existing scheduling systems treat large models as black-box tasks, ignoring the heterogeneous computational differences of operators such as position encoding and state memory in spatiotemporal modeling. This causes high-end computing power to idle in critical tasks. Therefore, there is an urgent need for a heterogeneous intelligent computing power optimization management and scheduling solution to accelerate large model inference tasks. Summary of the Invention

[0005] In view of the aforementioned existing problems, the present invention is proposed.

[0006] This invention provides a heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks. It solves the problems of low efficiency in long sequence inference and inability to cope with sudden loads caused by existing systems due to fragmented memory, coarse scheduling granularity, and rigid hardware adaptation.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] This invention provides a heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks, comprising:

[0009] The heterogeneous operator perception module analyzes the computation graph of a large model in real time through lightweight probes injected during compilation, and identifies at least one type of heterogeneous sensitive operator among position-encoded computation tasks, state memory operators, and nonlinear activation functions.

[0010] The dynamic hardware matching engine, based on a pre-built hardware capability fingerprint library, allocates position encoding calculation tasks to CPU clusters that support vector instruction sets, and allocates state memory operators to GPU computing nodes with high-bandwidth video memory.

[0011] The fragmented memory management unit splits the Attention key-value cache into dynamically sized blocks and establishes a direct mapping channel with the host memory;

[0012] The burst traffic response module includes a lightweight container pool and a modal density analyzer deployed on edge nodes;

[0013] The routing optimization module includes an integer routing proxy subunit and a sorting acceleration subunit.

[0014] The pipeline coordination unit is configured with a double-buffered intermediate result storage area and a delayed response mechanism.

[0015] As a preferred embodiment of the heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks as described in this invention, the hardware capability fingerprint database includes:

[0016] Mark the computing unit support type for SSE4.2 / AVX2 / AMX instruction sets or NEON / SVE instruction sets;

[0017] Memory bandwidth grading labels based on HBM2, GDDR6, and DDR4;

[0018] A dynamic operator compatibility list generated by the cpuid instruction at runtime.

[0019] As a preferred embodiment of the heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks according to the present invention, the adjustment strategy of the dynamic size block is as follows:

[0020] Use a 64KB fixed block when the sequence length is ≤64K tokens;

[0021] The block size is dynamically calculated when the sequence length is greater than 64K tokens.

[0022] As a preferred embodiment of the heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks according to the present invention, wherein: in the dynamic size block adjustment strategy, dynamic size block calculation is performed when the sequence length is greater than 64K tokens, and the process includes:

[0023] When the detected sequence length satisfies L>65536, the sequence expansion factor λ=L / 65536 is calculated.

[0024] Where L is the current sequence length in tokens, 65536 is the baseline threshold in tokens, and λ is the sequence expansion factor, which is dimensionless.

[0025] The target block size is dynamically updated based on the expansion factor, and the update formula is as follows:

[0026]

[0027] Where, N t The target number of blocks, unit blocks, N b The base number of blocks is 1024, and α is the block density adjustment index, which is dimensionless.

[0028] Derive the initial size of the misaligned block based on the number of target blocks:

[0029]

[0030] Among them, B s This is the initial block size, in bytes, C. kv The number of bytes occupied by a single-token key-value cache, in bytes / token, depends on the hidden dimension and data type. kv represents the index identifier of the key-value pair cache.

[0031] Perform cropping:

[0032] B c =min(max(B s B min ),B max ),

[0033] Among them, B c This refers to the size of the cropped block, in bytes (B). min The minimum allowed block size is set to 16KB, B. max The maximum allowed block size is set to 256KB.

[0034] The final dynamic block size is obtained according to the page alignment rules:

[0035]

[0036] Among them, B d The aligned block size, in bytes, P align To align with the page size, 4096 bytes;

[0037] The block density adjustment index is adaptively mapped by the memory bandwidth and is expressed as:

[0038]

[0039] Where κ is the slope adjustment coefficient, b w This represents the current effective GPU memory bandwidth, in GB / s. b0 is the baseline bandwidth value, 900 GB / s.

[0040] As a preferred embodiment of the heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks as described in this invention, wherein: the direct connection mapping channel executes:

[0041] Hot-swap of video memory blocks based on PCIe / CXL protocol;

[0042] The gradient mutation detection algorithm is used to locate active cache blocks;

[0043] Migrate inactive blocks to host memory via PCIe atomic write operations.

[0044] As a preferred embodiment of the heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks according to the present invention, wherein: during the execution of the direct connection mapping channel, when the sequence exceeds a threshold, gradient mutation detection is performed to locate active cache blocks, and the process includes:

[0045] When the sequence length L > 65536 is detected, each block B is mapped on the direct-connection channel side. j Constructing a smooth access signal: S t =ηA t +(1-η)S t-1 ,

[0046] Among them, S t This represents the smooth access signal for token number t, measured in times per step. A t Represents the original access count, times / step, η represents the exponential smoothing factor, t is the token index, and S t-1 This is a smoothed value of the previous token;

[0047] For block B j The formula for calculating the maximum discrete gradient within a block is:

[0048]

[0049] Where, δ j This represents the gradient magnitude of the j-th block, expressed in increments per step, where j is the block index.

[0050] After summing the gradients of all blocks, calculate the global mean and standard deviation:

[0051]

[0052] Where, μ δ The gradient mean is expressed in terms of times per step, σ.δ N represents the gradient standard deviation, expressed in terms of times per step. t The target number of blocks;

[0053] Introducing a threshold amplification factor to adapt to different fragment densities:

[0054]

[0055] Where β is the threshold amplification factor, which is dimensionless, γ is the logarithmic slope coefficient, and N b The base number of blocks is 1024;

[0056] The condition for determining an active cache block is δ. j >μ δ +βσ δ Block B that meets the conditions j Marked as active, it remains in GPU memory, while the remaining blocks are migrated to host memory via PCIe / CXL channels.

[0057] As a preferred embodiment of the heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks according to the present invention, wherein: the lightweight container pool preloads a simplified model with redundant position coding layers removed and the weights of the fully connected layers converted to 8-bit integers.

[0058] The modal density analyzer locks the visual operator to the NPU node when the video stream percentage is greater than 60%.

[0059] As a preferred embodiment of the heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks as described in this invention, wherein: the integerized routing agent subunit performs:

[0060] The expert selection weights are prequantized into 8-bit integer tensors;

[0061] Verify the AVX2 instruction set support status of the computing power nodes;

[0062] The sorting acceleration subunit uses a counting sorting algorithm to generate an expert index.

[0063] As a preferred embodiment of the heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks as described in this invention, the following steps are taken in the integerized routing agent subunit: first, the expert selection weights are symmetrically quantized using int8; then, AVX2 support verification is performed on the computing power nodes to generate a scheduling mask. The process includes:

[0064] The floating-point weight vector w for the i-th expert i ∈R d First, calculate the mean and standard deviation:

[0065]

[0066] Where, μ i The mean of the weights of the i-th expert is dimensionless, d represents the vector dimension, and w represents the mean of the weights of the i-th expert. i,k This represents the k-th dimension element of the weight vector, dimensionless, where k represents the dimension index, 1-d, σ i Standard deviation is a dimensionless quantity.

[0067] The scaling factor is adaptively determined based on the weight distribution:

[0068]

[0069] Among them, s i μ represents the scaling factor for the i-th expert. i With σ i Same as above, q p This represents the distribution coverage coefficient, ranging from 2.5 to 3.0, and is dimensionless.

[0070] After centralization is completed, quantification will be implemented:

[0071] q i =clip(round(s) i (w i -μ i ),-128,127),

[0072] Where, q i The integer 8 weight tensor of the i-th expert is represented by a byte, clip(·) represents the interval clipping operator, round(·) represents the round nearest integer operator, and the constants -128 and 127 represent the signed 8-bit range, also in bytes.

[0073] The cpuid instruction is invoked on the nth node, with primary leaf 7 and child leaf 0, and register EBX is read. n ;

[0074] Extracting AVX2 flags using bitwise operations:

[0075]

[0076] Among them, f n The AVX2 flag representing the nth node, either 0 or 1, EBX n This represents the EBX register value returned by cpuid, 32 bits, hexadecimal, and a constant 2. 6 ,2 5 Used to locate the 5th flag bit;

[0077] Construct a scheduling mask based on the flags:

[0078]

[0079] Where, χn This represents the availability mask for the nth node, either 0 or 1;

[0080] Finally, the cluster support vector j = [χ1,χ2,…,χ] is obtained. M ], where h represents the node availability vector with a length of M, and M represents the total number of nodes in the cluster.

[0081] As a preferred embodiment of the heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks according to the present invention, wherein: the dual-buffered intermediate result storage area is configured with a ping-pong buffer structure to alternately access data;

[0082] The latency response mechanism enables cached results to be pushed back when the GPU node latency is greater than 5ms or the NPU node latency is greater than 8ms.

[0083] The beneficial effects of this invention are as follows: This invention reconstructs the computing power management paradigm through heterogeneous operator-level scheduling. The synergistic effect of dynamic size block partitioning and gradient mutation detection transforms the memory fragmentation problem in long-sequence scenarios into a controllable block migration task, breaking through the performance bottleneck of traditional memory swapping mechanisms. The operator-level scheduling strategy based on a hardware capability fingerprint library enables computationally intensive tasks such as position encoding to accurately match vector instruction set hardware, eliminating resource mismatch losses caused by black-box scheduling. Integerized routing and counting sorting algorithms reconstruct the expert selection process, achieving near-lossless inference on instruction set-constrained nodes and activating the potential value of legacy computing power pools. This invention achieves a leap from passive resource allocation to active computing flow shaping through deep coupling of data flow and hardware state: In scenarios with sudden traffic surges, a lightweight container pool and a modality density analyzer form an elastic buffer layer to ensure the service quality of critical tasks. A dual-buffered storage area and a latency response mechanism construct a fault-tolerant barrier, significantly reducing the risk of pipeline stalls in heterogeneous node collaboration. Attached Figure Description

[0084] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0085] Figure 1 This is a schematic diagram of the framework of the heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks in Example 1. Detailed Implementation

[0086] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0087] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0088] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0089] Example 1, referring to Figure 1 This embodiment provides a heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks, including:

[0090] The heterogeneous operator perception module analyzes the computation graph of a large model in real time through lightweight probes injected during compilation, and identifies at least one type of heterogeneous sensitive operator among position-encoded computation tasks, state memory operators, and nonlinear activation functions.

[0091] The dynamic hardware matching engine, based on a pre-built hardware capability fingerprint library, allocates position encoding calculation tasks to CPU clusters that support vector instruction sets, and allocates state memory operators to GPU computing nodes with high-bandwidth video memory.

[0092] The hardware capability fingerprint database includes:

[0093] Mark the computing unit support type for SSE4.2 / AVX2 / AMX instruction sets or NEON / SVE instruction sets;

[0094] Memory bandwidth grading labels based on HBM2, GDDR6, and DDR4;

[0095] A dynamic operator compatibility list generated by the runtime cpuid instruction;

[0096] The fragmented memory management unit splits the Attention key-value cache into dynamically sized blocks and establishes a direct mapping channel with the host memory;

[0097] The adjustment strategy for dynamic size blocks is as follows:

[0098] Use a 64KB fixed block when the sequence length is ≤64K tokens;

[0099] The block size is dynamically calculated when the sequence length is greater than 64K tokens.

[0100] In the dynamic-size block adjustment strategy, dynamic-size block calculation is performed when the sequence length is greater than 64K tokens. The process includes:

[0101] When the detected sequence length satisfies L>65536, the sequence expansion factor λ=L / 65536 is calculated.

[0102] Where L is the current sequence length in tokens, 65536 is the baseline threshold in tokens, and λ is the sequence expansion factor, which is dimensionless.

[0103] The target block size is dynamically updated based on the expansion factor, and the update formula is as follows:

[0104]

[0105] Where, N t The target number of blocks, unit blocks, N b The base number of blocks is 1024, and α is the block density adjustment index, which is dimensionless.

[0106] Derive the initial size of the misaligned block based on the number of target blocks:

[0107]

[0108] Among them, B s This is the initial block size, in bytes, C. kv The number of bytes occupied by a single-token key-value cache, in bytes / token, depends on the hidden dimension and data type. kv represents the index identifier of the key-value pair cache.

[0109] Perform trimming to avoid pieces being too large or too small:

[0110] B c =min(max(B s B min ),B max ),

[0111] Among them, B c This refers to the size of the cropped block, in bytes (B). min The minimum allowed block size is set to 16KB, B. max The maximum allowed block size is set to 256KB.

[0112] The final dynamic block size is obtained according to the page alignment rules:

[0113]

[0114] Among them, B d The aligned block size, in bytes, P align To align with the page size, 4096 bytes;

[0115] The block density adjustment index is adaptively mapped by the memory bandwidth and is expressed as:

[0116]

[0117] Where κ is the slope adjustment coefficient, taken as 3.5, b w This represents the current effective memory bandwidth of the GPU, in GB / s. b0 is the baseline bandwidth value, 900 GB / s.

[0118] Specifically, by leveraging the linkage between the expansion factor and the bandwidth-driven exponent, the number and size of convergent blocks are adaptively adjusted in the case of ultra-long sequences; the pruning step limits the block size to a controllable range, while page alignment ensures that the efficiency of video memory page mapping is aligned with DMA transfer; the bandwidth mapping function dynamically adjusts the adjustment exponent, causing high-bandwidth nodes to tend to generate more small blocks to reduce cache hit conflicts, and low-bandwidth nodes to tend to generate larger blocks to reduce the number of cross-bus moves.

[0119] Direct connection mapping channel execution:

[0120] Hot-swap of video memory blocks based on PCIe / CXL protocol;

[0121] The gradient mutation detection algorithm is used to locate active cache blocks;

[0122] Migrate inactive blocks to host memory via PCIe atomic write operations;

[0123] During the execution of the direct mapping channel, when the sequence exceeds the threshold, gradient mutation detection is performed to locate active cache blocks. The process includes:

[0124] When the sequence length L > 65536 is detected, each block B is mapped on the direct-connection channel side. j Constructing a smooth access signal: S t =ηA t +(1-η)S t-1 ,

[0125] Among them, S t This represents the smooth access signal for token number t, measured in times per step. A t Represents the original access count, times / step; η represents the exponential smoothing factor, 0.125-0.25; t is the token index; S t-1 This is a smoothed value of the previous token;

[0126] For block B j The formula for calculating the maximum discrete gradient within a block is:

[0127]

[0128] Where, δj This represents the gradient magnitude of the j-th block, expressed in increments per step, where j is the block index.

[0129] After summing the gradients of all blocks, calculate the global mean and standard deviation:

[0130]

[0131] Where, μ δ The gradient mean is expressed in terms of times per step, σ. δ N represents the gradient standard deviation, expressed in terms of times per step. t The target number of blocks;

[0132] Introducing a threshold amplification factor to adapt to different fragment densities:

[0133]

[0134] Where β is the threshold amplification factor, dimensionless, γ is the logarithmic slope coefficient, ranging from 0.4 to 0.6, and N... b The base number of blocks is 1024;

[0135] The condition for determining an active cache block is δ. j >μ δ +βσ δ Block B that meets the conditions j The active blocks are retained in the GPU memory, while the remaining blocks are migrated to the host memory via the PCIe / CXL channel to free up memory space for subsequent calculations.

[0136] Specifically, the gradient mutation detection process maintains stable judgment even in high-frequency jitter scenarios by applying exponential smoothing to the access signal; the maximum discrete gradient takes the most active position within the block as the representative and establishes a global reference with the mean and standard deviation, eliminating the scale bias caused by different sequence lengths; the threshold amplification factor maps the block density information to an adaptive adjustment of the threshold, making the judgment more stringent when the fragmentation rate is high and allowing more blocks to enter the active set when the fragmentation rate is low.

[0137] The burst traffic response module includes a lightweight container pool and a modal density analyzer deployed on edge nodes;

[0138] A simplified model is preloaded with a lightweight container pool, which removes redundant positional encoding layers and converts the weights of fully connected layers to 8-bit integers.

[0139] The modal density analyzer locks the visual operator to the NPU node when the video stream percentage is >60%;

[0140] The routing optimization module includes an integer routing proxy subunit and a sorting acceleration subunit.

[0141] Integerized routing proxy subunit execution:

[0142] The expert selection weights are prequantized into 8-bit integer tensors;

[0143] Verify the AVX2 instruction set support status of the computing power nodes;

[0144] The sorting acceleration subunit uses a counting sort algorithm to generate an expert index;

[0145] In the integer routing agent subunit, the expert selection weights are first symmetrically quantized using int8, then AVX2 support verification is performed on the computing power nodes to generate a scheduling mask. The process includes:

[0146] The floating-point weight vector w for the i-th expert i ∈R d First, calculate the mean and standard deviation:

[0147]

[0148] Where, μ i The mean of the weights of the i-th expert is dimensionless, d represents the vector dimension, and w represents the mean of the weights of the i-th expert. i,k This represents the k-th dimension element of the weight vector, dimensionless, where k represents the dimension index, 1-d, σ i Standard deviation is a dimensionless quantity.

[0149] The scaling factor is adaptively determined based on the weight distribution:

[0150]

[0151] Among them, s i μ represents the scaling factor for the i-th expert. i With σ i Same as above, q p This represents the distribution coverage coefficient, ranging from 2.5 to 3.0, and is dimensionless.

[0152] After centralization is completed, quantification will be implemented:

[0153] q i =clip(round(s) i (w i -μ i ),-128,127),

[0154] Where, q i The integer 8 weight tensor of the i-th expert is represented by a byte, clip(·) represents the interval clipping operator, round(·) represents the round nearest integer operator, and the constants -128 and 127 represent the signed 8-bit range, also in bytes.

[0155] The cpuid instruction is invoked on the nth node, with primary leaf 7 and child leaf 0, and register EBX is read. n ;

[0156] Extracting AVX2 flags using bitwise operations:

[0157]

[0158] Among them, f n The AVX2 flag representing the nth node, either 0 or 1, EBX n This represents the EBX register value returned by cpuid, 32 bits, hexadecimal, and a constant 2. 6 ,2 5 Used to locate the 5th flag bit;

[0159] Construct a scheduling mask based on the flags:

[0160]

[0161] Where, χ n This represents the availability mask for the nth node, either 0 or 1;

[0162] Finally, the cluster support vector j = [χ1,χ2,…,χ] is obtained. M ], where h represents the node availability vector with a length of M, and M represents the total number of nodes in the cluster;

[0163] Specifically, the quantization process compresses the dynamic range of floating-point weights through mean centering and standard deviation scaling, so that the data falls into the symmetrical int8 interval, and the quantization noise is mainly adjusted by the distribution coverage coefficient.

[0164] The pipeline coordination unit is equipped with a double-buffered intermediate result storage area and a delayed response mechanism.

[0165] The dual-buffered intermediate result storage area is configured with a ping-pong buffer structure for alternating data access.

[0166] The latency response mechanism enables cached results to be pushed back when the GPU node latency is >5ms or the NPU node latency is >8ms.

[0167] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks, characterized in that, include, The heterogeneous operator perception module analyzes the computation graph of a large model in real time through lightweight probes injected during compilation, and identifies at least one type of heterogeneous sensitive operator among position-encoded computation tasks, state memory operators, and nonlinear activation functions. The dynamic hardware matching engine, based on a pre-built hardware capability fingerprint library, allocates position encoding calculation tasks to CPU clusters that support vector instruction sets, and allocates state memory operators to GPU computing nodes with high-bandwidth video memory. The fragmented memory management unit splits the Attention key-value cache into dynamically sized blocks and establishes a direct mapping channel with the host memory; The burst traffic response module includes a lightweight container pool and a modal density analyzer deployed on edge nodes; The routing optimization module includes an integer routing proxy subunit and a sorting acceleration subunit. The pipeline coordination unit is equipped with a double-buffered intermediate result storage area and a delayed response mechanism. The adjustment strategy for the dynamic size block is as follows: Use a 64KB fixed block when the sequence length is ≤64K tokens; The block size is dynamically calculated when the sequence length is greater than 64K tokens. In the dynamic size block adjustment strategy, dynamic size block calculation is performed when the sequence length is greater than 64K tokens. The process includes: When the monitored sequence length satisfies When calculating the sequence expansion factor , in, The length of the current sequence, in tokens. The baseline threshold is represented by the unit "token". The sequence expansion factor is dimensionless. The target block size is dynamically updated based on the expansion factor, and the update formula is as follows: ， in, The target number of blocks, per unit block. The base number of blocks is 1024. The density adjustment index is dimensionless. Derive the initial size of the misaligned block based on the number of target blocks: ， in, This is the initial block size, in bytes. The number of bytes used for a single-token key-value cache, in bytes / token, depends on the hidden dimension and data type. Indicates the index of the key-value pair cache; Perform cropping: ， in, This refers to the size of the cropped block, in bytes. The minimum allowed block size is set to 16KB. The maximum allowed block size is set to 256KB. The final dynamic block size is obtained according to the page alignment rules: ， in, The aligned block size, in bytes. To align with the page size, 4096 bytes; The block density adjustment index is adaptively mapped by the memory bandwidth and is expressed as: ， in, This is the slope adjustment coefficient. This represents the current effective memory bandwidth of the GPU, in GB / s. This is the baseline bandwidth value, 900GB / s.

2. The heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks as described in claim 1, characterized in that, The hardware capability fingerprint database includes: Mark the computing unit support type for SSE4.2 / AVX2 / AMX instruction sets or NEON / SVE instruction sets; Memory bandwidth grading labels based on HBM2, GDDR6, and DDR4; A dynamic operator compatibility list generated by the cpuid instruction at runtime.

3. The heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks as described in claim 1, characterized in that, The direct-connect mapping channel executes: Hot-swap of video memory blocks based on PCIe / CXL protocol; The gradient mutation detection algorithm is used to locate active cache blocks; Migrate inactive blocks to host memory via PCIe atomic write operations.

4. The heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks as described in claim 3, characterized in that, During the execution of the direct mapping channel, when the sequence exceeds a threshold, gradient mutation detection is performed to locate active cache blocks. The process includes: When the sequence length is detected Then, on the direct-connect mapping channel side, for each block Constructing smooth access signals: , in, This indicates the smooth access signal for token number t, measured in times per step. This represents the original access count, in times per step. Represents the exponential smoothing factor. Indexed by token This is a smoothed value of the previous token; For blocks The formula for calculating the maximum discrete gradient within a block is: ， in, Indicates the first The gradient magnitude of the block, in units of times per step. For block indexes; After summing the gradients of all blocks, calculate the global mean and standard deviation: ， in, This is the gradient mean, expressed in terms of times per step. This represents the gradient standard deviation, expressed in times per step. The target number of blocks; Introducing a threshold amplification factor to adapt to different fragment densities: ， in, The threshold amplification factor is dimensionless. The logarithmic slope coefficient, This is the base number of blocks, with a value of 1024. The conditions for determining an active cache block are as follows: Blocks that meet the conditions Marked as active, it remains in GPU memory, while the remaining blocks are migrated to host memory via PCIe / CXL channels.

5. The heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks as described in claim 1, characterized in that, The lightweight container pool preloads a simplified model that removes redundant position coding layers and converts the weights of fully connected layers to 8-bit integers. The modal density analyzer locks the visual operator to the NPU node when the video stream percentage is greater than 60%.

6. The heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks as described in claim 1, characterized in that, The integerized routing proxy subunit performs: The expert selection weights are prequantized into 8-bit integer tensors; Verify the AVX2 instruction set support status of the computing power nodes; The sorting acceleration subunit uses a counting sorting algorithm to generate an expert index.

7. The heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks as described in claim 6, characterized in that, In the integerized routing agent subunit, the expert selection weights are first symmetrically quantized using int8, and then AVX2 support verification is performed on the computing power nodes to generate a scheduling mask. The process includes: For the The floating-point weight vector of each expert First, calculate the mean and standard deviation: ， in, Indicates the first The mean of the expert weights, dimensionless. Represents the vector dimension. Represents the weight vector of the first 1-dimensional element, dimensionless Indicates a dimension index, 1- , Standard deviation is a dimensionless quantity. The scaling factor is adaptively determined based on the weight distribution: ， in, Indicates the first Scaling factor for each expert and Same as above. This represents the distribution coverage coefficient, ranging from 2.5 to 3.0, and is dimensionless. After centralization is completed, quantification will be implemented: ， in, Indicates the first An expert's int8 weighted tensor, byte This represents the interval clipping operator. Represents the nearest integer operator, a constant. Represents a signed 8-bit range, byte; For the Each node calls the cpuid instruction, primary leaf 7, child leaf 0, and reads the register. ; Extracting AVX2 flags using bitwise operations: ， in, Indicates the first The AVX2 flag of each node, 0 or 1. This represents the EBX register value returned by cpuid, 32-bit, hexadecimal, constant. Used to locate the 5th flag bit; Construct a scheduling mask based on the flags: ， in, Indicates the first Availability mask for each node, either 0 or 1; Finally, the cluster support vectors are obtained. ,in, Represents the node availability vector, with a length of , This indicates the total number of nodes in the cluster.

8. The heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model inference tasks as described in claim 1, characterized in that, The dual-buffered intermediate result storage area is configured with a ping-pong buffer structure to alternately access data. The latency response mechanism enables cached results to be pushed back when the GPU node latency is greater than 5ms or the NPU node latency is greater than 8ms.

Images