A heterogeneous computing power node load balancing method based on tensor parallel segmentation

Abstract

This invention relates to the field of distributed parallel computing technology, specifically to a load balancing method for heterogeneous computing power nodes based on tensor parallel partitioning. The invention calculates dynamic capability scores based on node resource characteristics to determine the macroscopic partitioning ratio, performs biaxial orthogonal partitioning to obtain the theoretical partitioning boundary, rounds down the theoretical boundary to the smallest step unit and merges the remainders to construct sub-matrix blocks, concurrently executes local computation and prefetching, performs full reduction aggregation within the homogeneous domain after cross-layer temporal alignment, extracts differential residual tensors between domains for compression, transmission, and reconstruction, and finally collects node running status features to input into the reinforcement learning controller to generate fine-tuning values, shifts the partitioning boundary, and re-executes alignment constraints. This invention effectively avoids memory access latency, reduces cross-domain communication load, and maintains the overall inference throughput of the cluster.

Description

Technical Field

[0001] This invention relates to the field of distributed parallel computing technology, specifically to a load balancing method for heterogeneous computing power nodes based on tensor parallel partitioning. Background Technology

[0002] With the exponential growth in the parameter size of large language models, the memory of a single node can no longer accommodate models with hundreds of billions of parameters. Therefore, tensor parallelism, as a distributed computing paradigm, is widely used in autoregressive inference services for large language models in the cloud. Traditional tensor parallelism technology usually assumes that the underlying computing cluster is a homogeneous environment, that is, all nodes have completely consistent computing power, memory capacity, and network bandwidth, and performs static uniform partitioning of the weight matrix to be computed based on this assumption.

[0003] However, in actual cloud multi-tenant service deployments, computing clusters are often composed of a mix of graphics processing units or neural network processing units from different generations. These heterogeneous nodes exhibit significant differences in theoretical peak computing power, available memory capacity, and minimum execution tile size of the underlying tensor cores. When using a traditional static uniform partitioning strategy, the computational load cannot match the actual physical execution capabilities of the nodes. Nodes with weaker computing power or limited available memory become lagging nodes in the system due to longer local tensor processing times, forcing stronger nodes in the homogeneous partitioning paradigm to remain idle while waiting for global cross-layer synchronization, leading to severe idle computing resources and pipeline congestion. Furthermore, large computing clusters are typically distributed across multiple physical racks, resulting in extreme physical topology heterogeneity in the underlying interconnect network. Nodes within the same rack are often interconnected via a high-speed direct memory access physical backplane, possessing extremely high communication bandwidth; while nodes across racks rely more on Ethernet or InfiniBand links, which have limited bandwidth and high transmission latency. Traditional tensor parallel communication primitives require a global full reduction operation on the aggregated state matrix after each layer of computation. Faced with the concurrent synchronization requirements of high-dimensional tensor features, this dense floating-point communication mechanism, which ignores the underlying physical topology, is highly susceptible to network port queue congestion on low-bandwidth cross-domain links, leading to a non-linear surge in the proportion of communication overhead in the overall inference time. Furthermore, large language model inference tasks are highly dynamic; the transient memory allocation rate and chip core temperature fluctuate continuously over time, making static scheduling mechanisms based on offline performance evaluation unable to adapt to runtime load changes.

[0004] In summary, in complex computing clusters with multi-generational heterogeneous computing nodes and hierarchical physical interconnection topologies, how to reduce the long-tail latency caused by heterogeneous computing and reduce the physical communication load of cross-domain synchronization has become a core technical problem that urgently needs to be solved in this field.

[0005] To address this, a load balancing method for heterogeneous computing nodes based on tensor parallel partitioning is proposed. Summary of the Invention

[0006] The purpose of this invention is to provide a load balancing method for heterogeneous computing power nodes based on tensor parallel partitioning. This invention calculates dynamic capability scores based on node resource characteristics to determine the macroscopic partitioning ratio, performs biaxial orthogonal partitioning to obtain the theoretical partitioning boundary, rounds down the theoretical boundary to the smallest step unit and merges the remainders to construct sub-matrix blocks, concurrently executes local computation and prefetching, performs full reduction aggregation within the homogeneous domain after cross-layer temporal alignment, extracts differential residual tensors between domains for compression, transmission, and reconstruction, and finally collects node operating status features to input into the reinforcement learning controller to generate fine-tuning values, shifts the partitioning boundary, and re-executes alignment constraints. This invention effectively avoids memory access latency, reduces cross-domain communication load, and maintains the overall inference throughput of the cluster.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A load balancing method for heterogeneous computing power nodes based on tensor parallel partitioning includes: Calculate dynamic capability scores based on the resource characteristics of cluster nodes, and determine the macroscopic segmentation ratio of the weight matrix to be calculated; Based on the macro-segmentation ratio, the weight matrix to be calculated is orthogonally segmented along two axes to obtain the theoretical segmentation boundary; the theoretical segmentation boundary is aligned with the constraint and rounded down to the physical boundary that satisfies the maximum integer multiple of the alignment granularity of the matrix multiplication and addition hardware tile corresponding to the node, and a sub-matrix block is constructed; the local calculation and data prefetching of the sub-matrix block are executed concurrently, and global tensor aggregation is triggered after cross-layer temporal alignment; During the global tensor aggregation phase, within the isomorphic domain, the local computation results are fully reduced and aggregated; across domains, the differential residual tensor between the current and pre-stored historical time series aggregation results is calculated, and after bandwidth reduction transformation, it is transmitted to the remote node for inverse transformation and accumulation, completing the global tensor aggregation and updating the pre-stored data. Based on the node running state characteristics generated during local computation and global tensor aggregation, a reinforcement learning controller is used to predict lagging nodes and output fine-tuning values. The boundary is then segmented according to the fine-tuning value translation theory and the alignment constraint is retried to construct the sub-matrix block for the next time step.

[0008] Preferably, dynamic capability scores are calculated based on the resource characteristics of cluster nodes, and the macroscopic segmentation ratio of the weight matrix to be calculated is determined, including: collecting the transient memory allocation rate, cache hit rate, and floating-point throughput of each node as the resource characteristics; obtaining the theoretical peak computing power of the underlying hardware of each node, and performing a division operation on the floating-point throughput to convert it into relative computing power utilization; inputting the transient memory allocation rate, cache hit rate, and relative computing power utilization into a preset multivariate evaluation model, and outputting the dynamic capability score; wherein, the dynamic capability score represents the real-time available comprehensive computing power of the node; performing normalization processing on the dynamic capability score of each node to obtain the allocated weight, and determining the ratio of the number of row elements of each node on the node dimension axis of the weight matrix to be calculated based on the allocated weight, as the macroscopic segmentation ratio; each node covers the complete column range of the weight matrix to be calculated on the feature dimension axis.

[0009] Preferably, the theoretical partitioning boundary is obtained by performing a biaxial orthogonal partitioning of the weight matrix to be calculated according to the macro-partitioning ratio; the theoretical partitioning boundary is aligned and constrained to construct sub-matrix blocks, including: dividing the row number interval of each node into sub-matrix blocks according to the macro-partitioning ratio, and taking the column number interval of each node as the complete column range of the weight matrix to be calculated, and using the start and end row numbers of the row number interval and the complete column range to jointly determine the theoretical partitioning boundary of each node; obtaining the matrix block hardware size of each node's physical execution unit for a single scheduling and setting it as the smallest step unit of the matrix multiplication and addition hardware tile alignment granularity; extracting the number of rows not exceeding the total number of rows within the theoretical partitioning boundary and that can be most... The largest integer number of rows divisible by the smallest step unit row number; perform a single alignment operation on the total number of columns of the weight matrix to be calculated, and extract the largest integer number of columns divisible by the smallest step unit column number; construct a physical boundary using the largest integer number of rows and the largest integer number of columns, and extract data to build a sub-matrix block; subtract the largest integer number of rows from the total number of rows of the theoretical partition boundary to obtain the row direction margin value; on the node dimension axis, the row direction margin value of each node is sequentially passed forward and accumulated according to the node number order, and appended to the theoretical partition row number interval of the next node, which is then included in the physical boundary coverage area when the next node is rounded down for alignment.

[0010] Preferably, triggering global tensor aggregation after cross-layer temporal alignment includes: establishing a double-buffered prefetch area for feature data in the memory of each node; using a direct memory access controller in a background thread to load the dependent features of the next temporal step into the double-buffered prefetch area for feature data, completing data prefetching; executing matrix multiplication and addition instructions for sub-matrix blocks in the current temporal step in a foreground thread, completing local calculations; after all matrix multiplication and addition instructions in the current model layer have been executed, sending a completion timestamp signal to the master node; after the master node receives the completion timestamp signals from all nodes, generating a temporal alignment confirmation instruction, removing the communication barrier, and triggering entry into the global tensor aggregation stage.

[0011] Preferably, the homogeneous domain refers to a set of server racks connected via a physical backplane using Direct Memory Access Interconnect Protocol (DMI) and whose internal nodes have a consistent computing architecture; the cross-domain refers to a set of different server racks connected via Ethernet switches; performing full reduction aggregation includes: for each node within the homogeneous domain, extracting the original local result tensor after its local computation is completed; maintaining the data precision and dimensionality of the original local result tensor unchanged, and without performing data compression and sparse filtering, performing element-level numerical accumulation operations at corresponding spatial locations on the original local result tensors of all nodes within the homogeneous domain via bus-level interconnect links; and generating the complete result... The result tensor is synchronously written to each node within the isomorphic domain; the method further includes: the first and last adjacent time steps before and after triggering the split boundary translation event constitute a boundary change detection window period; within the boundary change detection window period, the differential residual compression mechanism is disabled, and the current time-series aggregation result is fully transmitted; within the stable interval of the split boundary after the end of the boundary change detection window period, the differential residual compression mechanism is re-enabled; the differential residual tensor of the current and pre-stored historical time-series aggregation results is extracted, including: within the same stable interval of the split boundary, calculating the numerical difference between the current time-series aggregation result and the previous time-series aggregation result continuously stored locally on the node, and generating a differential residual tensor.

[0012] Preferably, after bandwidth reduction and conversion, the data is transmitted to the remote node for inverse conversion and accumulation to complete global tensor aggregation and update pre-stored data. This includes: performing mask filtering on the differential residual tensor according to the numerical retention limit, extracting non-zero values ​​and coordinate indices of elements outside the limit to generate sparse dictionary format data; performing octet quantization on the non-zero values ​​to generate transmission data packets and sending them to the remote node; at the remote node, performing dequantization and restoration on the non-zero values ​​in the transmission data packets, and mapping the restored non-zero values ​​to the corresponding positions in the zero-value matrix using the coordinate indices to generate a reconstructed residual tensor; performing matrix accumulation on the reconstructed residual tensor and the pre-stored historical time-series aggregation results stored locally at the remote node to generate a global tensor aggregation result, and updating the pre-stored data locally at the remote node; after sending the transmission data packets, the sending node updates the pre-stored historical time-series aggregation results stored locally, and asynchronously receives confirmation synchronization messages carrying version numbers from the remote node via a background thread to perform status verification.

[0013] Preferably, based on the node operating state characteristics, the reinforcement learning controller predicts lagging nodes and outputs fine-tuning values, including: continuously collecting the node operating state characteristics of each node in the cluster within a preset sampling time window, including allocable memory capacity, core temperature, network queue buffer length, and local computation callback time, splicing and flattening them into a multi-dimensional state vector and inputting it into the reinforcement learning controller; predicting the latency increment prediction value of each node in the local computation callback time in the next time step; marking the node with the highest latency increment prediction value as a lagging node; outputting the negative proportional constant of the node dimension axis and the negative proportional constant of the feature dimension axis for lagging nodes as fine-tuning values ​​through the reinforcement learning controller; and directly calculating and determining the positive proportional constant of non-lagging nodes through the normalized algebraic constraint of the node splitting ratio.

[0014] Preferably, the sub-matrix block for the next time step is constructed by shifting the theoretical partition boundary according to the fine-tuning value and re-triggering the alignment constraint. This includes: superimposing the corresponding negative proportional constant on the initial biaxial allocation ratio of the lagging nodes in the previous time step, and superimposing the positive proportional constant calculated according to the algebraic constraint on the initial biaxial allocation ratio of the non-lagging nodes to obtain the updated biaxial partition ratio; performing normalization processing on the updated biaxial partition ratio, and recalculating the boundary coordinates on the biaxial axis according to the result to realize the translation of the theoretical partition boundary; and re-performing the downward rounding alignment according to the translated theoretical partition boundary to satisfy the physical boundary that satisfies the maximum integer multiple of the node matrix multiplication and hardware tile alignment granularity, and extracting data to generate the sub-matrix block for the next time step.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention achieves effective matching between the dimensions of the sub-matrix after asymmetric partitioning and the hardware scheduling granularity of the multiply-accumulate execution unit at the underlying node level by aligning the dual-axis theoretical partitioning boundary to the alignment granularity of the matrix multiply-accumulate hardware tiles at the node level, and in conjunction with a residual data merging mechanism. This structural alignment method avoids the merged memory access latency and pipeline bubbles caused by misaligned memory accesses, enabling a high degree of mapping between computational load and hardware execution units, thereby effectively improving the synchronization of local tensor multiply-accumulate instructions executed by each computing node on the physical time axis and the actual utilization rate of computing resources.

[0016] 2. This invention constructs a hierarchical aggregation and differential transmission architecture based on the network topology characteristics of heterogeneous clusters. While performing full reduction within the bus-level topology link, for cross-domain scenarios in the network-level topology link, it calculates the state difference residual tensor of the aggregation result between the current time series and the pre-stored historical time series, and performs bandwidth reduction transformation operations such as sparse extraction and bit-width truncation for transmission and reconstruction. This mechanism, while ensuring the convergence accuracy and numerical stability of the macroscopic tensor aggregation result, converts dense floating-point matrix synchronization into residual sparse synchronization, effectively reducing the data throughput load of low-bandwidth cross-domain physical links and improving communication efficiency in distributed parallel computing.

[0017] 3. This invention constructs load scheduling as an adaptive closed-loop feedback process that evolves over time. By setting an asynchronous state observation window, it collects multi-dimensional physical operating states of nodes, such as transient memory, core temperature, and callback latency, and inputs these into a reinforcement learning controller. This allows for the identification of lagging nodes based on objective latency increment predictions, and dual-axis proportional fine-tuning through boundary segmentation using translation theory and re-triggered hardware alignment. Under the boundary conditions where the physical links and hardware states of cluster nodes are in normal operating condition, this control logic enables the computing cluster to smoothly redistribute computing power across cycles based on dynamic load fluctuations during runtime. This mechanism alleviates the imbalance in computation time among heterogeneous nodes without resetting the global distributed computing graph, reduces the risk of pipeline blockage caused by long-tail latency and local memory overload, and thus improves the overall throughput performance of heterogeneous systems in continuous autoregressive inference tasks. Attached Figure Description

[0018] Figure 1 A flowchart of a heterogeneous computing power node load balancing method based on tensor parallel partitioning provided in an embodiment of the present invention; Figure 2 This is a flowchart of tensor biaxial partitioning and physical boundary alignment provided in an embodiment of the present invention; Figure 3 This is a diagram of a hierarchical aggregation and cross-domain communication architecture provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figures 1 to 3 This invention provides a load balancing method for heterogeneous computing power nodes based on tensor parallel partitioning, the technical solution of which is as follows: A load balancing method for heterogeneous computing power nodes based on tensor parallel partitioning includes: Calculate dynamic capability scores based on the resource characteristics of cluster nodes, and determine the macroscopic segmentation ratio of the weight matrix to be calculated; Based on the macro-segmentation ratio, the weight matrix to be calculated is orthogonally segmented along two axes to obtain the theoretical segmentation boundary; the theoretical segmentation boundary is aligned with the constraint and rounded down to the physical boundary that satisfies the maximum integer multiple of the alignment granularity of the matrix multiplication and addition hardware tile corresponding to the node, and a sub-matrix block is constructed; the local calculation and data prefetching of the sub-matrix block are executed concurrently, and global tensor aggregation is triggered after cross-layer temporal alignment; During the global tensor aggregation phase, within the isomorphic domain, the local computation results are fully reduced and aggregated; across domains, the differential residual tensor between the current and pre-stored historical time series aggregation results is calculated, and after bandwidth reduction transformation, it is transmitted to the remote node for inverse transformation and accumulation, completing the global tensor aggregation and updating the pre-stored data. Based on the node running state characteristics generated during local computation and global tensor aggregation, a reinforcement learning controller is used to predict lagging nodes and output fine-tuning values. The boundary is then segmented according to the fine-tuning value translation theory and the alignment constraint is retried to construct the sub-matrix block for the next time step.

[0021] Example 1: This embodiment proposes a typical application scenario, namely, in the deployment of inference services for multi-tenant large language models (such as autoregressive generative models with hundreds of billions of parameters) in the cloud, the computing power cluster is composed of heterogeneous nodes of different generations, different memory capacities and different microinstruction architectures (for example, some nodes are equipped with high-performance chips with 80GB of memory, and some are equipped with old chips with 32GB of memory), and these nodes are distributed in multiple physical racks connected by links with different bandwidths.

[0022] As one embodiment of the present invention, refer to Figure 1 A flowchart of a heterogeneous computing node load balancing method based on tensor parallel partitioning.

[0023] In the large language model inference scenario of this embodiment, the weight matrix to be calculated is the weight matrix of the linear transformation layer, and its shape is the number of output features × the number of input features. The node dimension axis corresponds to the row direction of this matrix, that is, the output feature dimension, and different nodes undertake the calculation tasks of different output neurons along this axis; the feature dimension axis corresponds to the column direction of this matrix, that is, the input feature dimension; each node covers the complete column range of the weight matrix to be calculated on the feature dimension axis, and there is no proportional division of the column direction between nodes.

[0024] Further, dynamic capability scores are calculated based on the resource characteristics of cluster nodes, and the macroscopic segmentation ratio of the weight matrix to be calculated is determined, including: collecting the transient memory allocation rate, cache hit rate, and floating-point throughput of each node as the resource characteristics; obtaining the theoretical peak computing power of the underlying hardware of each node, and performing a division operation on the floating-point throughput to convert it into relative computing power utilization; inputting the transient memory allocation rate, cache hit rate, and relative computing power utilization into a preset multivariate evaluation model, and outputting the dynamic capability score; wherein, the dynamic capability score represents the real-time available comprehensive computing power of the node; performing normalization processing on the dynamic capability score of each node to obtain the allocated weight, and determining the ratio of the number of row elements of each node on the node dimension axis of the weight matrix to be calculated based on the allocated weight, as the macroscopic segmentation ratio; each node covers the complete column range of the weight matrix to be calculated on the feature dimension axis.

[0025] Specifically, at the beginning of each inference task scheduling cycle, the cluster management controller continuously collects the operating status of each node through underlying probes. The collected data includes transient memory allocation rate and cache hit rate. For the floating-point throughput, which characterizes the chip's current actual computing power output, due to its extremely large physical dimensions, the theoretical peak computing power data of each node's underlying hardware is first extracted. The measured floating-point throughput is then divided by this theoretical peak computing power to calculate the relative computing power utilization rate in the [0,1] interval, thus completing the dimensionless unification of all features. The specific conversion formula is as follows: ; in, For nodes Actual floating-point throughput, For nodes The theoretical peak computing power of the underlying hardware is nominally specified. This is the dimensionless relative computing power utilization rate calculated.

[0026] After data acquisition and conversion, the three dimensionless features—transient memory allocation rate, cache hit rate, and relative computing power utilization—are input into a pre-defined multivariate evaluation model to calculate the dynamic capability score. This multivariate evaluation model uses the following linear formula: ; in, For nodes Dynamic ability score, For nodes Transient memory allocation rate, For nodes The cache hit rate. The formula uses... The algebraic transformation physically represents the negative correlation between higher memory usage and weaker node ability to receive new tasks. , and The first, second, and third weights are respectively set to 0.4, 0.3, and 0.3 in this embodiment. The weight coefficients within this model are not randomly set, but rather calculated offline using a historical scheduling trajectory dataset of long-sequence inference tasks performed by a heterogeneous cluster. The objective function is to maximize the overall throughput of the cluster, and the baseline values ​​are obtained through regression fitting. Specifically, based on 100 different load scenarios running on the target cluster, the above weight combination is trained using the reciprocal of the actual time taken for each node to complete a standard matrix multiplication operation as a label. After obtaining the capability score of each node through the above weighted summation operation, all scores are summed and normalized. The proportion of computational load that each node should bear in the weight matrix to be calculated is determined according to the normalized weights. The normalization formula is: ; Where N is the total number of nodes participating in the current phase in the heterogeneous computing cluster. The final calculated allocation weights... It is directly used as the ratio of the number of elements in the macroscopic segmentation ratio to guide subsequent tensor biaxial segmentation operations.

[0027] This invention introduces a theoretical peak division conversion mechanism for floating-point throughput when evaluating the computing power of heterogeneous nodes. This mechanism maps multidimensional heterogeneous physical parameters to a dimensionless interval, eliminating numerical overload and feature failure caused by differences in dimensional scales at the underlying data structure level. Combined with a multivariate weighted summation model determined by fitting historical scheduling trajectories, the load assessment process no longer relies on black-box functions or manual static presets. This allows for a more objective and realistic reflection of the dynamic carrying capacity of heterogeneous computing nodes in a multi-tenant concurrent environment, improving the rationality of macroscopic partitioning ratio allocation.

[0028] Furthermore, the biaxial orthogonal partitioning refers to simultaneously and independently performing partitioning operations on both the node dimension axis (row direction) and the feature dimension axis (column direction) of the weight matrix to be calculated. The two axes are geometrically orthogonal in the two-dimensional matrix coordinate space, and the distribution ratio of row elements on the node dimension axis and the distribution ratio of column elements on the feature dimension axis of each node are independently determined and unconstrained by each other, thereby forming an orthogonal partitioning pattern of two-dimensional regions in the matrix plane.

[0029] Reference Figure 2The flowchart describes the tensor biaxial partitioning and physical boundary alignment process. Based on the macroscopic partitioning ratio, the weight matrix to be calculated is orthogonally partitioned along two axes to obtain the theoretical partitioning boundary. The theoretical partitioning boundary is then aligned and constrained to construct sub-matrix blocks, including those on the node dimension axis and feature dimension axis of the weight matrix to be calculated. The macroscopic partitioning ratio divides the row number intervals of each node's sub-matrix, and the column number intervals of each node are taken from the complete column range of the weight matrix to be calculated. The theoretical partitioning boundary of each node is determined by the start and end row numbers of the row number intervals and the complete column range. The number of rows and columns of the matrix block hardware size of each node's physical execution unit in a single scheduling is obtained and set as the smallest step unit for matrix multiplication and addition hardware tile alignment granularity. The process also includes extracting... The largest integer number of rows that exceeds the total number of rows within the theoretical partition boundary and is divisible by the minimum step unit row number; a single alignment operation is performed on the total number of columns of the weight matrix to be calculated, and the largest integer number of columns that is divisible by the minimum step unit column number is extracted; the largest integer number of rows and the largest integer number of columns form the physical boundary and the data is extracted to construct the sub-matrix block; the row direction margin value is obtained by subtracting the largest integer number of rows from the total number of rows of the theoretical partition boundary; the row direction margin value of each node is sequentially accumulated and passed forward on the node dimension axis according to the node number order, and appended to the theoretical partition row number interval of the next node, and included in the physical boundary coverage area when the next node is rounded down for alignment.

[0030] In this embodiment, the matrix multiplication-add hardware tile alignment granularity refers to the tile size of a single hardware scheduling of the matrix multiplication-add instruction in the underlying tensor computation unit of a node. For a node equipped with a 16×16 floating-point multiplication-add array, the number of rows and columns of its matrix multiplication-add hardware tile alignment granularity is 16; for a node equipped with a 32×32 floating-point multiplication-add array, the number of rows and columns is 32. This parameter is obtained by reading the tensor core tile size register in the device query interface of each node's computation chip. The "maximum integer multiple" in the alignment operation refers to the largest integer value that does not exceed the theoretical partition boundary and is divisible by the smallest step unit of the sub-matrix block. It should be noted that due to the different hardware tile sizes of different nodes, the physical boundaries of the sub-matrix blocks may not be precisely stitched together at the element level. The initial setting of the macroscopic partition ratio and the merging mechanism of the residual offset band have ensured that the sum of the coverage of the sub-matrix blocks of all nodes is equal to the complete set of elements of the original weight matrix. Therefore, minor differences in the shape of the sub-matrix blocks of each node will not lead to element omission or double calculation.

[0031] Specifically, firstly, the hardware size of the matrix block for a single scheduling of the underlying multiply-accumulate physical execution unit of all nodes in the cluster is obtained, for example, 16×16 for node 1 and 32×32 for node 2. The number of rows and columns of this matrix block hardware size is set as the smallest step unit for the alignment granularity of the matrix multiply-accumulate hardware tiles. Then, the node dimension axis and feature dimension axis of the weight matrix to be calculated are obtained. On the node dimension axis, the row number intervals of the sub-matrix for each node are divided according to the calculated macro-segmentation ratio. On the feature dimension axis, the column number intervals of the sub-matrix for each node are all taken from the complete column range of the weight matrix to be calculated (i.e., column index 0 to the total number of columns minus one), without dividing column intervals between nodes according to the macro-segmentation ratio. The starting and ending row numbers of the sub-matrix row number intervals and the starting and ending column numbers of the complete column range are used to jointly determine the theoretical partitioning boundary of each node in the tensor coordinate system.

[0032] Since the theoretical partitioning boundary, based on proportional division, often fails to perfectly match the hardware block size, alignment constraints are applied to each interval. Specifically: the largest integer number of rows within the theoretical partitioning boundary that is divisible by the number of rows in the matrix block hardware size is extracted; the largest integer number of columns within the theoretical partitioning boundary that is divisible by the number of columns in the matrix block hardware size is also extracted. These maximum integer number of rows and columns form the updated physical boundary. Data points are then extracted from the weight matrix to be calculated based on this physical boundary to construct sub-matrix blocks. During the floor alignment process, the total number of rows in the theoretical partitioning boundary is subtracted from the maximum integer number of rows to generate a row direction margin value. This row direction margin value is accumulated forward along the node dimension axis according to the node number, forming a row direction margin offset band. This band is then included in the physical boundary coverage of the next adjacent node during floor alignment. For the column direction, since the column range of each node is consistent and covers the entire column, column direction hardware alignment only performs a single floor operation on the total number of columns in the entire matrix, without generating a column direction margin offset band that needs to be passed between nodes. The forward accumulation refers to appending the remaining value of the current node to the theoretical partitioning boundary of the next node in the next node's numbering order. When this next node expands downwards along an integer multiple of the hardware size for floor division and alignment, it incorporates this remaining value into its own sub-matrix block coverage, thus achieving lossless data truncation. For example, if node A has a theoretical row count of 70 and its underlying matrix block hardware row size is 16, then its physical row count is rounded down to 64, with a row-wise remaining value of 6. This 6-row remaining value is appended sequentially to the theoretical row number range of node B. When node B performs downward floor division and alignment, it incorporates this remaining value into its own physical boundary coverage. Node B's original theoretical row count was 90; after absorbing the 6-row remaining value from node A, it expands to 96 rows. Its underlying matrix block hardware row size is 32, so 96 is divisible by 32, resulting in a physical row count of 96 and a remaining value of zero.

[0033] This invention transforms the theoretical segmentation ratio at the mathematical level into a physical execution boundary that conforms to the characteristics of the underlying microarchitecture of heterogeneous chips. Through forced alignment and margin sequential merging mechanisms, it avoids the staggered memory access latency and pipeline stalls caused by misaligned memory addresses. This ensures that each segmented sub-matrix block precisely matches the scheduling granularity of the underlying chip's multiply-accumulate execution unit in the physical dimension, allowing direct invocation of the underlying standard tensor multiply-accumulate hardware instructions for execution without padding, thus improving the execution synchronization of local tensor multiply-accumulate calculations in heterogeneous chips.

[0034] Furthermore, after cross-layer temporal alignment, global tensor aggregation is triggered, including: establishing a double-buffered prefetch area for feature data in the memory of each node; using the direct memory access controller in the background thread to load the dependent features of the next temporal step into the double-buffered prefetch area for feature data, completing data prefetching; executing matrix multiplication and addition instructions for the sub-matrix block in the current temporal step in the foreground thread, completing local calculations; after all matrix multiplication and addition instructions in the current model layer have been executed, sending a completion timestamp signal to the master node; after the master node receives the completion timestamp signals from all nodes, generating a temporal alignment confirmation instruction, removing the communication barrier, and triggering entry into the global tensor aggregation stage.

[0035] Specifically, a double-buffered prefetch area for feature data is established in the video memory of each node for the next timing step. Before triggering the background prefetch operation, the direct memory access controller first queries the local cross-cycle control register to read the target physical segmentation boundary status allocated to this node by the system for the next timing step. If there is a boundary fine-tuning value in the cross-cycle control register that is in a pending state, the direct memory access controller calculates the physical start address, addressing step size, and data block byte length of the required dependent features in the video memory based on the updated target physical boundary, and generates a prefetch address mapping table accordingly; if there is no fine-tuning value, the same calculation is performed using the physical boundary of the current cycle to generate the prefetch address mapping table. Subsequently, the direct memory access controller, in an independent background thread, strictly follows the prefetch address mapping table to preload the dependent feature data required for the next timing step into the feature data double-buffered prefetch area to complete the data prefetch.

[0036] Meanwhile, the foreground thread invokes the tensor computation core to execute the matrix multiplication and addition instructions of the sub-matrix block in the current timing step to complete the local computation. During the execution of the matrix multiplication and addition instructions, each node focuses on local data processing and does not explicitly exchange data with other nodes regarding their execution status. Only after all matrix multiplication and addition instructions in the current model layer have been executed will each node send a completion timestamp signal to the master node. After continuously listening to and collecting the completion timestamp signals of all nodes in the heterogeneous computing cluster, the master node confirms that all nodes have reached the computation endpoint of the current layer and then generates a timing alignment confirmation instruction. Based on the timing alignment confirmation instruction, the global communication barrier across layers is removed, and the global tensor aggregation is triggered after the cross-layer timing alignment is completed.

[0037] This invention achieves a high degree of overlap in the physical timeline between tensor execution of the foreground computation kernel and data movement via direct memory access in the background by constructing a double-buffered prefetch area and decoupling the foreground and background concurrent thread control logic. During local computation, explicit wait dependencies between nodes are eliminated, and a unified timestamp barrier is set at the end of the model layer. This mechanism, while strictly maintaining the temporal correctness of distributed hierarchical dependencies in long-sequence models, maximizes the masking of single-node memory access and prefetch latency. Combined with boundary state pre-checking of cross-cycle control registers, it avoids the risk of cache pollution caused by dynamic partitioning and improves the effective utilization of the tensor computation core.

[0038] Furthermore, referring to Figure 3 The hierarchical aggregation and cross-domain communication architecture diagram shows that the homogeneous domain refers to a set of server racks connected via a physical backplane using the Direct Memory Access Interconnect Protocol (DMI) and whose internal nodes have a consistent computing architecture; the cross-domain refers to a set of different server racks connected via Ethernet switches; the full reduction aggregation is performed, including: for each node in the homogeneous domain, extracting the original local result tensor after its local computation is completed; keeping the data precision and dimensionality of the original local result tensor unchanged, and without performing data compression and sparse filtering, performing element-level numerical accumulation operations at the corresponding spatial positions on the original local result tensors of all nodes in the homogeneous domain through bus-level interconnect links; and converting the generated data into a single data type. The complete result tensor is synchronously written to each node in the isomorphic domain; the method further includes: the first and last adjacent time steps before and after triggering the split boundary translation event constitute a boundary change detection window period; within the boundary change detection window period, the differential residual compression mechanism is disabled, and the current time-series aggregation result is fully transmitted; within the stable interval of the split boundary after the end of the boundary change detection window period, the differential residual compression mechanism is re-enabled; the differential residual tensor of the current and pre-stored historical time-series aggregation results is extracted, including: within the same stable interval of the split boundary, calculating the numerical difference between the current time-series aggregation result and the previous time-series aggregation result continuously stored locally on the node, and generating a differential residual tensor.

[0039] The step of storing the first time-series aggregation result in the local contiguous storage space of the node as the pre-stored historical time-series aggregation result includes: calling the underlying hardware acceleration instructions to truncate the first time-series aggregation result from high-precision floating-point format to low-precision floating-point format for storage, and initializing the error compensation tensor in the local video memory of the node. The step of calculating the numerical difference between the current time-series aggregation result and the pre-stored historical time-series aggregation result in the contiguous storage space element by element includes: using the mixed-precision tensor multiply-add instructions of the underlying computing chip to read the pre-stored historical time-series aggregation result in low-precision floating-point format, the current time-series aggregation result in high-precision floating-point format, and the error compensation tensor in parallel, and performing asymmetric precision element-wise algebraic operations to generate a signed differential residual tensor; wherein, the numerical retention limit is determined according to the absolute value of each element in the signed differential residual tensor, and the non-zero numerical values ​​transmitted retain their positive or negative signs.

[0040] Specifically, when performing full reduction aggregation within a homogeneous domain, the reduction operation is completed through the following sequence of underlying hardware actions: First, a full reduction instruction is issued to the Direct Memory Access Interconnect (DMI) controllers of each node within the homogeneous domain. Each node's DMI controller, based on a pre-registered node topology mapping table, broadcasts the physical memory address of the original local result tensor after local computation to the remaining nodes within the homogeneous domain via the DMI protocol physical backplane link. Upon receiving the physical memory addresses of other nodes, each node's DMI controller initiates a point-to-point remote memory read operation via the DMI protocol bus, retrieving the original local result tensors from other nodes into the built-in temporary buffer of its own DMI controller. The entire process... The data payload transfer process is completed autonomously by the interconnect controller, without the need for the central processing unit of each node to intervene in the underlying byte-by-byte data copying, thus effectively offloading the communication intervention overhead of the computing core. After the original local result tensors of all nodes in the homogeneous domain have been pulled to the local temporary storage buffer, the tensor computing core of each node calls the underlying vector addition instruction to perform element-by-element numerical accumulation operation on the tensor data of all nodes in the temporary storage buffer, that is, to accumulate the values ​​of each node sequentially according to the corresponding spatial coordinate positions to generate a complete result tensor. Each node then calls the direct memory access write-back instruction to synchronously write the complete result tensor into the corresponding video memory or memory address space of its own node, completing the full reduction aggregation in the homogeneous domain and ensuring that the aggregation results held by each node in the homogeneous domain are completely consistent in value.

[0041] When the reinforcement learning controller triggers a split boundary translation event, it marks the two consecutive time steps before and after the translation as a boundary change detection window. Within this window, due to the change in the split ratio causing a change in the shape of the sub-matrix block, differential residual compression transmission is actively disabled, and each node is forced to perform cross-domain transmission of the current time-series aggregation result of full floating-point data to ensure business continuity.

[0042] When the detection window ends, the system enters the stable interval of the segmentation boundary. Within this stable interval, after the first timing step completes local computation and full reduction within the isomorphic domain, it invokes the type conversion acceleration instructions of the underlying hardware to truncate and convert the high-precision floating-point format of the first timing aggregation result into a low-precision floating-point format. This low-precision data is then stored in the node's local contiguous storage space as a pre-stored historical timing aggregation result. Simultaneously, an error compensation tensor with all zeros is allocated and initialized in the local video memory. In the subsequent second timing step, after performing the same operation to generate the high-precision current timing aggregation result, since both are on the same segmentation boundary and have identical shape dimensions, the hybrid-precision tensor core on the underlying computing chip directly and in parallel reads the low-precision floating-point format pre-stored historical timing aggregation result, the high-precision floating-point format current timing aggregation result, and the error compensation tensor from the contiguous storage space by executing asymmetric precision multiply-accumulate instructions. The underlying computing chip is controlled to perform element-wise algebraic operations on absolute coordinates in the same dimension. That is, the high-precision current result is subtracted from the low-precision historical result, and the cumulative error value in the error compensation tensor is added to calculate the error-corrected signed difference residual tensor, which provides a data basis for subsequent bandwidth reduction.

[0043] This invention effectively isolates the risk of shape and dimension changes caused by dynamic segmentation by constructing a boundary change detection window and a full / differential switching mechanism. During the full reduction aggregation phase within the homogeneous domain, the direct memory access interconnect controller bypasses the central processing unit to perform point-to-point remote memory reads, and the tensor computation core calls vector addition instructions to complete element-wise accumulation and write-back, decoupling computational resource consumption from communication bandwidth consumption and ensuring the numerical consistency of aggregation results across nodes within the homogeneous domain. Historical aggregation results are converted to a low-precision format for storage, and an error compensation loop mechanism is introduced. While performing asymmetric precision algebraic operations using mixed-precision tensor operators, numerical divergence is suppressed, ensuring strong consistency of the state machines among distributed nodes.

[0044] Further, after bandwidth reduction and transformation, the data is transmitted to the remote node for inverse transformation and accumulation, completing global tensor aggregation and updating pre-stored data. This includes: performing mask filtering on the differential residual tensor according to the numerical retention limit, extracting non-zero values ​​and coordinate indices of elements outside the limit to generate sparse dictionary format data; performing 8-bit integer quantization on the non-zero values, generating transmission data packets and sending them to the remote node; at the remote node, dequantizing and restoring the non-zero values ​​in the transmission data packets, parsing the coordinate indices to calculate the physical addressing offset of each element in the original tensor dimension, and directly assigning the restored non-zero values ​​to the corresponding continuous memory addresses of the pre-initialized zero-value matrix based on the physical addressing offsets to generate the reconstructed residual tensor; and then... The difference tensor is matrix-accumulated with the pre-stored historical time-series aggregation result stored locally at the remote node to generate a global tensor aggregation result, which then overwrites and updates the pre-stored data locally at the remote node. After sending the transmission data packet, the sending node synchronously updates the pre-stored historical time-series aggregation result stored locally using the non-zero values ​​extracted locally, and immediately releases the foreground computing thread to execute the next time-series step. After completing the local data overwrite, the remote node asynchronously sends back a confirmation synchronization message carrying a version number to the sending node. The sending node calls an independent background network listening thread to receive the confirmation synchronization message and performs a version number comparison and verification in the background network listening thread. If a version number misalignment is detected or no confirmation synchronization message is received within a preset clock cycle, a state rollback instruction is triggered.

[0045] The numerical retention limit is set using a relative threshold evaluation method based on fast sampling. Specifically, when processing the current local difference residual tensor quantum block, the system first calls the underlying reduction instruction to extract the local maximum absolute value of the sub-block. The system multiplies the aforementioned local maximum absolute value by an outlier scaling factor (e.g., 0.2) pre-calibrated based on the offline activation distribution characteristics of the large model to calculate an absolute outlier protection threshold independent of the sampling distribution. This threshold is based on accurately locking high-energy outlier extreme points carrying key semantic information in the feedforward network layer of the large language model. Before performing mask filtering, key outlier elements whose absolute values ​​exceed the absolute outlier protection threshold are forcibly locked and retained. Subsequently, for the remaining regular elements, the system calls the parallel pseudo-random number generator of the underlying computing chip to perform hierarchical sampling, extracting residual elements of a preset size to form a lightweight sampling kernel. The system takes the absolute value of the elements in the sampling kernel and performs local sorting using on-chip registers, using the value at the preset target percentile (e.g., the 90th percentile) as the mask filtering threshold representing the regular elements of the sub-block. The preset target percentile is a hyperparameter configured through offline game theory derivation, based on the upper limit of available transmission bandwidth of cross-domain physical links and the minimum precision loss tolerance for convergence of large language model inference. This setting achieves sparse filtering without compromising the core expressive power of the model by isolating long-tail extrema from the main distribution. This mechanism, without causing cross-node computational blockage, statistically still approximately controls each sub-block to retain about 10% of significant residual elements for transmission, achieving approximately ten times the sparse compression ratio while maximizing the preservation of key activation feature expressive power. When the local maximum absolute value of the sub-block, quickly obtained through underlying hardware reduction instructions, is insufficient... In this case, the transmission of the sub-block is skipped directly, and the remote node retains the corresponding historical aggregation result unchanged.

[0046] The step of quantizing the non-zero values ​​in the sparse dictionary format data to generate a transmission data packet includes: independently extracting the local maximum absolute value from the retained elements of the locally generated local differential residual tensor quantum block, using the local maximum absolute value as a unit; generating a local scaling factor using the local maximum absolute value; performing a symmetric 8-bit integer quantization operation on the retained signed non-zero residual values; synchronously calculating the rounding error between the original high-precision value and the quantized restored value, and accumulating and updating the rounding error in the error compensation tensor of the node; and encapsulating the generated 8-bit integer data and its corresponding local scaling factor into the frame header of the transmission data packet and sending it to the remote node. The process of dequantizing and restoring non-zero values ​​in the transmitted data packet includes: parsing the frame header to obtain the local scaling factor, and performing a symmetrical dequantization multiplication operation on the corresponding eight-bit integer data according to the local scaling factor to restore it to a non-zero value in floating-point format.

[0047] This invention completes the closed-loop action of quantization and dequantization from the underlying mathematical logic by introducing "symmetric octet integer quantization" and the corresponding rounding error accumulation and update mechanism, effectively suppressing the cumulative bias caused by simple "truncation"; and, based on the characteristic that the symmetric octet integer quantization has the characteristic of zero point forced alignment to 0, the whole process does not require additional transmission and calculation of offset, which meets the hardware acceleration requirements of tensor communication.

[0048] Specifically, after generating the differential residual tensor, a numerical retention limit is set, and a masking filtering operation is performed on the residual tensor to remove elements whose absolute values ​​are within the limit range, extracting only non-zero values ​​and their global physical coordinate indices to generate sparse dictionary-formatted data. Subsequently, each node independently scans and calculates the local maximum absolute value within its locally generated local differential residual tensor sub-block as the physical unit, generating a local scaling factor. The node uses this local scaling factor to perform an 8-bit integer quantization operation on the extracted non-zero values ​​and encapsulates and records the local scaling factor in the frame header of the transmission data packet, sending it to the remote node via slow Ethernet. Upon receiving the data, the remote node first parses the frame header to extract the corresponding local scaling factor, performs dequantization and restoration on the non-zero values ​​based on the local scaling factor, and maps them to the corresponding positions in the zero-value matrix using the coordinate index to generate a reconstructed residual tensor. The remote node performs matrix addition on this reconstructed residual tensor and the locally stored historical aggregation results to generate a global tensor aggregation result and overwrite and update the locally pre-stored data. After the update is complete, the remote node generates a confirmation synchronization message carrying the version number of the current aggregation result and sends it back to the sending node. Upon receiving the confirmation message, the sending node synchronizes and updates its local historical data according to the version number, ensuring that the data state machines of the sending node and the remote node remain consistent before proceeding to the next time step.

[0049] This invention transforms cross-domain transmitted data from a dense floating-point matrix into a low-bit sparse index structure by combining mask filtering and local asynchronous quantization encoding operations. The mechanism of independently generating local scaling factors in units of local difference blocks and encapsulating them in the data frame header eliminates cross-node blocking waiting time caused by searching for global extrema, allowing the quantization process to overlap with the computation pipeline. Combined with closed-loop feedback of confirmation synchronization messages with version numbers, this scheme effectively reduces the network congestion load of low-bandwidth cross-domain physical links, improves the execution efficiency of state aggregation timing, effectively maintains the synchronization reliability of historical data state machines among distributed nodes, and significantly suppresses numerical distortion during the dequantization reconstruction process.

[0050] Furthermore, based on the node operating state characteristics, a reinforcement learning controller is used to predict lagging nodes and output fine-tuning values. This includes: continuously collecting the node operating state characteristics of each node in the cluster within a preset sampling time window, including allocable memory capacity, core temperature, network queue buffer length, and local computation callback time; splicing and flattening these characteristics into a multi-dimensional state vector and inputting it into the reinforcement learning controller; predicting the latency increment of each node in the next time step based on the local computation callback time; marking the node with the highest latency increment prediction value as a lagging node; outputting the negative proportional constant of the node dimension axis and the negative proportional constant of the feature dimension axis for lagging nodes as fine-tuning values ​​through the reinforcement learning controller; and directly calculating and determining the positive proportional constant for non-lagging nodes through the normalized algebraic constraint of the node splitting ratio.

[0051] In this embodiment, the reinforcement learning controller adopts an Actor-Critic architecture. Both the Actor and Critic networks use a three-layer fully connected structure with 128 hidden layers and ReLU activation function. The state vector is 20N-dimensional (where N is the total number of nodes participating in the computation within the cluster), obtained by concatenating and flattening the allocable GPU memory bytes, physical chip core temperature monitoring value, network port send queue buffer length, and local computation callback time of each node within five consecutive sampling time windows (each sampling window is 10 time steps or 500ms, whichever arrives first). Each dimension is Z-score normalized. The local computation callback time refers to the total time from the matrix multiplication and addition instructions of the submatrix block being issued to the GPU stream to the GPU completing the computation and triggering the CPU-side callback event.

[0052] The Actor network outputs two continuous values, corresponding to the negative scaling constants of the node dimension axis and the feature dimension axis of the lagging nodes, respectively. To prevent load oscillations and no-load conditions, the original output of each linear node in the last layer of the network is... First, the range needs to be compressed to the (0,1) interval by the Sigmoid activation function. Then, a linear scaling operation is performed to strictly limit it to the value range of [0.005, 0.1]. The specific calculation formula for this linear scaling operation is as follows: ; For the positive proportionality constant system of non-lag nodes, instead of prediction via the network, a normalized constraint formula for the splitting ratio between nodes is used to deterministically distribute the computational load stripped from lag nodes to non-lag nodes. Assume the total proportion of computational load stripped from the set of lag nodes is... Then non-lag nodes The new segmentation ratio allocated in the next time step Satisfy the following algebraic constraints: ; in, Non-lag node The original segmentation ratio before the current timing step adjustment. This is a set of lagging nodes. This mechanism ensures, from a physical level, that the sum of the global partitioning ratios of the reconstructed cluster is always strictly equal to 1. The system embeds a hardware boundary anti-drop mechanism when performing dual-axis proportional translation: the system sets that the area of ​​the physical matrix mapped to the tensor dual axes of any node after its updated partitioning ratio must not be less than the hardware tile alignment granularity of the matrix multiplication and addition corresponding to the node's underlying execution unit. When a lagging node, due to continuous load stripping, causes the calculation ratio of its next time step to reach this lower limit threshold, the system forcibly freezes the negative proportional constant output of that lagging node, maintaining its partitioning ratio at the hardware minimum step unit, thereby avoiding underlying instruction execution errors caused by scheduling out-of-bounds errors. The Critic network then outputs a one-dimensional state value estimate.

[0053] Training employs the PPO algorithm and is conducted in an offline simulation environment. The simulation environment is built upon the cluster's historical operation logs: computation time, memory usage variation curves, and temperature variation curves of each node in the target heterogeneous cluster are collected across 200 mini-batch inference tasks. A time-series simulator of node performance is constructed using a Long Short-Term Memory (LSTM) network. Specifically, the memory usage rate and core temperature measurements of each node within consecutive historical time steps are extracted to construct a time-series input matrix, with the actual computation time corresponding to the heterogeneous nodes serving as the target label. The parameters of the LSM network are updated via backpropagation using the mean squared error loss function. The LSM network after training convergence serves as the offline simulation environment for reinforcement learning, receiving the splitting boundary actions output by the reinforcement learning controller and outputting the predicted computation time as the basis for calculating the feedback state vector. Each training episode contains 100 time-series steps. The reward function is defined as the current step... The objective function is the negative of the maximum local callback time for all nodes, plus the negative of the standard deviation of the callback times between nodes. Its mathematical objective function is expressed as: ; in, For nodes In the current step The local calculation callback time, This represents the average time taken by all nodes in the cluster at this timing step. This formula, with a purely negative reward scalar, drives the controller to simultaneously minimize tail latency and time differences between nodes. Training uses a learning rate of... The discount factor is 0.99 and the truncation parameter is 0.2. After training convergence, the Actor network weights are exported and deployed to the inference service process of each node. During the inference phase, the Actor network runs in feedforward mode and no longer explores.

[0054] Specifically, a reinforcement learning controller is deployed in the background control plane, and the execution thread of this controller is physically isolated from the foreground computation pipeline that executes tensor multiply-accumulate instructions. An asynchronous state observation window containing a preset number of timing steps is set. In this embodiment, the asynchronous state observation window is set to contain 20 to 50 consecutive inference timing steps each time, with a corresponding physical duration of approximately 0.5 to 2 seconds. The specific number of steps in the window is dynamically determined with reference to the average generation length of the current decoding request—a larger value is taken for short text generation scenarios to ensure statistical stability, and a smaller value is taken for long text generation scenarios to ensure timely load adjustment. During the running cycle of the asynchronous state observation window, the background control plane continuously collects the allocatable memory byte capacity, core temperature monitoring value, and local computation callback time of each node, and concatenates them to form a multidimensional state vector sequence.

[0055] During the production deployment phase, the reinforcement learning controller is configured in deterministic policy inference mode, and the exploration noise generation module is disabled. Since the feedforward neural network layer lacks internal memory units for processing dynamic temporal data, a flattening operation is performed on the multidimensional state vector sequence before input, concatenating and recombining historical state vectors at different time stamps into a high-dimensional complete spatial vector. The controller uses this high-dimensional spatial vector as input, utilizes the feedforward neural network layer to extract cross-cycle state correlation features, and extracts the time stamps of each node within the asynchronous state observation window that arrive at the cross-layer state aggregation barrier. The reciprocal or negative value of the time stamp delay distribution variance is calculated as the load balancing reward function.

[0056] Based on the extracted cross-cycle state correlation features, the reinforcement learning controller predicts the delay increment of each node in the next time step in the background. The node with the highest predicted delay increment is marked as a lagging node, and a fine-tuning value is output. This fine-tuning value consists of a negative proportionality constant for lagging nodes and a positive proportionality constant for non-lagging nodes. To avoid blocking the high-frequency autoregressive inference pipeline in the foreground, the output of the fine-tuning value by the reinforcement learning controller is performed asynchronously with the foreground computation, and the output fine-tuning value is written into the cross-cycle control register. When the current asynchronous state observation window ends and the next sub-matrix block reconstruction instruction is triggered, the theoretical dividing boundary is shifted and alignment constraints are re-triggered based on the fine-tuning value output in the cross-cycle control register to construct the sub-matrix block for the next observation cycle. This asynchronous observation and delayed activation mechanism eliminates the synchronization delay overhead of state acquisition and decision inference on the high-frequency autoregressive generation step.

[0057] This invention isolates the foreground computation pipeline through an asynchronous background state observation window, decoupling state acquisition, policy inference, and autoregressive generation. At the input end, the historical multidimensional state sequence is flattened and reconstructed, allowing the feedforward neural network layer to extract cross-cycle state correlation features through the multiplication of the weight matrix of the fully connected layer and the calculation of a nonlinear activation function. Lagging nodes are then identified based on the predicted value of the time delay increment. Combined with a delayed-effect mechanism that writes the output dual-axis adjustment constant into the cross-cycle control register, this invention provides preventative load intervention capabilities while avoiding the synchronous delay blocking of the main computation pipeline by model inference and feature extraction, thus maintaining the dynamic scheduling throughput performance of the cluster.

[0058] Furthermore, based on the theoretical boundary division of the fine-tuning value translation and re-triggering the alignment constraints, the sub-matrix block for the next time step is constructed, including: superimposing the corresponding negative proportional constants on the initial biaxial allocation ratios of the lagging nodes in the previous time step, and superimposing the positive proportional constants calculated based on algebraic constraints on the initial biaxial allocation ratios of the non-lagging nodes to obtain the updated biaxial division ratios; performing normalization processing on the updated biaxial division ratios, and recalculating the boundary coordinates on the biaxial axes based on the results to realize the translation of the theoretical division boundary; and re-performing the downward rounding alignment based on the translated theoretical division boundary to satisfy the physical boundary that satisfies the maximum integer multiple of the node matrix multiplication and hardware tile alignment granularity, and extracting data to generate the sub-matrix block for the next time step.

[0059] The step of extracting the data to be calculated based on the physical boundary after the second update and generating the sub-matrix block for the next timing step includes: executing the difference coordinate extraction instruction between the theoretical partition boundary before translation and the theoretical partition boundary after translation to generate the physical index of the boundary incremental data block with ownership change; controlling each node to maintain the original resident state of the unchanged data in the local video memory; for the data corresponding to the physical index of the boundary incremental data block, initiating a point-to-point local data migration between heterogeneous nodes through the direct memory access protocol to bypass the central processing unit, merging the incremental data into the original resident data to form the sub-matrix block for the next timing step.

[0060] Specifically, the initial allocation ratios of each node in the previous timing step are read. The initial dual-axis allocation ratios of lagging nodes are added to their corresponding negative scaling constants to reduce their loads; the initial dual-axis allocation ratios of non-lagging nodes are added to their corresponding positive scaling constants to increase their loads, resulting in an updated set of dual-axis partitioning ratios for each node. To prevent data omissions or boundary overlaps after the ratio adjustment, a normalization process using summation and division is performed on the updated set, forcibly constraining the sum of the partitioning ratios of each node on both the node dimension axis and the feature dimension axis to be strictly equal to 1. Subsequently, based on the normalized new ratios, the boundary coordinates are recalculated on the dual axes of the original filling alignment matrix to achieve a physical translation of the theoretical partitioning boundary. After the boundary translation, the alignment operation instruction is called again, and the new coordinates are rounded down to ensure that they again meet the requirement of being a maximum integer multiple of the matrix multiplication and addition hardware tile alignment granularity corresponding to each node. When extracting the data to be calculated according to the physical boundaries that are updated twice and conform to hardware regularity to generate the sub-matrix block for the next timing step, the difference coordinate extraction program is executed on the central processing unit of the master node: the row number interval and column number interval of the theoretical partition boundary of each node before translation are used to form the coordinate set before translation, and the row number interval and column number interval of the theoretical partition boundary of each node after translation are used to form the coordinate set after translation. The set difference operation is performed on the two sets of coordinates to extract the coordinate elements that belong only to the coordinate set after translation and not to the coordinate set before translation. Based on the memory base address of the weight matrix to be calculated... The total number of columns, step size, and data element byte width are used to perform algebraic operations on the aforementioned difference coordinate elements to calculate the linear addressing offset in the one-dimensional continuous video memory address space, generating a physical index list of boundary incremental data blocks with physical start address and block byte length as fields. The above difference coordinate extraction process is completed entirely on the central processing unit side using software algebraic operations, without relying on dedicated hardware instructions. The generated physical index list of boundary incremental data blocks is then submitted to the direct memory access controller as a direct memory access transfer descriptor to drive subsequent point-to-point local data migration. Each node is controlled to maintain the in-place resident state of unmodified data in its local video memory. For the data corresponding to the aforementioned boundary incremental data block physical index, the direct memory access protocol is invoked to initiate point-to-point local data migration bypassing the central processing unit between the corresponding heterogeneous nodes. After the incremental data arrives, it is directly merged and overwritten into the in-place resident data, thereby completing the reconstruction of the sub-matrix block in the next timing step.

[0061] This invention achieves effective transition and physical boundary adaptation for dynamic load allocation through a closed-loop mechanism involving proportional adjustment, strict normalization, and secondary hardware alignment. In the boundary migration phase, a reconstruction method combining unchanged data residing at its original location with direct memory access migration of incremental boundary blocks is employed, changing the traditional scheduling mode of global interruption and data redistribution. This mechanism effectively shields large-scale network transport of static data and significantly reduces the computational cost of memory reconstruction caused by boundary changes. This scheme, without interrupting model inference services, facilitates a smooth transfer of computational pressure to idle nodes, maintaining the continuity of the heterogeneous cluster computing pipeline and a stable balance in overall resource utilization.

[0062] This embodiment first improves the rationality of the macroscopic partitioning ratio allocation of heterogeneous nodes through dimensionless transformation and multivariate evaluation models. Secondly, by forcibly aligning the theoretical partitioning boundary to the smallest step unit of the underlying hardware and combining it with residual sequential merging, it avoids cross-step memory access latency and improves the execution synchronization of local computations. Simultaneously, it maximizes the masking of single-node memory access latency by utilizing a front-end / back-end decoupled double-buffer prefetch mechanism. Furthermore, by introducing asymmetric precision differential computation, local error compensation, and asynchronous quantization operations in cross-domain communication, it effectively reduces the data throughput load of low-bandwidth links and ensures strong consistency of the state machines among distributed nodes. Finally, by decoupling the front-end inference through a background asynchronous observation window, it utilizes the dual-axis fine-tuning values ​​output by reinforcement learning to shift the boundary, combined with in-situ dwell and local memory migration mechanisms, to achieve smooth redistribution of computing power across cycles without interrupting autoregressive services, maintaining a stable and balanced overall inference throughput performance of the cluster.

[0063] Example 2: This embodiment focuses on the core hardware and software collaborative flow process of the heterogeneous computing node load balancing method of the present invention in continuous autoregressive inference tasks, mainly covering the technical implementation steps of physical boundary reconstruction, topology-aware communication and asynchronous closed-loop scheduling.

[0064] After confirming the macroscopic partitioning ratio, a physical boundary alignment mechanism is initiated to address the issue that the theoretical partitioning ratio cannot directly match the execution granularity of heterogeneous hardware. First, the hardware size of the matrix block for a single scheduling by the underlying physical execution unit of each heterogeneous node participating in the calculation is obtained and set as the minimum step unit. After calculating the theoretical partitioning row coordinates of each node on the node dimension axis based on the macroscopic partitioning ratio, the largest integer number of rows that does not exceed the theoretical row boundary and is completely divisible by the corresponding minimum step unit is extracted. Each node covers the entire column range on the feature dimension axis, and the column direction only performs a single hardware alignment operation on the total number of columns of the entire matrix. Through this rounding down operation, the theoretical partitioning interval is shrunk and truncated to physical boundaries that conform to hardware regularity, and the corresponding data is extracted to generate sub-matrix blocks. For row direction surplus values ​​that are outside the physical boundaries due to the rounding down of the row direction, on the node dimension axis, strictly following the preset node numbering order, the surplus value of the current node is appended and passed to the theoretical partitioning interval of the next adjacent node. When performing the down-rounding alignment operation on the next adjacent node, its physical boundary is controlled to expand outward in an integer multiple of the smallest step unit, completely merging the remainder passed by the preceding node into its own physical coverage area. This mechanism ensures that the data set of all segmented submatrix blocks is always identical to the original weight matrix, and local calculation instructions avoid memory access latency caused by memory address misalignment.

[0065] Upon completion of local computation and entry into the global tensor aggregation phase, a hierarchical aggregation strategy is executed based on the differences in the underlying interconnect topology. For homogeneous domains connected via direct memory access backplane, the original high-precision format and dimensions of the local computation results of each node are maintained. The physical link is directly invoked to perform element-level full accumulation at the corresponding spatial location, and the results are synchronously written to the contiguous memory of each node within the domain. For cross-domain environments interconnected via Ethernet switches, differential residual transmission logic is initiated. Within the interval where the partition boundary is in a stable state, after the current timing step completes local computation, the mixed-precision tensor operator of the underlying heterogeneous computing chip is invoked to read in parallel the low-precision historical aggregation results, the currently generated high-precision aggregation results, and the pre-initialized error compensation tensor stored in the local contiguous storage space. Asymmetric precision element-wise algebraic operations are performed, the historical benchmark is subtracted, and the historical truncation error is injected to generate a signed differential residual tensor. Subsequently, a mask filtering based on numerical preservation limits is performed on the residual tensor to extract non-zero values ​​and coordinate indices to form a sparse dictionary, and then encapsulated using eight-bit integer quantization. After receiving the quantized data packet, the remote node performs inverse quantization and coordinate mapping, accumulates the reconstructed residual tensor with the locally stored data matrix to generate a global aggregation result, and sends back a confirmation synchronization message carrying a version number to trigger the sender's state machine update.

[0066] During the aforementioned computation and communication execution, an asynchronous state observation window is maintained in the background. The memory allocation capacity, core temperature, network queue buffer length, and local computation callback time of each node in the cluster are continuously collected, flattened, and concatenated into a multi-dimensional state vector, which is then input into a reinforcement learning controller deployed on the control plane. This controller extracts cross-cycle state features through a feedforward neural network to predict the latency increment of each node in the next time-series step. For the lagging node with the largest latency increment, the reinforcement learning controller only outputs the negative proportionality constant between the node dimension axis and the feature dimension axis. After obtaining this negative constant, combined with the normalized algebraic constraint logic of the node splitting ratio, the positive proportionality constant corresponding to the other non-lagging nodes is directly calculated. The original dual-axis splitting ratios of each node are superimposed with the corresponding constants and normalized. Based on this, the boundary coordinates are recalculated on the dual axes to achieve the physical translation of the theoretical splitting boundary. After the boundary translation command is issued, the aforementioned hardware alignment operation is performed again based on the new coordinates to update the physical boundary. When constructing the sub-matrix block for the next timing step, data in the local video memory that has not undergone ownership change is kept at its original address. For incremental data whose ownership has changed due to boundary translation, a point-to-point local memory migration bypassing the central processing unit is initiated through the direct memory access protocol to merge the incremental data into the original address data, thus completing the dynamic cross-cycle reallocation of heterogeneous loads.

[0067] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A load balancing method for heterogeneous computing power nodes based on tensor parallel partitioning, characterized in that, include: Calculate dynamic capability scores based on the resource characteristics of cluster nodes, and determine the macroscopic segmentation ratio of the weight matrix to be calculated; Based on the macro-segmentation ratio, the weight matrix to be calculated is orthogonally segmented along two axes to obtain the theoretical segmentation boundary; the theoretical segmentation boundary is aligned with the constraint and rounded down to the physical boundary that satisfies the maximum integer multiple of the alignment granularity of the matrix multiplication and addition hardware tile corresponding to the node, and a sub-matrix block is constructed; the local calculation and data prefetching of the sub-matrix block are executed concurrently, and global tensor aggregation is triggered after cross-layer temporal alignment; During the global tensor aggregation phase, within the isomorphic domain, the local computation results are fully reduced and aggregated; across domains, the differential residual tensor between the current and pre-stored historical time series aggregation results is calculated, and after bandwidth reduction transformation, it is transmitted to the remote node for inverse transformation and accumulation, completing the global tensor aggregation and updating the pre-stored data. Based on the node running state characteristics generated during local computation and global tensor aggregation, a reinforcement learning controller is used to predict lagging nodes and output fine-tuning values. The boundary is then segmented according to the fine-tuning value translation theory and the alignment constraint is retried to construct the sub-matrix block for the next time step.

2. The method for load balancing of heterogeneous computing power nodes based on tensor parallel partitioning according to claim 1, characterized in that, The dynamic capability score is calculated based on the resource characteristics of the cluster nodes, and the macroscopic segmentation ratio of the weight matrix to be calculated is determined. This includes: collecting the transient memory allocation rate, cache hit rate, and floating-point throughput of each node as the resource characteristics; obtaining the theoretical peak computing power of the underlying hardware of each node, and performing a division operation on the floating-point throughput to convert it into relative computing power utilization; inputting the transient memory allocation rate, cache hit rate, and relative computing power utilization into a preset multivariate evaluation model to output the dynamic capability score; wherein, the dynamic capability score represents the real-time available comprehensive computing power of the node; performing normalization processing on the dynamic capability score of each node to obtain the allocated weight, and determining the ratio of the number of row elements of each node on the node dimension axis of the weight matrix to be calculated based on the allocated weight as the macroscopic segmentation ratio; each node covers the complete column range of the weight matrix to be calculated on the feature dimension axis.

3. The heterogeneous computing power node load balancing method based on tensor parallel partitioning according to claim 1, characterized in that, The theoretical partitioning boundary is obtained by orthogonally partitioning the weight matrix to be calculated along two axes according to the macro-partitioning ratio. The theoretical partitioning boundary is then aligned and constrained to construct sub-matrix blocks, including: dividing the row number intervals of each node according to the macro-partitioning ratio; taking the complete column range of the weight matrix to be calculated from the column number intervals of each node; and using the start and end row numbers of the row number intervals and the complete column range to determine the theoretical partitioning boundary of each node. The hardware size of the matrix block for a single scheduling of the physical execution unit of each node is obtained and set as the minimum step unit for aligning matrix multiplication and addition hardware tiles. Finally, the minimum step unit is used to extract rows that do not exceed the total number of rows within the theoretical partitioning boundary. The largest integer number of rows divisible by the minimum step unit number of rows; perform a single alignment operation on the total number of columns of the weight matrix to be calculated, and extract the largest integer number of columns divisible by the minimum step unit number of columns; construct a physical boundary using the largest integer number of rows and the largest integer number of columns, and extract data to build a sub-matrix block; subtract the largest integer number of rows from the total number of rows of the theoretical partition boundary to obtain the row direction margin value; on the node dimension axis, the row direction margin value of each node is sequentially passed forward and accumulated according to the node number order, and appended to the theoretical partition row number interval of the next node, which is then included in the physical boundary coverage area when the next node is rounded down for alignment.

4. The method for load balancing of heterogeneous computing power nodes based on tensor parallel partitioning according to claim 1, characterized in that, After cross-layer temporal alignment, global tensor aggregation is triggered, including: establishing a double-buffered prefetch area for feature data in the memory of each node; using the direct memory access controller in the background thread to load the dependent features of the next temporal step into the double-buffered prefetch area for feature data, completing data prefetching; executing matrix multiplication and addition instructions for sub-matrix blocks in the current temporal step in the foreground thread to complete local calculations; after all matrix multiplication and addition instructions in the current model layer have been executed, sending a completion timestamp signal to the master node; after the master node receives the completion timestamp signals from all nodes, generating a temporal alignment confirmation instruction, removing the communication barrier, and triggering the entry into the global tensor aggregation stage.

5. The method for load balancing of heterogeneous computing power nodes based on tensor parallel partitioning according to claim 1, characterized in that, The homogeneous domain refers to a set of server racks connected via a physical backplane using the Direct Memory Access Interconnect Protocol and whose internal nodes have a consistent computing architecture; the cross-domain refers to a set of different server racks connected via Ethernet switches. Performing full reduction aggregation includes: for each node in the isomorphic domain, extracting the original local result tensor after local computation; maintaining the data precision and dimensionality of the original local result tensor, and without performing data compression and sparse filtering, performing element-level numerical accumulation operations on the original local result tensors of all nodes in the isomorphic domain through bus-level interconnection links; synchronously writing the generated complete result tensor to each node in the isomorphic domain; the method further includes: the first and last adjacent time steps before and after triggering the split boundary translation event constitute a boundary change detection window period; within the boundary change detection window period, disabling the differential residual compression mechanism and performing full transmission of the current time-series aggregation result; within the stable interval of the split boundary after the end of the boundary change detection window period, re-enabling the differential residual compression mechanism; extracting the differential residual tensor between the current and pre-stored historical time-series aggregation results, including: within the same stable interval of the split boundary, calculating the numerical difference between the current time-series aggregation result and the previous time-series aggregation result continuously stored locally on the node, and generating a differential residual tensor.

6. The method for load balancing of heterogeneous computing power nodes based on tensor parallel partitioning according to claim 1, characterized in that, After bandwidth reduction and transformation, the data is transmitted to the remote node for inverse transformation and accumulation, completing global tensor aggregation and updating pre-stored data. This includes: performing mask filtering on the differential residual tensor according to the numerical retention limit, extracting non-zero values ​​and coordinate indices of elements outside the limit to generate sparse dictionary format data; performing octet quantization on the non-zero values ​​to generate transmission data packets and sending them to the remote node; at the remote node, dequantizing and restoring the non-zero values ​​in the transmission data packets, and mapping the restored non-zero values ​​to the corresponding positions in the zero-value matrix using the coordinate indices to generate a reconstructed residual tensor; performing matrix accumulation of the reconstructed residual tensor and the pre-stored historical time-series aggregation results stored locally at the remote node to generate a global tensor aggregation result, and overwriting and updating the pre-stored data locally at the remote node; after sending the transmission data packets, the sending node updates the pre-stored historical time-series aggregation results stored locally, and asynchronously receives confirmation synchronization messages carrying version numbers from the remote node via a background thread to perform status verification.

7. The heterogeneous computing power node load balancing method based on tensor parallel partitioning according to claim 1, characterized in that, Based on the node operating state characteristics, a reinforcement learning controller is used to predict lagging nodes and output fine-tuning values. This includes: continuously collecting the node operating state characteristics of each node in the cluster within a preset sampling time window, including allocable memory capacity, core temperature, network queue buffer length, and local computation callback time; splicing and flattening these characteristics into a multi-dimensional state vector and inputting it into the reinforcement learning controller; predicting the latency increment of each node in the next time step based on the local computation callback time; marking the node with the highest latency increment prediction value as a lagging node; and outputting the negative proportional constants of the node dimension axis and the feature dimension axis for lagging nodes as fine-tuning values ​​through the reinforcement learning controller. The positive proportional constants for non-lagging nodes are directly calculated and determined through the normalized algebraic constraint of the node splitting ratio.

8. The method for load balancing of heterogeneous computing power nodes based on tensor parallel partitioning according to claim 1, characterized in that, Based on the fine-tuning value translation theory of the partition boundary and re-triggering the alignment constraints, the sub-matrix block for the next time step is constructed, including: superimposing the corresponding negative proportional constant on the initial biaxial allocation ratio of the lagging nodes in the previous time step, and superimposing the positive proportional constant calculated according to the algebraic constraints on the initial biaxial allocation ratio of the non-lagging nodes to obtain the updated biaxial partition ratio; performing normalization processing on the updated biaxial partition ratio, and recalculating the boundary coordinates on the biaxial axis according to the result to realize the translation of the theoretical partition boundary; based on the translated theoretical partition boundary, re-performing the floor alignment to the physical boundary that satisfies the maximum integer multiple of the node matrix multiplication and addition hardware tile alignment granularity, and extracting data to generate the sub-matrix block for the next time step.

Images