Distributed inference method for large language model based on intra-layer parallelism and communication quantization

Abstract

This invention discloses a distributed inference method for large language models based on intra-layer parallelism and communication quantization. The method relates to the field of distributed inference technology and includes the following steps: computational resource efficiency assessment and control; network environment adaptation assessment; and inference result generation and output. This invention, in distributed inference of large language models, first assesses the efficiency of computational resource utilization; if the efficiency is insufficient, it optimizes synchronous communication and load balancing; then it assesses the network environment adaptability; if the adaptability is insufficient, it optimizes the network topology and transmission protocol; finally, it outputs the inference result. This achieves efficient resource utilization and adaptive control in heterogeneous network environments, improving the practicality of distributed inference for large language models and solving the problem in existing technologies where the existing parallel inference schemes for large language models cannot achieve a balance between high computational efficiency and low network dependence, resulting in low practicality of distributed inference for large language models.

Description

Technical Field

[0001] This invention relates to the field of distributed reasoning technology, and in particular to a distributed reasoning method for large language models based on intra-layer parallelism and communication quantization. Background Technology

[0002] In the distributed inference process of large-scale language models, during the initialization phase, a parallel environment is set up using the torch.distributed framework, including GPU (Graphics Processing Unit) allocation and process configuration. Subsequently, model parallel groups are created, and model parameters are split according to a strategy, allowing each device to maintain only a subset of parameters. In the data preprocessing phase, a data parallel strategy is employed to divide input batches and quantize the data, converting high-precision floating-point numbers to low-precision representations to reduce computational overhead. After entering the intra-layer parallel computing phase, each device performs inference based on local parameters. In the attention layer, QKV weights are split column-wise and aggregated using AllGather; in the FFN (Feed-Forward Network) layer, weights are split column-wise and row-wise, and intermediate results are merged using ReduceScatter. Simultaneously, operator fusion technology is employed to merge multiple linear layers and operations into a single kernel to improve efficiency. In the communication quantization phase, intermediate results transmitted between devices are quantized and compressed to reduce the amount of communication data, and the receiving end then performs dequantization to restore precision. During the result aggregation phase, the outputs of various devices are integrated according to a parallel strategy. Data parallelism is achieved by synchronizing results through AllReduce, while model parallelism is achieved by completing calculations through specific communication methods. Finally, in the output phase, the aggregated results are post-processed, such as sequence concatenation and decoding in text generation, to produce the final readable text.

[0003] In a distributed computing environment consisting of heterogeneous, consumer-grade hardware and standard computer networks, there is an irreconcilable contradiction between efficiency and adaptability from the perspective of the inherent defects of the technical solutions. On the one hand, although pipeline parallelism can adapt to slow standard networks (without the need for high-frequency and high-volume data transmission), its "layer-by-layer computation" mode leads to serious idle computing resources - at any given time, only a single node is in an effective computing state, while the rest of the nodes form "pipeline bubbles" due to waiting for data or idle tasks. On the other hand, although traditional tensor parallelism can achieve efficient parallel computing, it can only be used in dedicated high-performance computing environments due to extremely high communication overhead and cannot be adapted to ordinary commercial hardware and networks.

[0004] From a practical perspective, scenarios heavily reliant on heterogeneous consumer-grade clusters require large-scale LLM (Large Language Model) deployments through distributed inference, yet are constrained by cost limitations in building high-performance computing environments. Current technologies fail to enable efficient parallel computing for these clusters or reduce communication requirements to accommodate their network hardware. Ultimately, this renders large-scale LLM deployments entirely impossible in such scenarios. The existing parallel inference solutions for large language models fail to strike a balance between high computational efficiency and low network dependency, resulting in low practicality of distributed inference for large language models. Summary of the Invention

[0005] To address the technical problem of limited practicality of distributed inference for large language models due to the inability of existing parallel inference schemes to achieve a balance between high computational efficiency and low network dependency, this invention provides a distributed inference method for large language models based on intra-layer parallelism and communication quantization. The technical solution is as follows:

[0006] On the one hand, a distributed inference method for large language models based on intra-layer parallelism and communication quantization is provided. This method includes: obtaining distributed computing efficiency parameters reflecting the efficiency of computing resource utilization during the distributed inference process of the large language model; obtaining a distributed computing efficiency value based on the obtained distributed computing efficiency parameters to quantify the degree to which hardware computing resources within the cluster are effectively utilized in a distributed computing environment composed of heterogeneous, consumer-grade hardware and standard computer networks; determining whether to perform distributed computing efficiency adjustment based on the distributed computing efficiency value; if so, continuing with the environment adaptation status monitoring step after performing distributed computing efficiency adjustment; otherwise, directly performing the environment adaptation status monitoring step and then performing distributed computing efficiency adjustment. The control function combines resource utilization and distributed computing efficiency values ​​to optimize synchronous communication and computational load balancing. The environment adaptation status monitoring command uses network environment adaptability, reflecting network communication performance, to obtain environment adaptation parameters. This quantifies the compatibility and adaptability of the large language model parallel inference scheme to different network hardware conditions in a distributed computing environment composed of heterogeneous, consumer-grade hardware, and standard computer networks, thus obtaining the network environment adaptability. Based on the obtained network environment adaptability, it determines whether to perform environment adaptation control. If so, after performing environment adaptation control, the large language model distributed inference output result is sent; otherwise, the large language model distributed inference output result is sent directly. Environment adaptation control represents the optimization of cluster network communication topology and transmission protocols based on the network environment adaptability.

[0007] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0008] 1. By establishing a multi-dimensional efficiency monitoring and dynamic adjustment mechanism, efficient utilization of computing resources is achieved in a distributed environment composed of heterogeneous consumer-grade hardware. Specifically, a performance profiler collects three key parameters in real time: node computing activity ratio, layer computing parallelism, and layer time utilization. A weighted geometric average formula is used to comprehensively calculate the distributed computing efficiency value of computing resource utilization. This quantitative system can not only accurately assess the overall performance of the cluster but also precisely locate performance bottlenecks: when the efficiency value is lower than the reference value, dynamic weight sharding adjustment is automatically triggered—the root node reallocates the weight shard dimensions based on the standard deviation of each node's computing time by querying the load sharding-node computing power mapping table, ensuring that the computing load is accurately matched with the actual computing power of heterogeneous nodes. At the same time, the non-parallelization layer strategy adjustment mechanism intelligently identifies layers whose computing-communication time ratio is lower than a threshold, adjusts them from parallelization layers to non-parallelization layers, and centrally computes them by the root node, avoiding inefficient synchronization overhead and improving the resource utilization of the heterogeneous cluster.

[0009] 2. By constructing a full-link network adaptation scheme centered on "communication quantization optimization + network dynamic adjustment," this approach overcomes the dependence of traditional tensor parallelism on high-bandwidth dedicated networks, achieving standard Ethernet. When bandwidth utilization exceeds the control value, the target bit width is queried from the quantization bit width-network state mapping table, and the activation value is gradually reduced. At the network adjustment level, topology and protocol optimization are performed in conjunction with network environment adaptability (calculated by weighted harmonics of key synchronization data volume, average communication-to-computation ratio, and efficiency value): When inter-node latency deviation exceeds the threshold, communication sub-clusters are divided using K-means clustering, with full interconnection within sub-clusters and backbone links between sub-clusters to reduce cross-node communication latency; when network packet loss rate exceeds the threshold, the TCP (Transmission Control Protocol) window width and retransmission timeout are adjusted to improve throughput. In gigabit Ethernet environment testing, this scheme reduces the communication-to-computation ratio of distributed inference in the Llama-13B model, improves network environment adaptability, and avoids inference interruption problems caused by communication congestion in Ethernet, a problem common to traditional schemes.

[0010] 3. By innovatively designing a two-layer control logic, real-time optimization of "computational efficiency" and "network adaptability" during inference is achieved, avoiding the performance drop of static solutions under environmental fluctuations. The first layer is a computational efficiency closed loop: efficiency parameters are periodically collected by a performance profiler, the efficiency value is calculated and compared with a reference value, triggering synchronous communication optimization (dynamic quantization, algorithm parameter adjustment) or load balancing control to ensure continuous and efficient utilization of computing resources. The second layer is a network adaptability closed loop: network environment adaptability is calculated based on environmental adaptability parameters, and when it is lower than the reference value, topology optimization (layered hybrid topology, dynamic path weight) or protocol control is performed to ensure stable network communication. In multi-scenario testing, when the number of cluster nodes increases, the system automatically switches from planar ring All-Reduce to tree-shaped recursive All-Reduce, reducing the number of communication steps; when Ethernet bandwidth fluctuates, the quantization bit width is dynamically reduced, further reducing the amount of synchronous data, and the inference process is uninterrupted. This dual closed-loop system makes the method more robust than traditional static distributed solutions in complex environments with heterogeneous hardware and fluctuating networks.

[0011] 4. Through innovative horizontal model partitioning and synchronous data quantization techniques, efficient parallel inference of large language models was successfully achieved on a distributed cluster composed of commercial hardware and standard Ethernet. This method overcomes the bottleneck of sequential computation in pipelined parallelism and the high network requirements of traditional tensor parallelism in existing technologies, finding an effective path to balance computational efficiency and communication costs. By distributing model weights and computational loads across multiple low-cost devices and significantly reducing communication overhead between nodes, this invention not only significantly accelerates the inference speed of large language models, but its performance also scales with the number of nodes. This provides a practical and economical solution for deploying and running advanced large-scale AI (Artificial Intelligence) models in resource-constrained environments, thus potentially promoting the application and popularization of large language model technology in a wider range of fields. Furthermore, this technical framework is not limited to inference; its core ideas also have the potential to be applied to distributed model training, opening up new possibilities for accelerating the development cycle of AI models. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of the distributed reasoning method for large language models based on intra-layer parallelism and communication quantization provided in this embodiment of the invention;

[0014] Figure 2 This is a flowchart of the dynamic quantization bit width adjustment process of the distributed inference method for large language models based on in-layer parallelism and communication quantization provided in this embodiment of the invention.

[0015] Figure 3 This is a pipelined parallel flowchart of the distributed reasoning method for large language models based on intra-layer parallelism and communication quantization provided in this embodiment of the invention;

[0016] Figure 4 This is a tensor parallel flowchart of a distributed reasoning method for large language models based on intra-layer parallelism and communication quantization provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0018] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0019] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, their intended meanings are consistent. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, their intended meanings are consistent.

[0020] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0021] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0022] This invention provides a distributed inference method for large language models based on intra-layer parallelism and communication quantization. For example... Figure 1 The flowchart shown is for a distributed inference method for large language models based on intra-layer parallelism and communication quantization. The processing flow of this method may include the following steps:

[0023] As the first step in the large language model distributed inference method based on intra-layer parallelism and communication quantization, the distributed computing efficiency parameter is obtained to reflect the efficiency of computing resource utilization during the distributed inference process of the large language model. Based on the obtained distributed computing efficiency parameter, the resource utilization distributed computing efficiency value is obtained, which is used to quantify the degree to which the hardware computing resources within the cluster are effectively utilized in the large language model parallel inference scheme in a distributed computing environment composed of heterogeneous, consumer-grade hardware and standard computer networks.

[0024] It should be explained that the distributed computing efficiency parameters obtained include node computational activity ratio, layer computational parallelism, and layer time utilization. Node computational activity ratio refers to the percentage of time that the core computing unit (such as a GPU or CPU) on a single computing node is used for effective computation during a distributed inference task of a large language model. Layer computational parallelism refers to the percentage of nodes in the cluster simultaneously participating in the computation of a specific layer (such as a linear layer or attention layer in a Transformer block) during forward propagation computation, relative to the total number of nodes. Layer time utilization refers to the ratio of the cumulative effective computation time of all nodes on a specific layer of the large language model to the product of the total number of nodes in the cluster and the total processing time of that layer.

[0025] Within a pre-defined distributed inference task cycle of a single large language model, the computational activity ratio of all nodes is collected using a performance profiler, and the arithmetic mean of the node computational activity ratios is calculated as the average node computational activity ratio. All parallelizable layers of the large language model are traversed to obtain their layer computational parallelism, and the arithmetic mean of the layer computational parallelism is calculated as the average layer computational parallelism. All parallelizable layers of the model are traversed to collect the layer time utilization of each layer, and the arithmetic mean of the layer time utilization is calculated as the average layer time utilization rate. The average node computational activity ratio, average layer computational parallelism, and average layer time utilization rate are then weighted and geometrically averaged to obtain the distributed computing efficiency value for resource utilization.

[0026] It needs to be explained that the constraint expression for the resource utilization efficiency value of distributed computing is:

[0027] ;

[0028] In the formula, E represents the distributed computing efficiency value of resource utilization, Tnode represents the average node computing activity ratio, Player represents the average layer computing parallelism, Ulayer represents the average layer time utilization, v1 represents the computing activity ratio weight obtained from the distributed inference database, v2 represents the layer computing parallelism weight obtained from the distributed inference database, and v3 represents the layer time utilization weight obtained from the distributed inference database.

[0029] It should be understood that the node computing activity ratio determines the "base number of nodes that can participate in layer computing," and the layer computing parallelism depends on the "node base number to achieve synchronized fragmented computing." A higher node computing activity ratio means more fragments of layer parameters that can be partitioned, resulting in higher layer computing parallelism. Higher layer computing parallelism means shorter "effective computing time" for a single layer, while "invalid time (interference factors)" in the total runtime is zero. Higher layer computing parallelism also means higher layer time utilization. A higher node computing activity ratio indirectly shortens the effective computing time by "driving up layer computing parallelism," and because there are no interference factors, the total runtime is shortened synchronously, maintaining high layer time utilization. Furthermore, there is a positive correlation between the node computing activity ratio and resource utilization in distributed computing; a higher node computing activity ratio means more nodes are more efficient in distributed computing. Cluster computing resources are the carriers of the network. An increased activity ratio means more nodes are transitioning from "idle" to "participating in computation," directly increasing the base of available computing power in the cluster, resulting in a higher distributed computing efficiency value for resource utilization. There is a positive correlation between layer computing parallelism and distributed computing efficiency value for resource utilization. The greater the layer computing parallelism, the more nodes synchronously distribute the computational tasks of a single layer, shortening the layer computation time while fully releasing the computing power of active nodes, resulting in a higher distributed computing efficiency value for resource utilization. There is also a positive correlation between layer time utilization and distributed computing efficiency value for resource utilization. The higher the layer time utilization, the higher the "effective computation ratio" per unit time, avoiding wasted time and ensuring that the improvements in node activity ratio and layer computing parallelism are fully realized, resulting in a higher distributed computing efficiency value for resource utilization.

[0030] As the second step of the large language model distributed inference method based on intra-layer parallelism and communication quantization, it is determined whether to perform distributed computing efficiency regulation based on the distributed computing efficiency value of resource utilization. If yes, the environment adaptation status monitoring step is performed after the distributed computing efficiency regulation is performed. If no, the environment adaptation status monitoring step is performed directly. The execution of distributed computing efficiency regulation means optimizing synchronous communication and computing load balancing in combination with the distributed computing efficiency value of resource utilization. The environment adaptation status monitoring instruction is used to obtain the environment adaptation parameter by reflecting the network environment adaptability of network communication performance. This is used to quantify the compatibility and adaptability of the large language model parallel inference scheme to different network hardware conditions in a distributed computing environment composed of heterogeneous, consumer-grade hardware and standard computer networks, and to obtain the network environment adaptability.

[0031] Furthermore, the specific steps for determining whether to perform distributed computing efficiency tuning are as follows:

[0032] If the distributed computing efficiency value of resource utilization is greater than or equal to the distributed computing efficiency reference value, then distributed computing efficiency adjustment will not be performed.

[0033] If the distributed computing efficiency value is less than the distributed computing efficiency reference value, then distributed computing efficiency regulation will be implemented. Distributed computing efficiency regulation includes synchronous communication optimization regulation and computing load balancing regulation.

[0034] As a further explanation, synchronous communication optimization and control includes dynamic quantization bit width adjustment and set communication algorithm parameter adjustment;

[0035] like Figure 2 This is a flowchart of the dynamic quantization bit width adjustment process of the distributed inference method for large language models based on intra-layer parallelism and communication quantization provided in this embodiment of the invention. During the dynamic quantization bit width adjustment process, the system achieves precise control of the quantization bit width of the activation value by monitoring the relationship between the network bandwidth utilization rate and the preset bandwidth utilization rate reference value in real time. This mechanism can significantly optimize network communication efficiency and computing resource utilization while ensuring that the loss of inference accuracy is controllable. Specifically, the dynamic quantization bit width adjustment is as follows: it is determined whether the network bandwidth utilization rate is higher than the bandwidth utilization rate reference value. If so, the target quantization bit width is obtained by querying the constructed quantization bit width-network state mapping table based on the current single-layer synchronization data volume and computing efficiency offset. The quantization bit width of the activation value is gradually reduced according to the step size corresponding to the obtained target quantization bit width. The quantization bit width-network state mapping table is pre-generated and fixed in the memory of the control node through offline grid search or online Bayesian optimization. This operation can reduce the amount of data synchronized at a single layer, reduce network transmission pressure, lower bandwidth utilization, avoid communication blockage caused by bandwidth saturation, reduce idle time of computing nodes waiting for data, improve the average node computing activity ratio, and effectively alleviate the problem of "communication bottleneck suppressing computing efficiency".

[0036] If not, the target quantization bit width is obtained by querying the constructed quantization bit width-network state mapping table based on the current single-layer synchronization data volume and computational efficiency offset. The quantization bit width of the activation value is gradually increased according to the step size corresponding to the obtained target quantization bit width. This adjustment can reduce quantization error by improving the accuracy of the activation value without increasing the network burden, avoid repeated calculations caused by accuracy loss, and improve the distributed computing efficiency value of resource utilization, so as to achieve the optimization goal of "prioritizing inference accuracy and computational stability when network resources are redundant".

[0037] The computational efficiency offset represents the negative difference between the distributed computing efficiency value and the distributed computing efficiency reference value; the quantization bit width-network state mapping table stores the correspondence between the single-layer synchronization data volume-performance offset combination and the quantization bit width adjustment scheme; the quantization bit width adjustment scheme includes the target quantization bit width and the bit width adjustment step size, and the quantization bit width adjustment must ensure that the adjusted bit width is not lower than the preset minimum quantization bit width safety value.

[0038] In this embodiment, a dynamic balance system of "quantization accuracy - network load - computational efficiency" is constructed by real-time correlation between network bandwidth status and computational efficiency. Its technical effects are reflected in three dimensions: First, it achieves adaptive optimization of communication efficiency through precise response to bandwidth pressure. When network bandwidth utilization exceeds the reference value, the target bit width is matched from the mapping table based on the single-layer synchronous data volume and the computational efficiency offset (the negative difference between the distributed computational efficiency value and the reference value). The activation value quantization bit width is gradually reduced step by step (e.g., from 16 bits to 8 bits), which can reduce the single-layer synchronous data volume, allowing bandwidth utilization to quickly fall back to a safe range, avoiding node idleness caused by communication congestion, and improving the average node computational activity ratio, fundamentally avoiding "communication bottlenecks suppressing computational efficiency." The challenges are: first, achieving a synergistic improvement in accuracy and efficiency when bandwidth is ample—when bandwidth utilization is lower than the control value, the quantization bit width is increased through the same mapping logic (e.g., from 8 bits to 16 bits), reducing quantization errors without increasing network burden, avoiding redundant calculations caused by accuracy loss, improving distributed computing efficiency and ensuring that redundant network resources are converted into inference accuracy; second, through the standardized design of the quantization bit width adjustment scheme (including target bit width, adjustment step size, and minimum safe bit width limit), the inference accuracy loss is always controlled within a preset threshold during dynamic adjustment, while ensuring compatibility with the cluster communication protocol. Ultimately, this improves the stability of inference performance of heterogeneous consumer-grade hardware clusters under network fluctuation scenarios, significantly outperforming the static adaptability of fixed quantization bit width schemes.

[0039] As a further explanation, the dynamic cluster node number adaptation mechanism achieves precise selection of the ensemble communication algorithm by judging the relationship between the node size and the preset reference value in real time. This mechanism can maximize communication efficiency under different node sizes and significantly reduce the impact of synchronization overhead on overall inference performance. The ensemble communication algorithm parameter adjustment is as follows: it determines whether the number of cluster nodes is greater than the reference value. If so, it queries the constructed communication algorithm-node size mapping table based on the node number offset and computational efficiency offset to obtain the target communication algorithm configuration. Based on the obtained target communication algorithm configuration, it activates the tree-structured recursive All-Reduce algorithm. The node number offset represents the positive difference between the cluster node number and the reference value. The communication algorithm-node size mapping table dynamically calibrates the communication algorithm configuration through offline grid search or online Bayesian optimization. This avoids the additional overhead caused by excessive data sharding, shortens the idle time of computing nodes waiting for synchronization, and effectively alleviates the problem of "more nodes dragging down efficiency" in large-scale clusters. For large-scale clusters, the traditional planar ring algorithm's communication steps increase linearly with the number of nodes (O(N)), easily becoming a system bottleneck. The recursive All-Reduce algorithm reduces the number of communication steps to logarithmic levels (O(log₂N)) through binary tree topology. For example, in a 16-node cluster, the communication steps are reduced from 16 to 4. This qualitative change in communication complexity, combined with buffer parameters customized for different scales and environments in the mapping table, enables the system to maintain high scalability as the number of nodes increases, breaking through the technical barrier of poor scalability of traditional methods.

[0040] If not, the target communication algorithm configuration is obtained by querying the existing communication algorithm-node size mapping table, and the planar ring All-Reduce algorithm is activated based on the obtained target communication algorithm configuration. In small-scale clusters (such as 2 or 4 nodes), the communication steps themselves are fewer, and the startup latency and single transmission latency of the algorithm become the main overheads. The planar ring All-Reduce algorithm has a natural advantage in this scenario, and its simple ring topology structure brings lower theoretical latency. By optimizing the buffer size parameter for this scenario, this scheme can minimize communication waiting time, thereby ensuring the overall response speed of the inference task and achieving the best balance between resource utilization and response latency in small-scale heterogeneous clusters.

[0041] The communication algorithm-node size mapping table stores the correspondence between the range of node numbers and the communication algorithm configuration; the communication algorithm configuration includes algorithm type, communication topology, and buffer size parameters; the tree-based recursive All-Reduce algorithm is suitable for large-scale node clusters, reducing communication complexity through binary tree topology; the planar ring All-Reduce algorithm is suitable for small-scale node clusters, achieving low-latency communication through ring topology.

[0042] In this embodiment, a communication efficiency optimization system covering small- to large-scale clusters is constructed through a full-link logic of "node scale awareness - precise algorithm selection - parameter customization and matching." Its core technical effects are as follows: On the one hand, for large-scale cluster scenarios where the number of nodes exceeds the reference value, the target configuration is matched from the mapping table using the node number offset (the positive difference between the actual number of nodes and the reference value) and the computational efficiency offset, enabling a tree-structured recursive All-Reduce algorithm, reducing the linear communication complexity of the traditional planar ring algorithm (O(N)) to the logarithmic level (O(log₂N)). On the other hand, for small-scale clusters where the number of nodes does not exceed the reference value (such as 2-4 nodes), the system effectively optimizes communication efficiency. The algorithm automatically enables the planar ring All-Reduce algorithm on each node, leveraging its low startup latency due to its ring topology. Combined with a small buffer size (e.g., 4MB) optimized by the mapping table, it reduces the number of communication interactions and lowers single-round synchronization latency. In the low-bandwidth environment of consumer-grade Ethernet, it reduces the communication-to-computation ratio, achieving an optimal balance between resource utilization and response latency. Simultaneously, the pre-defined "node number range - algorithm configuration" correspondence in the communication algorithm-node size mapping table supports algorithm switching within a preset time during elastic cluster scaling up / down, with inference performance fluctuations during switching not exceeding a preset fluctuation range. Ultimately, it achieves full-scale coverage from 2 to 32 nodes, improving the overall synchronous communication efficiency of heterogeneous consumer-grade clusters of different sizes. This provides key technical support for the flexible deployment and stable performance of clusters for parallel inference of large language models.

[0043] As a further specific explanation, computational load balancing control includes dynamic weight partitioning adjustment and non-parallelization layer strategy adjustment;

[0044] The dynamic weight sharding adjustment is as follows: Before the root node performs the weight matrix splitting for the first time, it sends a preset computing task to all computing nodes and collects the computing time of each node to complete the preset computing task; the root node calculates the standard deviation of the computing time of all nodes and compares the standard deviation with the preset time standard deviation reference value; the time standard deviation reference value is the computing time standard deviation threshold of the large language model when it runs stably in the heterogeneous cluster.

[0045] If the calculated time standard deviation is greater than the time standard deviation control value, then perform the following operations:

[0046] Based on the root node, the time standard offset and computational efficiency offset of each node are calculated, and the existing load sharding-node computing power mapping table is invoked. This table stores the correspondence between the combination of time standard offset and computational efficiency offset and the weight sharding adaptation specification. The weight sharding adaptation specification includes the dimensional parameters of the weight fragments (i.e., the adjusted weight fragments). (where N' is the adjusted number of fragments).

[0047] The root node adapts to the weight sharding specifications based on the queried weights, and re-shards the weight matrix A of the linear layer along the column dimension to obtain N weight fragments adapted to the computing power of each node. , , ,..., The re-segmented weight fragments are then distributed to the corresponding computing nodes.

[0048] If the calculated time standard deviation is less than or equal to the time standard deviation control value, the initial segmentation specification of the weighted segment is maintained; the load sharding-node computing power mapping table dynamically optimizes the sharding specification through offline grid search or online Bayesian optimization.

[0049] In this embodiment, a dynamic weight sharding adjustment mechanism achieves precise load balancing optimization in a heterogeneous cluster environment. Specifically, during the initialization phase, the system calibrates the computing power of each node through preset computing tasks. When the standard deviation of computing time between nodes exceeds the stable operating threshold, the system automatically triggers a weight sharding reallocation process: based on the time standard offset and computing efficiency offset, it queries the load sharding-node computing power mapping table to obtain the weight sharding specifications adapted to the actual computing power of each node. By non-uniformly re-sharding the column dimensions of the weight matrix, high-performance nodes are assigned larger computing shards, and low-performance nodes are assigned smaller shards. This intelligent sharding strategy effectively eliminates the "barrel effect" in heterogeneous clusters, reduces node waiting time caused by traditional uniform sharding, improves overall resource utilization, and significantly enhances the inference efficiency and stability of large language models on consumer-grade hardware clusters.

[0050] As a further explanation, the non-parallelization layer strategy adjustment is as follows: During the forward propagation of the large language model, the root node collects the computation time and communication time of each parallelizable layer; the root node calculates the computation-communication time ratio of each layer using the formula: Computation-communication time ratio = Layer computation time / (Layer computation time + Layer communication time). The root node then compares the computation-communication time ratio of each layer with a preset computation-communication ratio threshold; the computation-communication ratio threshold is the minimum computation-communication time ratio threshold required for the large language model to maintain efficient parallelism in a distributed cluster. This shifts the system performance analysis from the macro level of the "entire model" to the micro level of the "individual neural network layer." By quantifying the cost-effectiveness of computation and communication at each layer, the system can accurately identify inefficient parallel layers with "small computational load but large communication overhead," providing reliable data support for subsequent precise optimization and overcoming the blindness of the "one-size-fits-all" parallel strategy in traditional distributed inference.

[0051] If the computation-communication time ratio of a certain layer is lower than the computation-communication ratio threshold, the constructed layer parallelism strategy-efficiency mapping table is invoked based on the computation-communication time ratio of that layer and its computational efficiency offset. This mapping table stores the correspondence between the computation-communication time ratio and parallelism strategy adjustment schemes. The parallelism strategy adjustment schemes include parallelization, non-parallelization, and execution node identification. The layer parallelism strategy-efficiency mapping table dynamically optimizes the selection of layer parallelism strategies through offline grid search or online Bayesian optimization. This is the core optimization effect of this scheme. For layers where parallelism yields negative returns (such as some lightweight normalization layers or activation function layers), the synchronization communication overhead caused by forced parallelism far outweighs their computational gains. By converting these layers to independent computation by the root node, the synchronization waiting time of all nodes in that layer is avoided, thereby significantly improving the layer time utilization and node computational activity ratio of the entire cluster.

[0052] The root node adjusts the layer from a parallelized layer to a non-parallelized layer based on the retrieved parallel strategy adjustment plan, and then reclaims the computational tasks of that layer for independent execution by the root node. If the computation-communication time ratio of a certain layer is higher than or equal to the computation-communication ratio threshold, the parallelization strategy for that layer is maintained, and all computing nodes continue to collaboratively execute the parallel computation of that layer, ensuring the accuracy of the optimization measures and avoiding "collateral damage." The system can continuously leverage the advantages of distributed parallel computing to handle core layers with relatively low computational overhead (such as large linear layers and attention mechanisms), thereby concentrating valuable cluster computing resources and network bandwidth on areas that bring the greatest performance improvement, maximizing the overall efficiency of the system.

[0053] In this embodiment, fine-grained performance optimization is achieved in distributed inference through adjustments to the non-parallelization layer strategy. The system delves into performance analysis from the macroscopic level of the model to the microscopic level of individual neural network layers. By collecting and calculating the computation-communication time ratio of each layer in real time, it accurately identifies inefficient parallel layers with low computational load but high communication overhead. When the computation-communication time ratio of a certain layer falls below a threshold, the system queries the layer parallelization strategy-efficiency mapping table based on the computational efficiency offset and automatically adjusts that layer from parallelization to non-parallelization, allowing the root node to independently execute the computation task. This completely eliminates the global synchronization waiting time of that layer, reduces idle time caused by synchronization bubbles, and significantly improves the layer time utilization and node computational activity ratio of the cluster. Simultaneously, the system maintains a parallelization strategy for computationally intensive core layers, ensuring that computational resources are concentrated on high-yield tasks, avoiding over-optimization, and ultimately achieving a significant improvement in overall throughput and response speed, making it particularly suitable for heterogeneous cluster environments with limited network bandwidth.

[0054] It should be explained that the environment adaptation parameters include resource utilization, distributed computing efficiency, single-layer synchronization data volume, and communication-to-computation ratio. The single-layer synchronization data volume refers to the total amount of data a single node needs to send and receive over the network to integrate partial results from all computing nodes during the forward propagation of a large language model when processing a specific layer (e.g., a linear or attention layer in a Transformer block). The communication-to-computation ratio refers to the ratio of the total amount of data transferred between all nodes in the entire cluster to the total number of floating-point computations required to complete a single distributed inference task of a large language model (e.g., processing a given input sequence and generating output).

[0055] Within a pre-defined distributed inference task cycle of a single large language model, the single-layer synchronization data volume of all parallelizable layers is collected, and the maximum value of the single-layer synchronization data volume is calculated as the key synchronization data volume. All parallelizable layers of the large language model are traversed, and the forward propagation computation and communication data volume of each layer are collected. The communication-to-computation ratio is calculated, and the arithmetic mean of the communication-to-computation ratios of all layers is calculated as the average communication-to-computation ratio. The distributed computing efficiency value of resource utilization within the current distributed inference task cycle of the large language model is obtained. The key synchronization data volume, average communication-to-computation ratio, and distributed computing efficiency value of resource utilization are weighted and harmonicly averaged to obtain the network environment adaptability. The constraint expression for network environment adaptability is:

[0056] ;

[0057] In the formula, R represents the network environment adaptability, Eresource represents the distributed computing efficiency value of resource utilization, Dcritical represents the amount of critical synchronization data, Wavg represents the average communication to computation ratio, w1 represents the distributed computing efficiency weight obtained from the distributed inference database, w2 represents the synchronization data weight obtained from the distributed inference database, and w3 represents the communication to computation ratio weight obtained from the distributed inference database.

[0058] It should be understood that the larger the amount of data synchronized at a single layer, the heavier the communication load, the longer the node idle time, and the lower the resource utilization distributed computing efficiency value. A higher communication-to-computation ratio means a higher proportion of communication time, squeezing out effective computation time and further reducing the resource utilization distributed computing efficiency value. A larger amount of data synchronized at a single layer directly increases communication time, thus increasing the communication-to-computation ratio. Simultaneously, there is a positive correlation between resource utilization distributed computing efficiency and network environment adaptability; a higher resource utilization distributed computing efficiency value means that necessary synchronization communication between nodes is completed quickly and does not become a bottleneck for the system. This is because the communication volume is controlled within a range that network bandwidth can easily handle, resulting in higher network environment adaptability. Conversely, there is a negative correlation between the amount of data synchronized at a single layer and network environment adaptability; a larger amount of data synchronized at a single layer makes running distributed inference on a slow network meaningless, even slower than running on a single machine, resulting in lower network environment adaptability. Finally, there is a negative correlation between the communication-to-computation ratio and network environment adaptability; a higher communication-to-computation ratio indicates that network bandwidth is a scarce resource in a cluster composed of consumer-grade devices, leading to lower network environment adaptability.

[0059] As the third step of the large language model distributed inference method based on intra-layer parallelism and communication quantization, it is determined whether to perform environment adaptation control based on the obtained network environment adaptability. If so, the large language model distributed inference output result is sent after the environment adaptation control is performed; otherwise, the large language model distributed inference output result is sent directly. Environment adaptation control means optimizing the cluster network communication topology and transmission protocol in combination with the network environment adaptability.

[0060] Furthermore, the specific steps for determining whether to perform environment adaptation control are as follows: if the network environment adaptability is higher than or equal to the environment adaptability reference value, then environment adaptation control is not performed; if the network environment adaptability is not higher than the environment adaptability reference value, then environment adaptation control is performed. Environment adaptation control includes network topology optimization control and transmission protocol parameter control.

[0061] As a further specific explanation, network topology optimization and control specifically refers to:

[0062] The root node periodically collects bidirectional network latency data between all nodes in the cluster, constructing an N×N inter-node latency matrix, where element L_ij represents the network latency from node i to node j. The coefficient of variation of the latency matrix L is calculated as a quantitative indicator of the network latency difference between cluster nodes, using the formula: Latency Difference = σ(L) / μ(L), where σ(L) is the standard deviation of all elements in matrix L, and μ(L) is the average value of all elements in matrix L. This achieves an accurate measurement of network heterogeneity. Traditional network evaluation often only focuses on average latency or bandwidth, while this method captures both the central tendency and the degree of dispersion of latency through the coefficient of variation, accurately identifying whether there are obvious "slow nodes" in the cluster and the overall uniformity of the network environment, providing a reliable quantitative basis for subsequent optimization.

[0063] The system determines whether the network latency deviation between cluster nodes exceeds a reference value. If so, it queries the pre-built network topology optimization mapping table based on the latency deviation and environment adaptation offset to obtain the target communication topology configuration parameters. The latency deviation represents the positive difference between the network latency deviation between cluster nodes and the reference value, while the environment adaptation offset represents the negative difference between the network environment adaptability and the environment adaptability reference value. The network topology optimization mapping table dynamically optimizes the network topology selection strategy through offline grid search or online Bayesian optimization. This reflects the intelligence of multi-dimensional joint decision-making. The system not only considers the latency difference of the network itself (latency deviation) but also combines the overall adaptability status of the current system (environment adaptability offset). This means that when the overall system efficiency is severely affected by the network environment, a more aggressive topology optimization strategy will be triggered; conversely, a conservative optimization may be adopted. This joint decision-making mechanism avoids unnecessary topology reconstruction under slight network fluctuations, ensuring the cost-effectiveness of optimization measures.

[0064] In this embodiment, the system dynamically reconstructs the physical communication topology of the cluster based on the target communication topology configuration parameters obtained from the query. Through the above process, this solution can achieve real-time matching between the communication topology and the network status. When significant network heterogeneity is detected, the system automatically switches the default fully interconnected topology to a more efficient hierarchical or grouped topology (such as dividing nodes with similar latency into sub-clusters), thereby limiting communication between high-latency nodes to the necessary minimum range while ensuring the communication efficiency between low-latency nodes. This dynamic optimization can reduce the communication latency of cross-node synchronous operations, significantly reduce "pipeline bubbles," and improve the overall throughput of distributed inference, making it particularly suitable for heterogeneous cluster environments composed of consumer-grade devices of different models and locations.

[0065] As a further specific explanation, network topology optimization and control also includes:

[0066] Based on the acquired target communication topology configuration parameters, a K-means clustering algorithm is executed based on the delay matrix to group nodes whose network delay deviations are within a preset delay deviation range into the same communication sub-cluster. The preset delay deviation range represents the closed interval formed by the lower and upper bounds of the delay deviation; this achieves intelligent partitioning of the network topology. Through the clustering algorithm, the system automatically groups nodes with similar delay characteristics into the same sub-cluster, effectively isolating the negative impact of network performance differences. Experimental results show that this strategy can reduce latency fluctuations in cross-node communication, laying a solid foundation for subsequent hierarchical communication architectures.

[0067] A fully interconnected topology is established within each sub-cluster, while backbone communication links are established between sub-clusters, forming a hierarchical hybrid communication topology. This hierarchical architecture achieves an optimal balance between communication efficiency and scalability. The fully interconnected topology within each sub-cluster ensures minimal latency for local communication, while the backbone links between sub-clusters effectively control the complexity of global communication. Test data shows that, compared to the traditional fully interconnected mode, this topology can reduce the synchronous communication overhead of large-scale clusters while maintaining communication efficiency.

[0068] If not, the topology optimization parameters are obtained by querying the constructed network topology optimization mapping table based on the delay deviation correction and environment adaptation offset. The deviation correction represents the negative difference between the network delay deviation between cluster nodes and the corresponding delay deviation value. Based on the obtained topology optimization parameters, while maintaining a fully interconnected topology, a pre-set dynamic weight is assigned to each communication path based on the delay matrix to prioritize the use of low-latency paths for transmitting critical synchronization data. This mechanism achieves intelligent scheduling of communication resources. Through dynamic weight allocation, critical data streams are automatically guided to the optimal communication path, allowing high-priority data to bypass network congestion areas. Actual deployment shows that this optimization can reduce the transmission latency of critical synchronization data while improving network bandwidth utilization, ensuring stable inference performance even in complex network environments. This fine-grained routing optimization is particularly suitable for handling real-time inference tasks, effectively improving the system's response speed and reliability.

[0069] In this embodiment, a significant improvement in distributed cluster communication efficiency is achieved through dynamic optimization of the network topology. First, based on the delay matrix and K-means clustering algorithm, the system intelligently divides nodes with similar latency characteristics into the same communication sub-cluster, effectively isolating the negative impact of network performance differences and laying the foundation for subsequent optimization. Building upon this, a hierarchical hybrid topology structure of "fully interconnected within sub-clusters + backbone links between sub-clusters" is constructed. This ensures minimal local communication latency while effectively controlling global communication complexity. Experimental results show that this structure can reduce the synchronous communication overhead of large-scale clusters while maintaining communication efficiency. When network latency differences are small, the system uses a dynamic weight allocation mechanism to configure differentiated weights for each communication path based on the delay matrix, prioritizing the routing of critical synchronous data to the optimal path. This intelligent scheduling strategy reduces the latency of critical data transmission, improves network bandwidth utilization, and ensures stable inference performance even in complex network environments, making it particularly suitable for the high reliability requirements of real-time inference tasks.

[0070] As further explained in detail, the control of transmission protocol parameters is as follows:

[0071] The system determines whether the network packet loss rate is higher than a reference value. If so, it queries the pre-built transmission parameter optimization mapping table based on the packet loss rate offset and the environment adaptation offset to obtain the throughput adjustment factor. The TCP window width and retransmission timeout are then adjusted according to the obtained throughput adjustment factor. If not, the pre-set transmission protocol configuration is maintained. The packet loss rate offset represents the positive difference between the network packet loss rate and the packet loss rate threshold. The transmission parameter optimization mapping table dynamically calibrates the transmission protocol parameter adjustment strategy through offline grid search or online Bayesian optimization. This enables early and accurate diagnosis of network faults. By jointly analyzing the packet loss rate and the overall system adaptation status, the system can distinguish between temporary fluctuations and persistent congestion, providing a reliable basis for subsequent parameter adjustments and avoiding the problem of delayed response to network conditions in traditional TCP protocols.

[0072] The specific steps for adjusting the TCP window width and retransmission timeout are as follows:

[0073] Determine if the throughput adjustment factor is greater than the throughput adjustment mapping value. If so, query the constructed transmission parameter optimization mapping table to obtain the TCP window width increase and retransmission timeout reduction based on the throughput adjustment correction and environment adaptation offset. Add the current TCP window width and the TCP window width increase to obtain the enhanced TCP window width. Perform difference processing based on the current retransmission timeout and the retransmission timeout reduction to obtain the reduced retransmission timeout. The throughput adjustment correction represents the positive difference between the throughput adjustment factor and the throughput adjustment mapping value.

[0074] If not, the throughput adjustment reference and environment adaptation offset are retrieved from the constructed transmission parameter optimization mapping table to obtain the TCP window width reduction and retransmission timeout increase. The difference between the current TCP window width and the TCP window width reduction is calculated to obtain the reduced TCP window width. The increased retransmission timeout is obtained based on the current retransmission timeout and the retransmission timeout increase. The throughput adjustment reference represents the negative difference between the throughput adjustment factor and the throughput adjustment mapping value. This bidirectional adjustment mechanism achieves precise optimization of transmission performance. Real-world data shows that under good network conditions, the combined strategy of increasing the window and reducing the timeout can improve throughput; while in congested environments, the conservative strategy of decreasing the window and increasing the timeout can reduce invalid retransmissions while maintaining link utilization. This complete control process achieves real-time adaptive matching of transmission protocol parameters and the network environment. Through intelligent parameter coordination, the system fully exploits bandwidth potential when the network is good and ensures transmission stability when the network is congested, enabling the distributed inference system to maintain optimal transmission performance under various network conditions, significantly improving the completion rate and real-time performance of distributed inference tasks in complex network environments.

[0075] In this embodiment, by intelligently sensing network conditions and dynamically adjusting transmission parameters, the communication efficiency and reliability of the distributed inference system in unstable network environments are significantly improved. The system first obtains a precise throughput adjustment factor by querying a transmission parameter optimization mapping table based on a joint analysis of network packet loss rate and environmental adaptability. This enables coordinated optimization of TCP window width and retransmission timeout: when network quality is good, the system automatically increases the TCP window width and reduces the retransmission timeout, thereby increasing network throughput; when network congestion or packet loss occurs, the system intelligently reduces the window width and extends the retransmission timeout, reducing invalid retransmissions while ensuring transmission reliability. This bidirectional adaptive adjustment mechanism allows the system to maintain optimal transmission performance under various network conditions. Real-world testing shows improved stability in effective bandwidth utilization, effectively ensuring the stable execution and real-time response of large language model distributed inference tasks in complex network environments.

[0076] What is needed is, such as Figure 3The diagram shows a pipelined parallel flowchart of a distributed inference method for large language models based on intra-layer parallelism and communication quantization provided in this embodiment of the invention. Specifically, pipelined parallelism, also known as vertical partitioning or inter-layer parallelism, is currently the most widely adopted strategy for distributed execution of large language models on multiple devices. Representative implementations include Petals and the MPI version of Llama.cpp. The core idea of ​​this strategy is to treat the entire neural network structure of the model as a sequentially executed pipeline and to perform "vertical" cutting along the depth direction of the model. Specifically, consecutive layers of the model are divided into different computing nodes; for example, in a cluster consisting of four nodes, node one may be responsible for processing layers 1 to 10, node two for layers 11 to 20, and so on, with node four processing the last layer until the output.

[0077] Its workflow is strictly sequential: the inference process begins at node one, which receives the initial input and computes the layers it is responsible for. After completion, it passes the intermediate results (i.e., activation values) to node two. Only after receiving this data does node two begin computed layers 11 through 20, and after completion, it passes the results to the next node. This chain of "compute-pass" continues until the last node produces the final output.

[0078] However, this seemingly intuitive parallel strategy has a fundamental limitation stemming from its sequential nature. At any given point in time, only one node in the entire cluster is actually performing effective computation, while all other nodes are in an idle waiting state—either waiting for the activation value required for computation from the preceding node, or their computation results are being awaited by the following node. This unavoidable idle gap in the pipeline is figuratively called a "pipeline bubble." Pipeline bubbles cause severe idleness and underutilization of computing resources, greatly limiting the overall inference throughput of the system. More seriously, as the number of nodes in the cluster increases, the proportion of these bubbles in the total processing time rises significantly, leading to a rapid deterioration in scalability. In other words, adding nodes not only fails to bring proportional speedup but may actually exacerbate the inefficiency problem due to the increased coordination and waiting overhead.

[0079] like Figure 4The diagram shows a tensor parallel flowchart of a distributed inference method for large language models based on intra-layer parallelism and communication quantization, provided in this embodiment of the invention. Tensor parallelism is another key model parallelism strategy, with foundational work represented by projects such as Megatron-LM. Unlike pipeline parallelism, which segments the model between layers, tensor parallelism delves into the granularity of parallelism within a single neural network layer, hence it is also called "intra-layer parallelism." Specifically, the large weight matrix (tensor) of a layer (e.g., a linear layer or attention head in a Transformer) is divided into multiple slices along specific dimensions such as rows or columns. These weight slices are then distributed to different computing devices in the cluster, with each device only needing to store and operate on its assigned portion of the model parameters in its local memory.

[0080] This partitioning strategy offers a fundamental advantage: all devices can simultaneously perform computation on their respective shards of the complete input data. This means that in any computational step, every node in the cluster is performing effective work, completely eliminating the "pipeline bubble" problem caused by sequential dependencies in pipelined parallelism, thus achieving true parallel computing and offering significant advantages in computational efficiency.

[0081] However, this high efficiency comes at a cost, with its application severely limited by a key challenge. Since the computation of the next layer in the model typically depends on the complete output of the current layer, the partial results computed by all nodes must be integrated after each parallel computation step. This process requires a collective communication operation called all-reduce, which is responsible for synchronizing and aggregating partial results across all devices. This synchronization step introduces frequent and massive inter-node communication. Each synchronization requires transmitting the entire layer's output activation values, placing extremely stringent demands on the bandwidth and latency of the cluster interconnect network. For this reason, traditional tensor parallelism is almost exclusively limited to high-performance computing clusters equipped with dedicated high-speed interconnect technologies such as NVLink or InfiniBand. For low-cost clusters composed of ordinary consumer-grade devices connected via standard gigabit Ethernet, limited network bandwidth immediately becomes an insurmountable performance bottleneck, and the enormous communication overhead quickly consumes the performance gains from parallel computation, making traditional tensor parallelism impractical in such scenarios.

[0082] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A distributed reasoning method for large language models based on intra-layer parallelism and communication quantization, characterized in that, Includes the following steps: The distributed computing efficiency parameters used to reflect the efficiency of computing resource utilization during the distributed inference process of large language models are obtained. Based on the obtained distributed computing efficiency parameters, the distributed computing efficiency value of resource utilization is obtained, which is used to quantify the degree to which the hardware computing resources in the cluster are effectively utilized in the parallel inference scheme of large language models in a distributed computing environment composed of heterogeneous, consumer-grade hardware and standard computer networks. The system determines whether to perform distributed computing efficiency control based on the distributed computing efficiency value of resource utilization. If yes, the system continues to perform the environment adaptation status monitoring step after performing distributed computing efficiency control. If no, the system directly performs the environment adaptation status monitoring step. The execution of distributed computing efficiency control means optimizing synchronous communication and computing load balancing in combination with the distributed computing efficiency value of resource utilization. The environment adaptation status monitoring instruction is used to obtain environment adaptation parameters by reflecting the network environment adaptability of network communication performance. This allows for the quantification of the compatibility and adaptability of the large language model parallel inference scheme to different network hardware conditions in a distributed computing environment composed of heterogeneous, consumer-grade hardware and standard computer networks, thus obtaining the network environment adaptability. Based on the obtained network environment adaptability, it is determined whether to perform environment adaptation control. If so, after performing environment adaptation control, the output result of the large language model distributed inference is sent. Otherwise, the output result of the large language model distributed inference is sent directly. The environment adaptation control means optimizing the cluster network communication topology and transmission protocol in combination with the network environment adaptability. The obtained distributed computing efficiency parameters include node computing activity ratio, layer computing parallelism, and layer time utilization. Within the preset distributed inference task cycle of a single large language model, the computational activity ratio of all nodes is collected through the performance profiler, and the arithmetic mean of the node computational activity ratios is calculated as the average node computational activity ratio. Traverse all parallelizable layers of the large language model, obtain the layer computational parallelism, and calculate the arithmetic mean of the layer computational parallelism as the average layer computational parallelism. Traverse all parallelizable layers of the model, collect the layer time utilization of each layer, and calculate the arithmetic mean of the layer time utilization as the average layer time utilization rate. The distributed computing efficiency value of resource utilization is obtained by taking the weighted geometric average of the average node computing activity ratio, average layer computing parallelism, and average layer time utilization. The specific steps for determining whether to perform distributed computing efficiency control are as follows: If the distributed computing efficiency value of resource utilization is greater than or equal to the distributed computing efficiency reference value, then distributed computing efficiency adjustment will not be performed. If the distributed computing efficiency value of resource utilization is less than the distributed computing efficiency reference value, then distributed computing efficiency regulation is performed. The distributed computing efficiency regulation includes synchronous communication optimization regulation and computing load balancing regulation.

2. The distributed reasoning method for large language models based on intra-layer parallelism and communication quantization according to claim 1, characterized in that, The synchronous communication optimization and control includes dynamic quantization bit width adjustment and set communication algorithm parameter adjustment; The dynamic quantization bit width adjustment is specifically as follows: determine whether the network bandwidth utilization rate is higher than the bandwidth utilization rate reference value. If so, query the constructed quantization bit width-network state mapping table based on the current single-layer synchronization data volume and computation efficiency offset to obtain the target quantization bit width, and gradually reduce the quantization bit width of the activation value according to the step size corresponding to the obtained target quantization bit width. If not, the target quantization bit width is obtained by querying the constructed quantization bit width-network state mapping table based on the current single-layer synchronization data volume and computational efficiency offset, and the quantization bit width of the activation value is gradually increased according to the step size corresponding to the obtained target quantization bit width. The computational efficiency offset represents the degree of negative deviation between the distributed computing efficiency value and the distributed computing efficiency reference value. The quantization bit width-network state mapping table stores the correspondence between the combination of single-layer synchronization data volume and performance offset and the quantization bit width adjustment scheme; The quantization bit width adjustment scheme includes a target quantization bit width and a bit width adjustment step size, and the quantization bit width adjustment must ensure that the adjusted bit width is not lower than the preset minimum quantization bit width safety value. The quantization bit width-network state mapping table is pre-generated and stored in the control node memory through offline grid search or online Bayesian optimization.

3. The distributed reasoning method for large language models based on intra-layer parallelism and communication quantization according to claim 2, characterized in that, The adjustment of the parameters of the collective communication algorithm is as follows: it is determined whether the number of cluster nodes is greater than the node number reference value. If so, the target communication algorithm configuration is obtained by querying the constructed communication algorithm-node size mapping table based on the node number offset and the computation efficiency offset. The recursive All-Reduce algorithm based on the tree structure is enabled according to the obtained target communication algorithm configuration. The node number offset represents the degree of positive deviation between the number of cluster nodes and the node number reference value. If not, the target communication algorithm configuration is obtained by querying the existing communication algorithm-node size mapping table, and the planar ring All-Reduce algorithm is enabled according to the obtained target communication algorithm configuration. The communication algorithm-node size mapping table stores the correspondence between the range of node numbers and the communication algorithm configuration; The communication algorithm configuration includes parameters such as algorithm type, communication topology, and buffer size. The tree-based recursive All-Raduce algorithm is suitable for large-scale node clusters and reduces communication complexity through binary tree topology. The planar ring All-Reduce algorithm is suitable for small-scale node clusters and achieves low-latency communication through a ring topology. The communication algorithm-node size mapping table dynamically calibrates the communication algorithm configuration through offline grid search or online Bayesian optimization.

4. The distributed reasoning method for large language models based on intra-layer parallelism and communication quantization according to claim 1, characterized in that, The computational load balancing control includes dynamic weight partitioning adjustment and non-parallelization layer strategy adjustment; The dynamic weight sharding adjustment is specifically as follows: before the root node performs the weight matrix splitting for the first time, it sends a preset computing task to all computing nodes and collects the computing time of each node to complete the preset computing task; the root node calculates the standard deviation of the computing time of all nodes and compares the standard deviation with a preset time standard deviation reference value; the time standard deviation reference value is the computing time standard deviation threshold for the stable operation of the large language model in the heterogeneous cluster. If the calculated time standard deviation is greater than the time standard deviation control value, then perform the following operations: Based on the root node, the time standard offset and computational efficiency offset of each node are calculated, and the constructed load sharding-node computing power mapping table is called. The load sharding-node computing power mapping table stores the correspondence between the combination of time standard offset and computational efficiency offset and the weight sharding adaptation specification. The weighted fragment adaptation specification includes the dimension parameters of the weighted fragments; The root node re-segments the weight matrix of the linear layer along the column dimension according to the weight sharding adaptation specification found in the query, and obtains a weight fragment that adapts to the computing power of each node, and distributes the re-segmented weight fragments to the corresponding computing nodes. If the calculated time standard deviation is less than or equal to the time standard deviation reference value, then the initial segmentation specification of the weighted segment is maintained; The load sharding-node computing power mapping table dynamically optimizes the sharding specifications through offline grid search or online Bayesian optimization.

5. The distributed reasoning method for large language models based on intra-layer parallelism and communication quantization according to claim 4, characterized in that, The non-parallelization layer strategy adjustment is as follows: during the forward propagation of the large language model, the root node collects the computation time and communication time of each parallelizable layer. The root node calculates the computation-communication time ratio of each layer and compares the computation-communication time ratio of each layer with a preset computation-communication ratio threshold. The computation-to-communication ratio threshold is the minimum computation-to-communication time ratio threshold for a large language model to maintain efficient parallelism in a distributed cluster. If the computation-communication time ratio of a certain layer is lower than the computation-communication ratio threshold, then the constructed layer parallel strategy-efficiency mapping table is invoked based on the computation-communication time ratio of that layer and the computation efficiency offset. The layer parallel strategy-efficiency mapping table stores the correspondence between the computation-communication time ratio and the parallel strategy adjustment scheme; the parallel strategy adjustment scheme includes parallelization, non-parallelization, and execution node identifier. The layer parallelism strategy-efficiency mapping table dynamically optimizes the selection of layer parallelism strategies through offline grid search or online Bayesian optimization; The root node adjusts the layer from a parallelized layer to a non-parallelized layer according to the parallel strategy adjustment scheme found in the query, and the computation tasks of this layer are returned to the root node for independent execution. If the computation-communication time ratio of a certain layer is higher than or equal to the computation-communication ratio threshold, then the parallelization strategy of that layer is maintained, and all computing nodes continue to collaboratively execute the parallel computation of that layer.

6. The distributed reasoning method for large language models based on intra-layer parallelism and communication quantization according to claim 1, characterized in that, The environment adaptation parameters include resource utilization rate, distributed computing efficiency value, single-layer synchronous data volume, and communication to computing ratio. Within the preset cycle of a single large language model distributed inference task, the amount of single-layer synchronization data of all parallelizable layers is collected, and the maximum value of the single-layer synchronization data is calculated as the key synchronization data amount. Traverse all parallelizable layers of the large language model, collect the forward propagation computation and communication data volume of each layer, calculate the communication to computation ratio, and calculate the arithmetic mean of the communication to computation ratio of all layers as the average communication to computation ratio. Obtain the resource utilization and distributed computing efficiency values ​​within the current distributed inference task cycle of the large language model; The network environment adaptability is obtained by weighting and harmonic averaging the key synchronization data volume, average communication-to-computation ratio, and distributed computing efficiency value of resource utilization.

7. The distributed reasoning method for large language models based on intra-layer parallelism and communication quantization according to claim 1, characterized in that, The specific steps for determining whether to perform environmental adaptation control are as follows: If the network environment adaptability is higher than or equal to the environment adaptability reference value, then environment adaptability control will not be performed; If the network environment adaptability is not higher than the environment adaptability reference value, then environment adaptability control is performed, which includes network topology optimization control and transmission protocol parameter control. The network topology optimization and control specifically refers to: The root node periodically collects bidirectional network latency data between all nodes in the cluster and constructs an inter-node latency matrix; The discrete coefficients of the delay matrix are calculated as a quantitative indicator of the network latency difference between cluster nodes; Determine whether the network latency deviation between cluster nodes is greater than the latency deviation reference value. If so, query the constructed network topology optimization mapping table based on the latency deviation amount and the environment adaptation offset to obtain the target communication topology configuration parameters. The latency deviation amount represents the positive deviation degree between the network latency deviation between cluster nodes and the latency deviation reference value, and the environment adaptation offset represents the negative deviation degree between the network environment adaptability and the environment adaptation reference value. The network topology optimization mapping table dynamically optimizes the network topology selection strategy through offline grid search or online Bayesian optimization.

8. The distributed reasoning method for large language models based on intra-layer parallelism and communication quantization according to claim 7, characterized in that, The network topology optimization and control also includes: Based on the obtained target communication topology configuration parameters, the K-means clustering algorithm is executed based on the delay matrix to divide the nodes whose network delay deviation is within a preset delay deviation interval into the same communication sub-cluster. The preset delay deviation interval represents the closed interval range formed by the lower bound of the delay deviation and the upper bound of the delay deviation. A fully interconnected topology is established within each sub-cluster, and backbone communication links are established between sub-clusters to form a hierarchical hybrid communication topology. If not, the topology optimization parameters are obtained by querying the constructed network topology optimization mapping table based on the delay deviation correction amount and the environment adaptation offset amount. The deviation correction amount represents the degree of negative deviation between the network delay deviation and the delay deviation comparison value between the cluster nodes. Based on the obtained topology optimization parameters, while maintaining the fully interconnected topology, a pre-set dynamic weight is assigned to each communication path based on the delay matrix to prioritize the use of low-latency paths for transmitting critical synchronization data.

9. The distributed reasoning method for large language models based on intra-layer parallelism and communication quantization according to claim 8, characterized in that, The control of the transmission protocol parameters is specifically as follows: Determine whether the network packet loss rate is higher than the packet loss rate reference value. If so, query the constructed transmission parameter optimization mapping table based on the packet loss rate offset and the environment adaptation offset to obtain the throughput adjustment factor. Adjust the TCP window width and retransmission timeout according to the obtained throughput adjustment factor. If not, maintain the preset transmission protocol configuration. The packet loss rate offset represents the degree of positive deviation between the network packet loss rate and the packet loss rate threshold. The specific steps for adjusting the TCP window width and retransmission timeout are as follows: Determine whether the throughput adjustment factor is greater than the throughput adjustment mapping value. If so, query the constructed transmission parameter optimization mapping table to obtain the TCP window width increase and retransmission timeout reduction amount based on the throughput adjustment correction amount and the environment adaptation offset. Add the current TCP window width and the TCP window width increase amount to obtain the enhanced TCP window width. Perform difference processing based on the current retransmission timeout time and the retransmission timeout reduction amount to obtain the reduced retransmission timeout time. The throughput adjustment correction amount represents the degree of positive deviation between the throughput adjustment factor and the throughput adjustment mapping value. If not, the throughput adjustment reference value and the environment adaptation offset are queried from the constructed transmission parameter optimization mapping table to obtain the TCP window width reduction amount and the retransmission timeout time increase amount. The difference between the current TCP window width and the TCP window width reduction amount is processed to obtain the reduced TCP window width. The retransmission timeout time is obtained after gaining based on the current retransmission timeout time and the retransmission timeout time increase amount. The throughput adjustment reference value represents the degree of negative deviation between the throughput adjustment factor and the throughput adjustment mapping value. The transmission parameter optimization mapping table dynamically calibrates the transmission protocol parameter adjustment strategy through offline grid search or online Bayesian optimization.

Images