Distributed reasoning scheduling method, computer device and computer program product

Abstract

The application is suitable for the field of distributed reasoning technology, and provides a distributed reasoning scheduling method, a computer device and a computer program product, comprising: acquiring reasoning concurrency request quantity, computing power of nodes in a DCN, bandwidth of a link and packet loss rate of the link; constructing a non-steady comprehensive quantitative index based on the reasoning concurrency request quantity, the computing power of the nodes, the bandwidth of the link and the packet loss rate; determining a hierarchical splitting ratio according to the non-steady comprehensive quantitative index, an initial splitting ratio and a large model hierarchical task quantity, determining task quantity of each subtask and performing task allocation; receiving a local reasoning result and performing aggregation to obtain a global aggregation result. The method can realize closed-loop perception and adaptive response of a non-steady environment of a DCN, solve the task scheduling imbalance and reasoning quality decline problem caused by computing power fluctuation, traffic burst and link congestion, and realize stable, low-latency and high-reliable distributed reasoning of a hundred-billion-level large model in a dynamic data center network.

Description

Technical Field

[0001] This application belongs to the field of distributed reasoning technology, and in particular relates to a distributed reasoning scheduling method, computer equipment, and computer program product. Background Technology

[0002] With the rapid development of artificial intelligence technology, the application of large models in distributed inference in data center networks is becoming increasingly widespread. Data center networks have a heterogeneous architecture with multiple racks and nodes, including switches, inference servers, and edge nodes. They undertake the entire process of hierarchical inference, subtask scheduling, and result aggregation for large models, and are the key hub connecting computing resources and inference services.

[0003] However, in existing technologies, distributed inference for large data center network models faces the problem of insufficient adaptation to non-steady-state scenarios. This leads to imbalanced subtask scheduling and increased inference interruption rate in scenarios such as dynamic server scaling, link congestion, and traffic bursts, making it difficult to meet industrial-grade requirements for low latency, high accuracy, low interruption rate, and low throughput decay. Summary of the Invention

[0004] In view of this, embodiments of this application provide a distributed inference scheduling method, computer equipment, and computer program product, which realizes closed-loop perception and adaptive response to the non-steady-state environment of DCN, solves the problem of task scheduling imbalance and inference quality degradation caused by computing power fluctuations, traffic bursts, and link congestion, and realizes stable, low-latency, and highly reliable distributed inference of large models of hundreds of billions in dynamic data center networks.

[0005] A first aspect of this application provides a distributed inference scheduling method, including: After detecting the user's inference request, the system obtains the number of concurrent inference requests for the large model, the computing power of each node in the DCN, the bandwidth of each link, and the packet loss rate of each link. An unsteady-state comprehensive quantitative index is constructed based on the concurrent request volume of the large model inference, the computing power of each node, the bandwidth of each link, and the packet loss rate. The hierarchical splitting ratio is determined based on the non-steady-state comprehensive quantization index, the initial splitting ratio, and the hierarchical task of the large model. The workload of each subtask is determined and the tasks are allocated based on the hierarchical splitting ratio, the computing power of each node, and the type of each node. It receives the local inference results sent by each node and aggregates them to obtain the global aggregation result.

[0006] In one implementation of the first aspect, the construction of a non-steady-state comprehensive quantization index based on the concurrent request volume of the large model inference, the computing power of each node, the bandwidth of each link, and the packet loss rate includes: The computing power fluctuation coefficient is calculated based on the computing power of each node and the following formula. :

[0007] in, The set of nodes i in the DCN Let be the computing power of node i at time t. Let i be the computing power of node i at time t-1; The traffic burst coefficient is calculated based on the concurrent request volume of the large model inference and the following formula. :

[0008] in, High concurrency of inference requests for large models To average the number of concurrent requests, , To adjust the sliding window size; The link congestion quantification index is calculated based on the packet loss rate of each link and the following formula. :

[0009] in, This represents the average bandwidth of the link. , is the node at time t i and j The bandwidth of the link between them The average packet loss rate, , For nodes i and j The set of links between them This represents the total number of links. An unsteady-state comprehensive quantitative index is constructed based on the computing power fluctuation coefficient, the traffic burst coefficient, and the link congestion quantification index. ,in, , The weight of the computing power fluctuation coefficient, The weight of the traffic burst coefficient, The weights of the link congestion quantification index.

[0010] In one implementation of the first aspect, determining the hierarchical splitting ratio based on the non-steady-state comprehensive quantization index, the initial splitting ratio, and the large model hierarchical task volume includes: The hierarchical splitting ratio is calculated based on the aforementioned unsteady-state comprehensive quantization index, the initial splitting ratio, the large model hierarchical task volume, and the following formula. :

[0011] in, This is the initial split ratio. It is a non-steady-state comprehensive quantitative index. The total task volume for reasoning the k-th layer of the large model.

[0012] In one implementation of the first aspect, determining the workload of each subtask and allocating tasks based on the hierarchical splitting ratio, the computing power of each node, and the type of each node includes: The allocation of the k-th layer subtask for each node i is calculated based on the computing power of each node and the following formula. :

[0013] Subtasks are assigned to each node i based on subtask allocation constraints, and the formula for the subtask allocation constraints is as follows:

[0014] in Let be the computing power of node i at time t. Let be the computation delay of node i at time t, and , The amount of subtask allocated to the k-th layer corresponding to node i at time t; Calculate the partitioning rationality index based on the allocation of the k-th layer subtasks corresponding to each node i. If the partitioning rationality index is greater than 1, readjust the hierarchical partitioning ratio and return the allocation of the k-th layer subtasks corresponding to each node i based on the computing power of each node and the following formula. The steps.

[0015] In one implementation of the first aspect, the method further includes: The scheduling path cost of each link is calculated based on the computing power of each node, the bandwidth of each link, and the packet loss rate. Construct a reinforcement learning reward function that includes node load balancing, scheduling link utilization, end-to-end latency, and inference interruption rate; Based on reinforcement learning, the scheduling paths for each subtask in the large model inference are calculated with the reinforcement learning reward function as the optimization objective.

[0016] In one implementation of the first aspect, before receiving the local inference results sent by each node and aggregating them to obtain the global aggregation result, the method further includes: The coding mode switching threshold is calculated based on the aforementioned unsteady-state comprehensive quantization index; If the link congestion quantification index is less than the coding mode switching threshold, RS coding is adopted as the target coding mode and the number of RS coding redundancy blocks is configured based on the following formula:

[0017] in, The number of RS-encoded data blocks, The number of redundant blocks, satisfying , The total number of blocks encoded by RS. The average packet loss rate, It is a non-steady-state comprehensive quantitative index; When the link congestion quantization index is not less than the coding mode switching threshold, LDPC coding is adopted and the LDPC coding rate is configured based on the following formula:

[0018] in, For LDPC encoding bitrate, A quantitative indicator for link congestion; The process of receiving local inference results from each node and aggregating them to obtain a global aggregation result includes: The system receives local inference results sent by each node based on the target encoding method and corresponding configuration, and aggregates them to obtain a global aggregation result.

[0019] In one implementation of the first aspect, receiving the local inference results sent by each node based on the target encoding method and corresponding configuration, and aggregating them to obtain a global aggregation result, includes: The coding efficiency evaluation index of the target coding method is calculated based on the following formula. :

[0020] in, The encoding recovery rate of the target encoding method is... The encoding overhead of the target encoding method; The aggregate weight of each node is calculated based on the computing power of each node, the average bandwidth of the associated links of each node, the coding efficiency evaluation index, and the following formula. :

[0021] The reasoning result error compensation term for each node is calculated based on the local reasoning results of multiple historical moments and the global aggregation results of multiple historical moments. Based on the local inference results sent by each of the nodes and the aggregate weights of each of the nodes. The inference results error compensation terms of each node are aggregated with the following formula to obtain the global aggregation result:

[0022] in, The global aggregation result, For DCN node set, For nodes The sent local inference results; For nodes i Error compensation term for the reasoning result.

[0023] In one implementation of the first aspect, after obtaining the global aggregation result through aggregation, the method further includes: The aggregation consistency index is calculated based on the local inference results sent by each node, the inference result error compensation term of each node, the global aggregation result, and the following formula. :

[0024] If the aggregation consistency index is less than the aggregation consistency threshold, the aggregation weights are readjusted.

[0025] A second aspect of this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in any of the first aspects.

[0026] A third aspect of this application provides a computer program product including a computer program that, when run, causes the method described in any of the first aspects to be executed.

[0027] The beneficial effects of the first aspect of this application's embodiments are as follows: By detecting user inference needs, the system acquires the concurrent inference request volume of the large model, the computing power of each node in the DCN, the bandwidth of each link, and the packet loss rate of each link, thus achieving a comprehensive perception of the data center network status. Based on the acquired parameters, a non-steady-state comprehensive quantitative index is constructed. This index integrates the computing power fluctuation coefficient, traffic burst coefficient, and link congestion quantitative index, accurately reflecting the non-steady-state degree of the DCN. This solves the problem in existing technologies where the description of the dynamic changes of the DCN using only a single-dimensional index is inaccurate in characterizing the non-steady-state degree. Based on the non-steady-state comprehensive quantitative index, node computing power, and the large model's layered task volume, a layering splitting ratio is determined. This splitting ratio is dynamically adjusted according to the non-steady-state degree, solving the problem in existing technologies where a fixed task splitting ratio cannot adapt to the characteristics of large model layered tasks and the heterogeneity of DCN nodes. Based on the layering splitting ratio and node type, the sub-task volume is determined and allocated, achieving precise mapping of feature layer tasks to switches, inference layer to servers, and fusion layer to edge nodes, solving the problems of wasted computing power and insufficient computing power in the inference layer. The global aggregation result is obtained by receiving and aggregating the local inference results sent by each node. This aggregation process integrates multi-dimensional weights of node computing power, link bandwidth, and coding efficiency, and introduces an error compensation mechanism to solve the problem of poor aggregation consistency in high packet loss scenarios. This design improves the task scheduling imbalance problem in DCN non-steady-state scenarios. Through a ternary collaborative optimization mechanism, it balances computing power utilization, link traffic, and inference accuracy. As a result, in non-steady-state scenarios such as server scaling and link congestion, the inference accuracy is improved by 5% to 9%, the end-to-end latency is controlled within 1ms, the inference interruption rate is controlled below 1.5%, and the throughput decay rate is controlled within 7%, fully meeting the requirements of industrial-grade deployment. In addition, the solution is compatible with existing hardware facilities, requiring no modification to existing DCN switches, servers, and edge nodes. It can be directly deployed on programmable switches and existing computing power nodes, reducing engineering deployment costs, improving computing resource utilization, adapting to various DCN non-steady-state scenarios, and providing comprehensive technical support for distributed inference of large models in data center networks.

[0028] It is understood that the beneficial effects of the second and third aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram illustrating the implementation flow of the distributed inference scheduling method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the topology of the data center network provided in the embodiments of this application; Figure 3 This is a schematic diagram illustrating the implementation flow of the distributed inference scheduling method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the distributed inference scheduling system provided in the embodiments of this application; Figure 5 This is a schematic diagram of the computer device provided in an embodiment of this application; Figure 6 This is a schematic diagram of a computer program product provided in an embodiment of this application. Detailed Implementation

[0031] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0032] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0033] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0034] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0035] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0036] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0037] As large-scale artificial intelligence models evolve to have hundreds of billions or even trillions of parameters, the computing power of a single node can no longer meet the requirements of low latency, high concurrency, and high reliability for distributed inference of large models. Data center networks (DCNs) have become the core infrastructure for carrying out such tasks. Currently, DCNs frequently experience non-steady-state phenomena such as dynamic scaling of servers, link congestion, traffic bursts, and computing power fluctuations. This leads to imbalances in subtask scheduling, uneven node load, and increased interruption rates in the transmission of local inference results, which in turn causes problems such as decreased global aggregation accuracy and excessive end-to-end latency.

[0038] like Figure 1 As shown in the figure, this application provides a distributed inference scheduling method, including: Step S1: After detecting the user's inference request, obtain the number of concurrent inference requests for the large model, the computing power of each node in the DCN, the bandwidth of each link, and the packet loss rate of each link.

[0039] In applications, the number of concurrent requests for large model inference ( ) refers to the number of large model inference service requests initiated per unit of time, which is the number of requests per second (QPS) and is used to characterize the current business traffic intensity.

[0040] like Figure 2 The diagram shown is a schematic of the DCN (Data Center Network) topology provided in an embodiment of this application. The DCN node set is... Among them, the switch is Inference server is Edge nodes are For example, a better-performing 3-rack architecture... The computing power of each node in DCN ( ) refers to the floating-point computing power that node i can provide at time t, with the unit being FLOPS. Its value varies significantly with the node type ().

[0041] Bandwidth of each link ( () refers to the available transmission rate of the communication link between node i and node j at time t, in Mbps, reflecting the instantaneous throughput capacity of the link.

[0042] Packet loss rate of each link ( The value of is the proportion of data packets lost during the transmission of data packets from node i to node j, and its range is [0, 0.3]. It is used to quantify link reliability.

[0043] For example, in a three-rack DCN architecture (|V| = 18, including 6 P4 programmable switches, 9 A100 inference servers, and 3 Jetson edge nodes), when a user submits a batch inference request for a large model with hundreds of billions of parameters (Q(t) = 5200 QPS), the system completes the collection of the entire network status within 1 ms and obtains the computing power of each switch node. ∈[10 8 10 9 FLOPS, computing power of each server node ∈[10¹ 0 ,10¹¹] FLOPS, bandwidth of each link ∈[200,800]Mbps, packet loss rate of each link ∈[0.002,0.18].

[0044] In applications, users / business terminals (such as servers, smart terminals, business systems, etc.) send explicit inference request instructions to the DCN large model inference service cluster, which include core information such as inference task type, data input, and inference accuracy requirements. After receiving the instruction, the DCN computing power nodes / scheduling nodes determine that the user's inference needs have been detected.

[0045] In application, DCN's edge aggregation nodes / switch nodes collect valid values ​​of the large model inference concurrent request volume Q(t) in real time (i.e., Q(t)>0), and the request volume reaches the preset inference task start threshold (which can be configured according to actual deployment needs). This is determined to be a detection of user inference needs and is suitable for triggering batch inference requests in high-concurrency scenarios.

[0046] Step S2: Construct an unsteady-state comprehensive quantitative index based on the concurrent request volume of the large model inference, the computing power of each node, the bandwidth of each link, and the packet loss rate.

[0047] In application, the nonsteady-state comprehensive quantitative index U(t) is a dimensionless normalized index used to characterize the current overall nonsteady-state degree of DCN. Its value ranges from [0, 1]. The larger the value, the more the system deviates from the steady-state operating conditions.

[0048] Step S3: Determine the layering split ratio based on the non-steady-state comprehensive quantization index, the initial split ratio, and the large model layering task volume.

[0049] In applications, the workload of large-scale model hierarchical tasks ( The term "computational load" (in FLOPs) refers to the computational load of a large-scale model inference task, functionally divided into a feature extraction layer (k=1), a core inference layer (k=2), and a result fusion layer (k=3). These three layers exhibit significant heterogeneity. For example, ; ; .

[0050] The task splitting ratio at level k ( This refers to the proportional coefficient used to distribute the total workload across the DCN node cluster in layers, satisfying... Its value is dynamically adjusted with U(t). The higher U(t) is, the more balanced the split becomes in order to avoid single-point bottlenecks.

[0051] Step S4: Determine the workload of each subtask and allocate tasks based on the hierarchical splitting ratio, the computing power of each node, and the type of each node.

[0052] In applications, the types of nodes include switch nodes ( ), inference server node ( ) and edge nodes ( The three types of nodes have inherent differences in computing power density, memory bandwidth, and network I / O capabilities, forming the basis of DCN heterogeneity; the workload of each subtask refers to the workload of the k-th layer. The result of secondary allocation based on the real-time computing power ratio of node i ensures that high-computing-power nodes bear more computing load; Step S5: Receive the local inference results sent by each node and aggregate them to obtain the global aggregation result.

[0053] In application, local inference results refer to the intermediate inference vectors or tensors output by node i after completing the assigned sub-tasks. Their data format and precision must meet the inter-layer interface protocol of the large model. Global aggregation results refer to the final inference output generated by fusing all local results according to specific rules. It maintains consistency with the original single-node inference results in terms of semantics and numerical precision.

[0054] In application, the global aggregation result of this application is the aggregation object, which is the local reasoning result generated by each node for the same user reasoning needs (such as the feature extraction result of the feature layer, the preliminary reasoning conclusion of the reasoning layer, and the feature fusion result of the fusion layer). The purpose of aggregation is to integrate the local results of hierarchical distributed reasoning to form a complete answer to the user's needs. The aggregation result is delivered directly to the user's needs.

[0055] This application constructs a comprehensive quantitative index for non-steady-state conditions, integrating computing power fluctuations, traffic bursts, and link congestion, by real-time acquisition of concurrent request volume, node computing power, link bandwidth, and packet loss rate. Using this index, combined with the hierarchical task volume of the large model and node computing power, a hierarchical splitting ratio is dynamically generated. Furthermore, considering node type differences, sub-tasks are precisely allocated to switches, servers, and edge nodes according to their capability profiles. Finally, the local results returned by heterogeneous nodes are aggregated to form a highly consistent global output. This technical solution achieves closed-loop perception and adaptive response to the non-steady-state environment of DCN, solving the problems of task scheduling imbalance and inference quality degradation caused by computing power fluctuations, traffic bursts, and link congestion. It provides an feasible technical path for stable, low-latency, and highly reliable distributed inference of trillion-level large models in dynamic data center networks.

[0056] In one embodiment, step S2, constructing a non-steady-state comprehensive quantization index based on the concurrent request volume of the large model inference, the computing power of each node, the bandwidth of each link, and the packet loss rate, includes: Step S21: Calculate the computing power fluctuation coefficient based on the computing power of each node and the following formula. :

[0057] in, The set of nodes i in the DCN Let be the computing power of node i at time t. Let be the computing power of node i at time t-1.

[0058] In applications, the Computing Fluctuation Coefficient is used to characterize the degree of dynamic change of DCN global computing power resources in the time dimension; this technical feature refers to the ratio of the sum of the absolute values ​​of the increase or decrease of computing power of each node per unit time to the total computing power of the previous period.

[0059] In this embodiment, the technical feature is one of the components of the non-steady-state comprehensive quantitative index. The larger the value, the more frequent the server scaling, node failure, or load migration behaviors are, which directly affects the compression adjustment of the subsequent layer split ratio and the scheduling path cost modeling.

[0060] For example, this application can be implemented in a three-rack DCN architecture (15 nodes in total), at any time... The total computing power of each node is FLOPS, until the time Four of the servers experienced changes in computing power due to automatic scaling. , , , With FLOPS and the computing power of other nodes remaining unchanged, the numerator is: Accumulation Substituting into the formula, we get This falls within the range of mild computing power fluctuations, corresponding to a low contribution of computing power disturbances to the non-steady-state comprehensive index.

[0061] Step S22: Calculate the traffic burst coefficient based on the concurrent request volume of the large model inference and the following formula. :

[0062] in, High concurrency of inference requests for large models To average the number of concurrent requests, , This is the size of the sliding window.

[0063] In applications, the Traffic Burst Coefficient is used to characterize the degree to which large model inference requests deviate from the long-term mean over time; this technical feature refers to the relative deviation between the number of concurrent requests at the current moment and the average value of the sliding window.

[0064] In this embodiment, the technical feature is one of the components of the non-steady-state comprehensive quantization index. The larger the value, the more the inference service faces sudden high load pressure, which will directly trigger the dynamic shrinkage of the layer split ratio, the increase of the number of redundant coding blocks, and the re-optimization of the scheduling path.

[0065] For example, this application can be in a sliding window Internal, historical average concurrent request volume QPS, the number of sudden requests detected at the current moment. QPS, then The value is significantly higher than the normal fluctuation threshold, indicating that the system has entered a moderate to severe traffic surge state, which will lead to a corresponding increase in the non-steady-state comprehensive quantitative index. This value drives more aggressive task splitting compression and higher redundancy RS coding configuration. Step S23: Calculate the link congestion quantification index based on the packet loss rate of each link and the following formula. :

[0066] in, This represents the average bandwidth of the link. , is the node at time t i and j The bandwidth of the link between them The average packet loss rate, , For nodes i and j The set of links between them This represents the total number of links.

[0067] In applications, the Link Congestion Quantification Index (RCI) is used to fuse two orthogonal dimensions: DCN link layer bandwidth attenuation and packet transmission unreliability. This technical feature is a dimensionless composite index in the form of a weighted linear combination. The first term reflects the proportion of average available bandwidth to maximum bandwidth attenuation, while the second term reflects the average packet loss level across the entire network.

[0068] In this embodiment, the technical feature is one of the components of the non-steady-state comprehensive quantization index. The larger its value, the worse the stability of the network communication subsystem. It will directly determine the coding mode switching threshold, LDPC coding rate configuration, and the participation strength of the link bandwidth factor in the aggregation weight.

[0069] For example, this application could be a DCN containing |E|=24 links, with the current measured average network bandwidth B(t)=620Mbps, and the maximum link bandwidth... =1000Mbps, average packet loss rate =0.12; Substituting into the formula, we get γ(t)=0.6×(1 0.62)+0.4×0.12=0.228+0.048=0.276, which is in the light congestion range. At this time, the system tends to maintain RS coding as the main method, moderately increase redundant blocks, and maintain a high aggregation weight allocation priority.

[0070] Step S24: Construct a non-steady-state comprehensive quantitative index based on the computing power fluctuation coefficient, the traffic burst coefficient, and the link congestion quantification index. ,in, , The weight of the computing power fluctuation coefficient, The weight of the traffic burst coefficient, The weights of the link congestion quantification index.

[0071] In applications, the Non-stationary Comprehensive Quantification Index is used to achieve unified scale quantization and linear coupling of the three disturbance sources of computing power, traffic, and links. This technical feature refers to the global stability evaluation scalar obtained by summing three dimensionless non-stationary sub-indices with learnable or preset weights.

[0072] In this embodiment, the numerical range of this technical feature is constrained to... Intervals, and with clear hierarchical semantics: when It was determined to be a slightly unsteady state. It is a moderately unsteady state. This level indicates severe non-steady state; the results at this level directly drive hierarchical splitting ratio compression, encoding method switching, and aggregation weight reconstruction.

[0073] For example, this application may be calculated based on the above three calculations. , , Under the premise of equal weighting ,but It falls into the moderately unstable region; therefore, the following will be executed: the encoding switching threshold will be set to... ,current Maintain RS encoding but increase the number of redundant blocks; introduce [a new feature] in the aggregation weight calculation. The coding efficiency is corrected. In application, this application achieves a unified quantification of the non-steady-state characteristics of data center networks through the synergistic effect of computing power fluctuation coefficient, traffic burst coefficient, and link congestion quantification index. Among them, computing power fluctuation coefficient describes the temporal disturbance on the supply side of computing resources, traffic burst coefficient describes the transient impact on the demand side of tasks, and link congestion quantification index describes the transmission reliability degradation on the communication channel side. The non-steady-state comprehensive quantification index is formed by weighted fusion of the three. This not only eliminates the differences in the dimensions and orders of magnitude of the original indicators, but also constructs a stability scale with clear hierarchical semantics, thereby providing a unified, reliable, and traceable basis for subsequent steps and ultimately solving the technical problems of inaccurate single-dimensional evaluation, lagging strategy response, and lack of ternary synergy in existing technologies.

[0074] In one embodiment, step S3, determining the hierarchical splitting ratio based on the non-steady-state comprehensive quantization index, the initial splitting ratio, and the large model hierarchical task volume, includes: Step S31: Calculate the layering splitting ratio based on the non-steady-state comprehensive quantization index, the initial splitting ratio, the large model layering task volume, and the following formula. :

[0075] in, This is the initial split ratio. It is a non-steady-state comprehensive quantitative index. The total task volume for reasoning the k-th layer of the large model.

[0076] Initial split ratio ( This is a newly added technical feature, referring to the preset baseline task allocation strategy when the DCN is in a steady state or slightly unstable state. Its numerical settings need to adapt to the inherent computational characteristics of each layer of the large model and the distribution of node computing power; for example, , , This corresponds to allocating 40% of the workload to the feature layer, 50% to the inference layer, and 10% to the fusion layer.

[0077] Non-steady-state comprehensive quantitative index ( The following is a technical feature already defined and explained in the foregoing embodiments. It is hereby referenced as a known input parameter and will not be explained again. Its function is to characterize the overall unsteady-state intensity of the DCN, and its value range is... The larger the value, the higher the degree of coupling between computing power fluctuations, traffic bursts and link congestion.

[0078] In the application, the calculated layer splitting ratio is normalized, and the sum of the ratios of each layer is 1.

[0079] For example, this application may detect within a certain scheduling period (Moderately non-steady state), initial split ratio is , , The task load of the three layers of the large model is as follows: , , First, calculate the normalization coefficient. Substituting into the formula, we get: , , ; The results were then normalized to make Finally obtained , , The results show that under moderate non-steady-state conditions, the system actively reduces the initial weight allocation of the feature layer and fusion layer while significantly increasing the proportion of the inference layer to ensure stable output of core inference capabilities; this adjustment process is entirely driven by formulas and requires no manual intervention.

[0080] This application provides a steady-state baseline through an initial split ratio, achieves negative dynamic adjustment by using a non-steady-state comprehensive quantization index, and completes normalized weighting by combining the task load of each layer. The three work together to form a closed-loop mechanism of baseline-perturbation-calibration, so that the split ratio of each layer can not only reflect the inherent structural characteristics of the large model, but also respond to the non-steady-state changes of DCN in real time. This provides an accurate, scalable, and conserved upper-level strategy basis for subsequent subtask allocation based on node type and computing power.

[0081] In one embodiment, step S4, determining the workload of each subtask and allocating tasks based on the hierarchical splitting ratio, the computing power of each node, and the type of each node, includes: Step S41: Calculate the allocation amount of the k-th layer subtask corresponding to each node i based on the computing power of each node and the following formula. :

[0082] In this application, the proportion of node i in the total network computing power is used as the weight coefficient for it to undertake the k-th layer sub-task. This mechanism in this embodiment is used to dynamically adapt sub-task allocation to the real-time computing power of nodes, enabling high-computing-power nodes to undertake more sub-tasks and low-computing-power nodes to undertake fewer sub-tasks, thereby avoiding resource idleness or overload caused by static equalization. For example, this application determines the k-th layer sub-task allocation based on the node's real-time computing power proportion; it also determines the k-th layer sub-task allocation based on the proportion of the node's historical average computing power as statistically analyzed using a sliding window; further, it determines the k-th layer sub-task allocation based on the ratio of the node's predicted computing power trend value to the total predicted computing power of the entire network. This application obtains differentiated, scalable, and convergent allocation results of the k-th layer sub-tasks among the nodes based on any of the above methods.

[0083] In the application, k is uniquely determined by the type of node i. For switch nodes, k is 1; for server nodes, k is 2; and for edge nodes, k is 3.

[0084] For example, this application may include three types of nodes (switches) in a DCN. ,server Edge nodes In a typical deployment scenario, let the computing power of each node at the current moment be as follows: , , The total computing power of the entire network is If the total task volume of the second layer (reasoning layer) Corresponding split ratio The number of server nodes allocated is The switch node only allocates This demonstrates a strict positive correlation between computing power and task carrying capacity.

[0085] Step S42: Assign subtasks to each node i based on subtask allocation constraints. The subtask allocation constraint formula is as follows:

[0086] in, Let be the computing power of node i at time t. Let be the computation delay of node i at time t, and , Let be the allocation amount of the k-th layer subtask corresponding to node i at time t.

[0087] In applications, a hard upper bound is modeled on the product of node computation latency and computing power in the subtask allocation constraints. This hard upper bound modeling (i.e. () refers to defining the maximum amount of computation that node i can complete per unit time as its latency capacity. Its physical meaning is: if the theoretical computation latency of node i... Exceeding the system's set end-to-end latency limit (e.g.) If the subtasks are not fully schedulable, they are considered unschedulable. In this embodiment, this modeling transforms abstract computing resources into verifiable latency constraints, ensuring that all allocated subtasks can be completed within the time limit, thus mitigating inference interruptions caused by sudden load spikes. For example, this application determines the upper limit of the total number of tasks a node can handle based on the product of its real-time computing power and the preset maximum computation latency. This application obtains a feasibility assessment basis for subtask allocation that satisfies end-to-end latency guarantees based on any of the above methods.

[0088] For example, this application assumes that node i has the current computing power The system requires a maximum allowable computation delay. Then its latency capacity limit is If the total amount of its three-tiered subtask allocation is calculated as described above... If this is violated, a subsequent adjustment process will be triggered.

[0089] Step S43: Calculate the partitioning rationality index based on the allocation of the k-th layer subtasks corresponding to each node i. If the partitioning rationality index is greater than 1, readjust the hierarchical partitioning ratio and return to step S41. Calculate the allocation of the k-th layer subtasks corresponding to each node i based on the computing power of each node and the following formula. The steps.

[0090] In application, the rationality index is broken down (denoted as...). This is a newly introduced technical feature first introduced in this embodiment, and it is defined as follows: ,in This indicator (splitting rationality indicator) is a dimensionless quantitative parameter representing the ratio of the total number of subtasks allocated to the theoretical maximum number of tasks that the entire network can handle; In this embodiment, this indicator is used to globally verify whether the subtask splitting strategy has exceeded its limits: when This indicates that even under ideal load balancing conditions, the total network computing power cannot be [fully utilized / balanced]. Complete all assigned tasks within the timeframe and reduce the splitting ratio of high-load layers (such as the inference layer). Increase the splitting ratio of low-load layers (such as feature layers). After normalization, the above split rationality verification is performed again.

[0091] For example, this application can be in a 3-rack DCN, where the total number of tasks allocated across the entire network is set to... The total computing power of the entire network is , ,but The constraints are satisfied; if the total allocation increases to [a certain value] after a certain expansion or contraction, [the following condition is met]. ,but ,trigger Reduced by 5% The 5% adjustment was repeated after 1-2 iterations. It converges to ≤1.

[0092] In application, when readjusting the layer splitting ratio, maintain the layer splitting ratio of the fusion layer. Remain unchanged, but change the ratio of the inference layer's hierarchical splitting. Lower the preset ratio, and at the same time, increase the feature layer splitting ratio. Increase the same preset ratio; normalize the adjusted three-layer split ratio to ensure... Substitute the adjusted layering and splitting ratio into the subtask allocation formula and recalculate. Then, the rationality index of the split is checked again until the rationality index of the split is ≤1.

[0093] For example, the preset adjustment ratio is 5%, which is the layering and splitting ratio of the inference layer. Reduce by 5%, the feature layer splitting ratio Increase by 5%. For example, after the initial adjustment, the ratio will be... Become This adjustment, in engineering terms, balances "computing overload mitigation" and "inference accuracy assurance," preventing accuracy loss due to excessive reduction in the inference layer ratio; after adjustment, it will... , , Normalization is performed to ensure that the sum of the three-layer splitting ratios is 1, thus guaranteeing the accuracy of the subtask allocation calculation and conforming to the overall splitting logic of the large model's hierarchical tasks. This application uses subtask allocation... The computing power normalization weighting mechanism, the hard modeling of node latency capacity in subtask allocation constraints, and the rationality indicators of partitioning. The quantitative verification of global task carrying capacity, the synergistic effect of the three, ensures accurate matching between the hierarchical tasks of the large model and the heterogeneity of DCN nodes, while elevating the computation latency constraint from a local constraint of a single node to a feasibility guarantee at the whole network system level. This fundamentally suppresses the inference interruption problem caused by computing power fluctuations in non-steady-state scenarios, and provides a stable, reliable and verifiable subtask allocation foundation for subsequent scheduling path optimization and anti-packet loss aggregation.

[0094] like Figure 3 As shown, in one embodiment, the method further includes: Step S61: Calculate the scheduling path cost of each link based on the computing power of each node, the bandwidth of each link, and the packet loss rate.

[0095] In applications, scheduling path cost refers to a dimensionless normalized index that reflects the coupled influence of node computing power constraints, link transmission capacity constraints, and communication reliability constraints. In this embodiment, scheduling path cost is used as the basis for action selection by the reinforcement learning agent; the smaller the value, the more suitable link ij is as the subtask transmission path.

[0096] In the application, the scheduling path cost of link ij at time t is: ; Among them, path cost weight , , The sum is 1.

[0097] For example, , , .

[0098] Step S62: Construct a reinforcement learning reward function that includes node load balancing, scheduling link utilization, end-to-end latency, and inference interruption rate.

[0099] In applications, node load balancing is defined as follows:

[0100] Node load balancing It refers to the index that characterizes the evenness of the distribution of computing tasks among nodes in a DCN. The larger the value, the smaller the difference in computing load between nodes.

[0101] In applications, the utilization rate of the scheduling link is defined as follows:

[0102] Scheduling link utilization It refers to the proportion of activated links to all physical links in a DCN. The larger the value, the more fully the link resources are utilized.

[0103] In applications, end-to-end latency (Lat(t)) refers to the maximum time delay from the issuance of a subtask to the return of the corresponding local inference result. The smaller the value, the more timely the inference response. In applications, the inference interruption rate (Intr(t)) refers to the proportion of subtasks that fail to execute due to link packet loss, node overload, or unreachable paths. The smaller the value, the more reliable the inference process. In this embodiment, the reinforcement learning reward function (R(t)) is a weighted combination of four-dimensional performance indicators, which is used to guide the reinforcement learning agent to seek the Pareto optimal scheduling strategy in multi-objective conflict scenarios.

[0104] This application constructs a reinforcement learning reward function by, for example, using a linear weighted approach of positive incentives for node load balancing, positive incentives for link utilization, negative penalties for end-to-end latency, and negative penalties for inference interruption rate. For instance, the reinforcement learning reward function (R(t)) is defined as follows:

[0105] Step S63: Based on reinforcement learning, calculate the scheduling path of each subtask in the large model inference with the reinforcement learning reward function as the optimization objective.

[0106] In applications, reinforcement learning can be used as a Deep Q-Network (DQN) algorithm, which models scheduling decisions as Markov Decision Processes (MDPs). The state space includes the computing power Ci(t) of each node, the bandwidth Bij(t) of each link, the packet loss rate ploss,ij(t), and the attributes of the current subtask to be scheduled; the action space is defined as the set of all possible paths between node pairs; and the reward signal is the R(t) constructed above.

[0107] In this embodiment, the reinforcement learning-based computation scheduling path can refer to the output of the state-action value (Q value) corresponding to each feasible path through the DQN network, and the selection of the path with the largest Q value as the optimal scheduling path for the current subtask.

[0108] This application can calculate the scheduling path by, for example, outputting the Q-values ​​of each path based on the forward propagation of the DQN network and selecting actions using an ε-greedy policy; it can also calculate the scheduling path by, for example, decoupling action selection and action evaluation using a dual DQN structure and reducing overestimation bias; furthermore, it can calculate the scheduling path by, for example, modeling state value and advantage function separately using a DuelingDQN structure and improving policy discrimination capability. Based on any of the above methods, this application obtains a sub-task scheduling path that satisfies multi-objective collaborative constraints, achieving four-dimensional joint optimization of computing power, traffic, latency, and reliability.

[0109] In one embodiment, before receiving the local inference results sent by each node and aggregating them to obtain the global aggregation result, step S5 further includes: Step S501: Calculate the encoding method switching threshold based on the non-steady-state comprehensive quantization index.

[0110] Among them, the encoding method switching threshold ( The threshold () is the dynamic boundary for determining link congestion quantification, used to distinguish between mild and severe congestion scenarios. This threshold is a value generated by linear weighting of the non-steady-state comprehensive quantification index U(t), and its expression is as follows: In this embodiment, U(t)∈[0,1], therefore ∈[0.5,0.7]; This threshold is used as the switching criterion between RS coding and LDPC coding in the current step. Its function is to couple the link congestion level with the overall unsteady state of DCN, so that the coding strategy has the ability to jointly respond to the three-element disturbance of computing power fluctuation, traffic burst and link congestion; when γ(t) < When γ(t) ≥ 0, it indicates that the link congestion is within a controllable range, making low-overhead, high-recovery-rate RS coding suitable; when γ(t) ≥ 0. When this occurs, it indicates that the link has entered a state of deep congestion, and LDPC coding, which has stronger error correction capabilities and better redundancy adaptability, needs to be enabled.

[0111] This application, for example, determines the encoding method switching threshold through linear weighting operations based on the non-steady-state comprehensive quantization index U(t), a preset bias term of 0.5, and a weighting coefficient of 0.2. For instance, in a certain scheduling period t, if the system detects U(t) = 0.42, then the following is calculated: =0.5 + 0.2 × 0.42 = 0.584; if the measured link congestion quantification index γ(t) is 0.53, then γ(t) < This triggers the RS encoding path; if U(t) rises to 0.65 in subsequent cycles, then... The value is updated to 0.63, and γ(t) is simultaneously increased to 0.66, which triggers the switch to LDPC encoding.

[0112] Step S502: If the link congestion quantification index is less than the coding mode switching threshold, RS coding is adopted as the target coding mode, and the number of RS coding redundancy blocks is configured based on the following formula:

[0113] in, The number of RS-encoded data blocks, The number of redundant blocks, satisfying , The total number of blocks encoded by RS. The average packet loss rate, It is a non-steady-state comprehensive quantitative index.

[0114] In application, the number of redundant RS-coded blocks (m(t)) is the number of blocks exceeding the original data blocks in the total number of RS-coded blocks. Technically, it represents the number of additional check blocks added to address data loss during transmission. This number is a directly proportional variable that grows in tandem with the average packet loss rate and the non-steady-state comprehensive quantization exponent. Its function is twofold: firstly, by introducing... This enables a direct response to the instantaneous packet loss characteristics of the link; on the other hand, it achieves this through coupling. Incorporating system-level disturbances such as computing power fluctuations and traffic bursts into redundancy configuration decisions avoids excessive or insufficient redundancy caused by relying solely on local link metrics; in this embodiment, These are fixed configuration parameters with values ​​ranging from [3, 10], corresponding to large model hierarchical tasks with different accuracy levels; The output is used to determine the total number of RS-coded blocks. This determines the total amount of data that needs to be transmitted after encoding.

[0115] For example, in this application, at time t, the system measures , ,set up Then the calculation yields Rounded up to 1, this means configuring 1 redundant block. The total number of RS-encoded blocks... This configuration ensures that the receiver can fully recover the original inference result even when at most one data block is lost, under conditions of an average packet loss rate of 12% and moderate non-steadiness.

[0116] Step S503: If the link congestion quantization index is not less than the coding mode switching threshold, LDPC coding is adopted and the LDPC coding rate is configured based on the following formula:

[0117] in, For LDPC encoding bitrate, This is a quantitative indicator for link congestion.

[0118] In applications, the LDPC coding rate (r(t)) characterizes the proportion of effective information bits in the total bitstream after LDPC coding. Its technical meaning is to control the balance between error correction strength and transmission overhead. This rate is a linear function that monotonically decreases with the link congestion quantization index γ(t). Its function is that the more severe the congestion (the larger γ(t)), the lower the rate, meaning more parity bits are inserted to enhance error resilience. Simultaneously, since LDPC decoding complexity is negatively correlated with the rate, this design also considers node-side decoding resource constraints. In this embodiment, γ(t)∈[0,1], therefore r(t)∈[0.5,0.9], satisfying the typical operating range of an industrial-grade LDPC codec. For example, this application uses the link congestion quantization index γ(t) through a linear mapping relationship of 0.9... 0.4γ(t) determines the LDPC coding rate.

[0119] For example, in this application: at time t, the system measures γ(t) = 0.62, then calculates r(t) = 0.9. 0.4 × 0.62 = 0.652; This code rate corresponds to approximately 34.8% of the verification overhead, which ensures a decoding success rate of ≥99.2% while keeping the increase in bandwidth usage within a reasonable range; If γ(t) is further increased to 0.68, then r(t) will decrease to 0.628, and the verification overhead will increase accordingly to cope with more severe packet loss pressure.

[0120] Step S5, receiving the local inference results sent by each node and aggregating them to obtain the global aggregation result, includes: Step S51: Receive the local inference results sent by each node based on the target encoding method and corresponding configuration, and aggregate them to obtain the global aggregation result.

[0121] In application, the target encoding method is based on the above-mentioned γ(t) and The comparison result selects either RS ​​encoding or LDPC encoding; the corresponding configuration is the number of RS redundant blocks m(t) determined above or the LDPC code rate r(t) determined above; the local inference result is the sequence of encoded blocks obtained by each node after completing the sub-task inference and processing by the target encoding method; the technical meaning of this step is to complete the reception, decoding and semantic restoration of the encoded data, and its function is to uniformly map the distributed inference output enhanced by packet loss back to the original semantic space, providing usable input for subsequent aggregation; in this embodiment, all nodes perform local encoding according to the same set of switching logic and configuration rules, and the edge aggregation nodes perform a unified decoding process accordingly to ensure global consistency.

[0122] This application, for example, calls the corresponding decoder module based on the target encoding method identifier (RS / LDPC) to perform the standard RS algorithm or the belief propagation algorithm (LDPC) on the received coded block sequence to restore the original local inference result. This application can also dynamically enable soft-decision decoding or iterative decoding enhancement mode based on the received block integrity verification result to improve the decoding success rate under high packet loss rates. Furthermore, this application can also trigger a retransmission request and temporarily downgrade to a low-complexity encoding strategy when decoding failures are detected for multiple consecutive cycles at a certain node. This application obtains the local inference result after packet loss recovery based on any of the above methods, thereby providing an accurate and complete data foundation for the aforementioned weighted aggregation.

[0123] For example, in this application: at time t, node A sends 6 blocks (5 data blocks + 1 redundant block) using RS encoding. The edge aggregation node actually receives 5 of these blocks (including all 5 data blocks), so it can be directly restored without the redundancy involved. Node B sends a bitstream with a bit rate of 0.652 using LDPC encoding, and receives a total of 1000 bits. After BP decoding, it successfully recovers the original 652-bit inference result. The local results decoded by the two nodes are then sent to the subsequent aggregation process.

[0124] This application switches the threshold through encoding. A mapping relationship is established between the non-steady-state comprehensive quantization index U(t) and the link congestion quantization index γ(t), and this threshold is used to achieve a smooth switch between RS coding and LDPC coding. Based on this, for mild congestion scenarios, the number of redundant RS coding blocks m(t) is used to adjust the average packet loss rate. By jointly modeling with the non-steady-state index U(t), the redundant configuration possesses both link adaptability and system disturbance perception capabilities. For heavily congested scenarios, the LDPC code rate r(t) is used to linearly respond to γ(t), achieving continuously adjustable error correction intensity. Finally, each node completes the encoding and transmission of local inference results based on a unified target encoding method and configuration, and the edge aggregation nodes complete the decoding and reconstruction accordingly, providing reliable input for subsequent multi-dimensional weighted aggregation. This collaborative mechanism effectively solves the technical problem that a single encoding scheme cannot simultaneously handle mild and heavy congestion scenarios, significantly improving the robustness and integrity of distributed inference result transmission under high packet loss environments.

[0125] In one embodiment, step S51, receiving local inference results sent by each node based on the target encoding method and corresponding configuration, and aggregating them to obtain a global aggregation result, includes: Step S511: Calculate the coding efficiency evaluation index of the target coding method based on the following formula. :

[0126] in, The encoding recovery rate of the target encoding method is... The encoding overhead of the target encoding method.

[0127] In applications, coding efficiency evaluation metrics The Encoding Efficiency Evaluation Index (EEI) is a comprehensive quantitative value that characterizes the reliability and effective recovery capability of the currently used coding method for local inference results under actual link packet loss conditions. The numerical range of the EEI is... A larger value indicates a stronger information recovery capability and higher transmission robustness per unit of overhead; In this embodiment, this metric serves as a key input for calculating the aggregate weight, and is used to incorporate link quality (through...) Reflection), inherent performance of the coding mechanism (through) and This reflects the coupling into a unified comparable dimension, thereby supporting the subsequent differentiated weighting processing of heterogeneous coding strategies adopted by different nodes.

[0128] In applications using RS encoding, if the number of received blocks is greater than or equal to the number of RS-encoded data blocks... ,but ,otherwise When using LDPC encoding, , The link congestion quantification metric is determined. Step S512: Based on the computing power of each node, the average bandwidth of the associated links of each node, the coding efficiency evaluation metric, and the following formula, the aggregate weight of each node is calculated. :

[0129] In applications, aggregate weights (AggregationWeight) can refer to the normalization coefficient used to adjust the contribution of the local inference result of node i to the global aggregation; In the relevant technical field, this weight is usually used to achieve weighted fusion of the output results of heterogeneous nodes in a distributed system. Its core function is to suppress the interference of low-quality channels, low computing power or low coding efficiency nodes on the final result. In this embodiment, the weight is introduced by synchronously introducing the node's real-time computing power. Average bandwidth of node-related links Node-level coding efficiency Three positive influencing factors jointly characterize the comprehensive service capabilities of nodes, ensuring that nodes with high computing power, high bandwidth, and high coding efficiency receive higher aggregation weights, thereby improving the accuracy and stability of the global aggregation results.

[0130] Step S513: Calculate the inference result error compensation term for each node based on the local inference results of multiple historical moments and the global aggregation results of multiple historical moments.

[0131] In application, the inference result error compensation term It can refer to historical statistics used to correct systematic deviations in the local inference results of node i; the inference result error compensation term is used to eliminate fixed or trend errors caused by long-term factors such as hardware differences, clock drift, and inconsistent model convergence in distributed inference.

[0132] In this embodiment, the inference result error compensation term is calculated based on the local inference results of node i over a certain number of past time steps (e.g., 5 consecutive time steps). Global aggregation results at the corresponding time point The residuals are averaged to generate a predictive correction for the deviation at the current time step, thereby proactively offsetting cumulative errors before aggregation and improving long-term inference consistency. For example, this application calculates a weighted moving average as an error compensation term based on the difference sequence between the local inference results and the global aggregation results of node i over the past 5 time steps, in order to improve the long-term stability of the global aggregation results.

[0133] Step S514: Based on the local inference results sent by each of the nodes and the aggregation weights of each of the nodes... The inference results error compensation terms of each node are aggregated with the following formula to obtain the global aggregation result:

[0134] in, The global aggregation result, For DCN node set, For nodes The sent local inference results; For nodes i Error compensation term for the reasoning result.

[0135] For example, this application addresses the following scenario: In a typical DCN deployment with 15 nodes (including 3 switches, 9 inference servers, and 3 edge nodes), when the local inference result of a certain inference server node i is detected to be within 5 consecutive time steps... All were lower than the global aggregation results at the corresponding time. 2.3%, then its error compensation term Calculated as Meanwhile, this node suffers from low computing power. FLOPS, average bandwidth of associated links Mbps, RS coding efficiency The aggregate weights are calculated. The remaining nodes are calculated similarly for their respective weights and compensation terms; finally, the weights and compensation terms of all nodes are calculated. The global aggregation result is obtained by weighting the results according to their respective weights. Compared to the aggregation method without error compensation, its average absolute error is reduced by 91% after 1000 iterations.

[0136] This application passes the coding efficiency evaluation index. The actual transmission performance of RS / LDPC encoding used by different nodes is quantitatively modeled using aggregate weights. The node computing power, associated link bandwidth, and coding efficiency are synergistically incorporated into the weight generation process, and an error compensation term for the inference result is introduced. The historical systematic bias is dynamically corrected, and finally based on the weighted compensation aggregation formula. This method achieves high-precision, high-consistency, and drift-resistant global aggregation, effectively solving the aggregation deviation problem caused by the traditional equal-weight aggregation ignoring node heterogeneity and coding differences. It supports the distributed inference scheduling method defined in this application to achieve stable and reliable global inference result output in an unsteady DCN environment.

[0137] In one embodiment, after obtaining the global aggregation result, the method further includes: Step S71: Calculate the aggregation consistency index based on the local inference results sent by each node, the inference result error compensation term of each node, the global aggregation result, and the following formula. :

[0138] Aggregate Consistency Metrics A dimensionless evaluation index used to quantify the maximum relative deviation between the local inference results of each node after error compensation and the global aggregation results; It reflects the fidelity of the global aggregation result's contribution distribution to each node, and its numerical range is... The closer the value is to 1, the more concentrated and consistent the outputs of each node are after error compensation, and the more robust the global aggregation result is; in this embodiment, Its function is to serve as a real-time probe of aggregation quality, providing a basis for determining whether subsequent aggregation weight readjustment is triggered; this indicator directly depends on the results of local inference. Error compensation item Aggregate results with the current round All three are data objects that have been explicitly generated and input into this step in the previous steps, without introducing new data sources or new processing logic.

[0139] This application may, for example, be based on the aggregation consistency index calculation logic, by comparing the absolute deviation between the local results of each node after error compensation and the global results, and normalizing it to the maximum amplitude of the local results in that group, thereby obtaining a scalar index reflecting the overall convergence.

[0140] For example, this application may be performed during a certain scheduling cycle. In this context, DCN contains 5 nodes ( The local inference results of each node after error compensation are as follows: Current global aggregation result If the error compensation term is simplified to 0, then the residuals at each node are: Its absolute value is the largest. The denominator is Substituting into the formula, we get If the threshold requirement is met, weight readjustment will not be triggered.

[0141] Step S72: If the aggregation consistency index is less than the aggregation consistency threshold, the aggregation weight is readjusted.

[0142] The aggregation consistency threshold (also known as the consistency threshold) is a preset threshold value used to determine whether the current aggregation result has reached an acceptable level of consistency. The minimum guarantee requirement for robustness of results based on industrial-grade large-scale model inference scenarios corresponds to a maximum relative deviation of no more than [missing information]. In this embodiment, the threshold is used to establish a decision-response closed-loop control logic. When When this occurs, it indicates that at least one node's compensated output deviates significantly from the global result, caused by instantaneous link jitter, abnormal fluctuations in node computing power, or local encoding failure. In this case, the weight adaptive correction process needs to be initiated to improve aggregation robustness.

[0143] This application may, for example, be based on the comparison result of the aggregation consistency index and a threshold, if it is determined that... If the inference task fails, the subsequent inference task issuance will be suspended, and the weight readjustment process will be executed first. This application may also record consistency failure events as an anomaly log and simultaneously trigger data back transmission from the weight readjustment and performance monitoring modules. Furthermore, this application may also combine the node's historical performance (such as the past 5 periods)... mean Volatility) is used to screen out suspected abnormal nodes, and only their associated weights are partially corrected, rather than being fully recalculated.

[0144] This application obtains the dynamic adjustment action of the aggregation weight based on any of the above methods, which is used to optimize the weighting strategy in a timely manner when aggregation instability is detected, so as to avoid the spread of erroneous results.

[0145] In application, the initial weight ratio is first calculated based on the error value of the results returned by each inference server and the node computing power load ratio. Then, the weight is dynamically corrected by combining the link latency fluctuation data collected by the non-steady state perception module, increasing the aggregation ratio of low latency and low load nodes and decreasing the weight of abnormal nodes.

[0146] For example, this application may be in At that time, it was detected Further analysis revealed that the absolute value of the residual at node 3 reached It far exceeds the other nodes (all less than) Furthermore, the packet loss rate of the link it was on suddenly increased to [a certain value] during that period. (Above the mean) The system determined this to be a momentary communication anomaly and immediately froze the weight of node 3. The weights of the remaining four nodes are normalized and redistributed according to their original proportions, and then redistributed in the next cycle. The system prioritizes sending a lightweight verification task to node 3 to verify its state recovery.

[0147] This application uses the aggregation consistency index. Online computation and threshold determination were used to construct a micro-loop mechanism of aggregation → verification → feedback → readjustment; with the help of The relative deviation between the compensated output of each node and the global result is quantified to evaluate the aggregation quality. Based on this, the aggregation weights are dynamically readjusted according to threshold triggering conditions. This preserves the collaborative foundation of multi-dimensional weights (computing power, bandwidth, coding efficiency) while enhancing its fault tolerance to transient anomalies. Ultimately, without increasing additional communication overhead, the consistency compliance rate of the aggregation results under non-steady-state link jitter scenarios is improved. Upgraded to Readjust average time This satisfies the dual constraints of real-time performance and robustness for distributed inference in large models.

[0148] In one embodiment, the distributed inference scheduling method provided by this social situation embodiment Also includes: The calculation includes the end-to-end delay, which consists of computation delay, communication delay, and encoding delay. .

[0149] Among them, communication latency , To transmit data volume, The encoding delay is [value missing], and the encoding method is RS encoding. When the encoding method is LDPC, the encoding delay is... Calculate the throughput decay rate. : .

[0150] in For maximum concurrent requests, the interruption rate The interruption rate is positively correlated with consistency and the degree of non-steady state.

[0151] Computational resource utilization : .

[0152] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. Figure 4 As shown in the illustration, this application also provides a distributed inference scheduling system 400, comprising: The non-steady-state perception module 401 is used to: after detecting a user's inference request, obtain the large model inference concurrency request volume, the computing power of each node in the DCN, the bandwidth of each link, and the packet loss rate of each link; and construct a non-steady-state comprehensive quantization index based on the large model inference concurrency request volume, the computing power of each node, the bandwidth of each link, and the packet loss rate.

[0153] The hierarchical dynamic splitting module 402 is used to determine the hierarchical splitting ratio based on the non-steady-state comprehensive quantization index, the initial splitting ratio, and the large model hierarchical task volume.

[0154] The ternary collaborative scheduling module 403 is used to determine the workload of each subtask and allocate tasks based on the hierarchical splitting ratio, the computing power of each node and the type of each node.

[0155] The adaptive aggregation module 404 is used to receive the local inference results sent by each node and aggregate them to obtain the global aggregation result. Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 5 As shown, the computer device 5 of this embodiment includes: at least one processor 50 ( Figure 5 (Only one is shown in the diagram) a processor, a memory 51, and a computer program 52 stored in the memory 51 and executable on the at least one processor 50, wherein the processor 50 executes the computer program 52 to implement the steps in any of the above-described distributed inference scheduling method embodiments.

[0156] The computer device 5 is a desktop computer, laptop, handheld computer, cloud server, or other computing device. This computer device may include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will understand that... Figure 5 The computer device 5 is merely an example and does not constitute a limitation on the computer device 5. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0157] The processor 50 is a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0158] In some embodiments, the memory 51 is an internal storage unit of the computer device 5, such as a hard disk or memory of the computer device 5. In other embodiments, the memory 51 is also an external storage device of the computer device 5, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 5. Further, the memory 51 may include both internal and external storage units of the computer device 5. The memory 51 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 51 can also be used to temporarily store data that has been output or will be output.

[0159] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0160] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0161] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0162] like Figure 6 As shown, this application embodiment provides a computer program product 60, including a computer program 52. When the computer program 52 is run, the steps in the above-described distributed inference scheduling method embodiments are executed.

[0163] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / computer equipment, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media are not electrical carrier signals or telecommunication signals.

[0164] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0165] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0166] In the embodiments provided in this application, it should be understood that the disclosed computer devices and methods can be implemented in other ways. For example, the computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed between each other is an indirect coupling or communication connection through some interfaces, devices, or units, and is electrical, mechanical, or other forms.

[0167] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0168] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A distributed reasoning scheduling method, characterized in that, include: After detecting the user's inference request, the system obtains the number of concurrent inference requests for the large model, the computing power of each node in the DCN, the bandwidth of each link, and the packet loss rate of each link. An unsteady-state comprehensive quantitative index is constructed based on the concurrent request volume of the large model inference, the computing power of each node, the bandwidth of each link, and the packet loss rate. The hierarchical splitting ratio is determined based on the non-steady-state comprehensive quantization index, the initial splitting ratio, and the hierarchical task of the large model. The workload of each subtask is determined and the tasks are allocated based on the hierarchical splitting ratio, the computing power of each node, and the type of each node. It receives the local inference results sent by each node and aggregates them to obtain the global aggregation result.

2. The distributed reasoning scheduling method as described in claim 1, characterized in that, The construction of the non-steady-state comprehensive quantitative index based on the concurrent request volume of the large model inference, the computing power of each node, the bandwidth of each link, and the packet loss rate includes: The computing power fluctuation coefficient is calculated based on the computing power of each node and the following formula. : in, The set of nodes i in the DCN Let be the computing power of node i at time t. Let i be the computing power of node i at time t-1; The traffic burst coefficient is calculated based on the concurrent request volume of the large model inference and the following formula. : in, High concurrency of inference requests for large models To average the number of concurrent requests, , To adjust the sliding window size; The link congestion quantification index is calculated based on the packet loss rate of each link and the following formula. : in, This represents the average bandwidth of the link. , is the node at time t i and j The bandwidth of the link between them The average packet loss rate, , For nodes i and j The set of links between them This represents the total number of links. A non-steady-state comprehensive quantitative index is constructed based on the computing power fluctuation coefficient, the traffic burst coefficient, and the link congestion quantification index. ,in, , The weight of the computing power fluctuation coefficient, The weight of the traffic burst coefficient, The weights for the link congestion quantification metrics.

3. The distributed reasoning scheduling method as described in claim 1, characterized in that, The step of determining the hierarchical splitting ratio based on the non-steady-state comprehensive quantization index, the initial splitting ratio, and the hierarchical task volume of the large model includes: The hierarchical splitting ratio is calculated based on the aforementioned unsteady-state comprehensive quantization index, the initial splitting ratio, the large model hierarchical task volume, and the following formula. : in, This is the initial split ratio. It is a non-steady-state comprehensive quantitative index. The total task volume for reasoning the k-th layer of the large model.

4. The distributed reasoning scheduling method as described in claim 1, characterized in that, The process of determining the workload of each subtask and allocating tasks based on the hierarchical splitting ratio, the computing power of each node, and the type of each node includes: The allocation of the k-th layer subtask for each node i is calculated based on the computing power of each node and the following formula. : Subtasks are assigned to each node i based on subtask allocation constraints, and the formula for the subtask allocation constraints is as follows: in Let be the computing power of node i at time t. Let be the computation delay of node i at time t, and , The amount of subtask allocated to the k-th layer corresponding to node i at time t; Calculate the partitioning rationality index based on the allocation of the k-th layer subtasks corresponding to each node i. If the partitioning rationality index is greater than 1, readjust the hierarchical partitioning ratio and return the allocation of the k-th layer subtasks corresponding to each node i based on the computing power of each node and the following formula. The steps.

5. The distributed reasoning scheduling method as described in claim 1, characterized in that, The method further includes: The scheduling path cost of each link is calculated based on the computing power of each node, the bandwidth of each link, and the packet loss rate. Construct a reinforcement learning reward function that includes node load balancing, scheduling link utilization, end-to-end latency, and inference interruption rate; Based on reinforcement learning, the scheduling paths for each subtask in the large model inference are calculated with the reinforcement learning reward function as the optimization objective.

6. The distributed reasoning scheduling method as described in claim 2, characterized in that, Before receiving the local inference results sent by each node and aggregating them to obtain the global aggregation result, the process also includes: The coding mode switching threshold is calculated based on the aforementioned unsteady-state comprehensive quantization index; If the link congestion quantification index is less than the coding mode switching threshold, RS coding is adopted as the target coding mode and the number of RS coding redundancy blocks is configured based on the following formula: in, The number of RS-encoded data blocks. The number of redundant blocks satisfies , The total number of blocks encoded by RS. The average packet loss rate, It is a non-steady-state comprehensive quantitative index; When the link congestion quantization index is not less than the coding mode switching threshold, LDPC coding is adopted and the LDPC coding rate is configured based on the following formula: in, For LDPC encoding bitrate, A quantitative indicator for link congestion; The process of receiving local inference results from each node and aggregating them to obtain a global aggregation result includes: The system receives local inference results sent by each node based on the target encoding method and corresponding configuration, and aggregates them to obtain a global aggregation result.

7. The distributed reasoning scheduling method as described in claim 6, characterized in that, The process of receiving local inference results sent by each node based on the target encoding method and corresponding configuration, and then aggregating them to obtain a global aggregation result, includes: The coding efficiency evaluation index of the target coding method is calculated based on the following formula. : in, The encoding recovery rate of the target encoding method is... The encoding overhead of the target encoding method; The aggregate weight of each node is calculated based on the computing power of each node, the average bandwidth of the associated links of each node, the coding efficiency evaluation index, and the following formula. : The reasoning result error compensation term for each node is calculated based on the local reasoning results of multiple historical moments and the global aggregation results of multiple historical moments. Based on the local inference results sent by each of the nodes and the aggregate weights of each of the nodes. The inference results error compensation terms of each node are aggregated with the following formula to obtain the global aggregation result: in, The global aggregation result, For DCN node set, For nodes The sent local inference results; For nodes i Error compensation term for the reasoning result.

8. The distributed reasoning scheduling method as described in claim 7, characterized in that, After obtaining the global aggregation result through aggregation, the process also includes: The aggregation consistency index is calculated based on the local inference results sent by each node, the inference result error compensation term of each node, the global aggregation result, and the following formula. : If the aggregation consistency index is less than the aggregation consistency threshold, the aggregation weights are readjusted.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 8.

10. A computer program product, characterized in that, Includes a computer program, which, when run, causes the method described in any one of claims 1 to 8 to be performed.

Images