Large model reasoning expert parallel communication optimization method and device and electronic equipment

Abstract

The invention provides a large model reasoning expert parallel communication optimization method and device and electronic equipment, and relates to the technical field of communication. The method comprises the following steps: receiving a full-to-full transmission task; according to the all-to-all transmission task, adding to-be-transmitted traffic to an unfinished transmission traffic queue, executing a traffic scheduling mechanism on the unfinished transmission traffic queue, and determining traffic data volume sent to each network card; determining the pre-distribution credit of the receiving end for each movable connection, and sending the pre-distribution credit of each movable connection to the sending end, so that each network card transmits the traffic data volume according to the corresponding pre-distribution credit; the pre-distribution credit is determined by the receiving end by dynamically adjusting credit distribution based on historical credit use information of each sending end. According to the method and the device, the all-to-all communication efficiency can be improved by uniformly managing and scheduling the intra-machine flow and the network card resources and dynamically adjusting and distributing the credit, so that the reasoning performance of the hybrid expert model is remarkably improved.

Description

Technical Field

[0001] This disclosure relates to the field of communication technology, and in particular to a method, apparatus and electronic device for optimizing parallel communication among large model inference experts. Background Technology

[0002] Currently, in order to support a large number of experts, hybrid expert models adopt an "expert parallelism" strategy, which means that different experts are distributed and deployed on multiple graphics processing units (GPUs).

[0003] Specifically, the aforementioned deployment method generates "all-to-all communication," meaning that each GPU is both a data sender and receiver and needs to establish communication connections with all other GPUs.

[0004] In related technologies, point-to-point transmission relying on traditional aggregated communication libraries is used to achieve full-to-full communication. However, the single-GPU-single-NIC bonding mode of point-to-point transmission in traditional aggregated communication libraries may result in some NICs being idle and underutilized at certain times, leading to a decrease in overall transmission efficiency. In addition, traditional aggregated communication libraries generally adopt multi-track network topologies, and the frequent cross-track communication in expert parallel full-to-full communication causes network congestion, reducing the efficiency of full-to-full transmission communication. Summary of the Invention

[0005] This disclosure provides a method, device, and electronic equipment for optimizing parallel communication of large model inference experts, which can improve the utilization rate of network card resources and the efficiency of all-to-one transmission communication to a certain extent, thereby improving the overall transmission efficiency.

[0006] According to a first aspect of this disclosure, a method for optimizing parallel communication among large-model inference experts is provided, the method comprising: Receive all-to-one transmission tasks; According to the full-to-full transmission task, the traffic to be transmitted is added to the incomplete transmission traffic queue, and a traffic scheduling mechanism is executed on the incomplete transmission traffic queue to determine the amount of traffic data sent by each network card; The receiving end determines the pre-allocated credit for each active connection and sends the pre-allocated credit for each active connection to the sending end, so that each network interface card transmits traffic data according to the corresponding pre-allocated credit; wherein, the pre-allocated credit is determined by the receiving end dynamically adjusting the credit allocation based on the historical credit usage information of each sending end.

[0007] In one possible implementation, the traffic to be transmitted is added to the incomplete transmission traffic queue according to the all-to-all transmission task, and a preset scheduling mechanism is executed on the incomplete transmission traffic queue to determine the amount of traffic data sent by each network interface card, including: Select the first traffic with the smallest remaining data volume in the incomplete transmission traffic queue; When it is determined that there is an available window to send the first traffic, a traffic transmission co-track communication connection is established, and the data volume of the first traffic and the load information of each network card are determined; The data volume of the first traffic is distributed to each network card in proportion to the load information of each network card, so as to determine the traffic data volume sent by each network card.

[0008] In one possible implementation, selecting the first traffic with the smallest remaining data volume in the incomplete transmission traffic queue includes: Based on the shortest task priority scheduling strategy, the first traffic with the smallest remaining data volume is selected from the queue of incomplete transmission traffic.

[0009] In one possible implementation, determining the pre-assigned credits for each active connection by the receiving end includes: The basic credit allocation for each active connection in this round is determined based on the number of active connections currently. Determine the transmission credit utilization rate of each active connection in the previous round among the currently active connections; The pre-allocated credit for each active connection in the current round is determined based on the previous round's transmission credit utilization rate and the basic allocated credit.

[0010] In one possible implementation, the pre-allocated credit for each active connection in the current round is determined based on the previous round's transmission credit utilization rate and the basic allocated credit, including: Determine the preset threshold; The previous round transmission credit utilization rate of each active connection is compared with the preset threshold to obtain multiple comparison results; Based on the obtained multiple comparison results and the basic allocation credit, the pre-allocated credit for each activity connection in this round is determined.

[0011] In one possible implementation, the pre-allocated credit for each activity connection in this round is determined based on the obtained multiple comparison results and the basic allocation credit, including: If the comparison result of the first active connection indicates that the credit utilization rate is less than the preset threshold, then the basic allocated credit is weighted based on the credit utilization rate to determine the pre-allocated credit of the first active connection in this round, and the excess credit is recycled to the credit pool.

[0012] In one possible implementation, the pre-allocated credit for each activity connection in this round is determined based on the obtained multiple comparison results and the basic allocation credit, including: If the comparison result of the second active connection indicates that the credit utilization rate is not less than the preset threshold, then the total credit of the second active connection is calculated. The total amount of credit to be allocated is determined based on the total credit in the second activity connection, the basic allocated credit, and the credit recovered from the credit pool; The total amount of credit to be allocated is distributed proportionally to the second active connection to determine the pre-allocated credit for the second active connection in this round.

[0013] According to a second aspect of this disclosure, a large-model inference expert parallel communication optimization apparatus is provided, the apparatus comprising: The receiving unit is used to receive all-to-one transmission tasks; The scheduling unit is used to add the traffic to be transmitted to the queue of incomplete transmission traffic according to the full-to-full transmission task, and to perform a traffic scheduling mechanism on the queue of incomplete transmission traffic to determine the amount of traffic data sent by each network card. The transmission unit is used to determine the pre-allocated credits of each active connection by the receiving end, and send the pre-allocated credits of each active connection to the sending end so that each network card transmits traffic data according to the corresponding pre-allocated credits; wherein, the pre-allocated credits are determined by the receiving end dynamically adjusting the credit allocation based on the historical credit usage information of each sending end.

[0014] In one possible implementation, the scheduling unit is configured to: Select the first traffic with the smallest remaining data volume in the incomplete transmission traffic queue; When it is determined that there is an available window to send the first traffic, a traffic transmission co-track communication connection is established, and the data volume of the first traffic and the load information of each network card are determined; The data volume of the first traffic is distributed to each network card in proportion to the load information of each network card, so as to determine the traffic data volume sent by each network card.

[0015] In one possible implementation, the scheduling unit is configured to: Based on the shortest task priority scheduling strategy, the first traffic with the smallest remaining data volume is selected from the queue of incomplete transmission traffic.

[0016] In one possible implementation, the transmission unit is used for: The basic credit allocation for each active connection in this round is determined based on the number of active connections currently. Determine the transmission credit utilization rate of each active connection in the previous round among the currently active connections; The pre-allocated credit for each active connection in the current round is determined based on the previous round's transmission credit utilization rate and the basic allocated credit.

[0017] In one possible implementation, the transmission unit is used for: Determine the preset threshold; The previous round transmission credit utilization rate of each active connection is compared with the preset threshold to obtain multiple comparison results; Based on the obtained multiple comparison results and the basic allocation credit, the pre-allocated credit for each activity connection in this round is determined.

[0018] In one possible implementation, the transmission unit is used for: If the comparison result of the first active connection indicates that the credit utilization rate is less than the preset threshold, then the basic allocated credit is weighted based on the credit utilization rate to determine the pre-allocated credit of the first active connection in this round, and the excess credit is recycled to the credit pool.

[0019] In one possible implementation, the transmission unit is used for: If the comparison result of the second active connection indicates that the credit utilization rate is not less than the preset threshold, then the total credit of the second active connection is calculated. The total amount of credit to be allocated is determined based on the total credit in the second activity connection, the basic allocated credit, and the credit recovered from the credit pool; The total amount of credit to be allocated is distributed proportionally to the second active connection to determine the pre-allocated credit for the second active connection in this round.

[0020] According to a third aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method of the first aspect described above and possible implementations thereof.

[0021] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method of the first aspect and possible implementations thereof by executing the executable instructions.

[0022] The technical solution disclosed herein has the following beneficial effects: In this embodiment, a full-to-full transmission task can be received. Then, based on the full-to-full transmission task, the traffic to be transmitted is added to the incomplete transmission traffic queue. A traffic scheduling mechanism is then executed on the incomplete transmission traffic queue to determine the amount of traffic data to be sent by each network interface card (NIC). The traffic scheduling mechanism can be understood as a mechanism for unified scheduling of full-to-full communication traffic and NIC resources, including mechanisms such as dynamic traffic priority scheduling, pre-allocation of internal NIC resources, and load-balanced data transmission. Therefore, this disclosure achieves efficient utilization of internal NIC resources for full-to-full communication through the traffic scheduling mechanism.

[0023] Furthermore, the receiver can determine the pre-allocated credits for each active connection and send these pre-allocated credits to the sender, enabling each network interface card (NIC) to transmit traffic data according to its corresponding pre-allocated credits. The pre-allocated credits are determined by the receiver dynamically adjusting credit allocation based on the historical credit usage information of each sender. In other words, a credit scheduling mechanism is introduced, achieving zero-congestion transmission at the receiver, overcoming the passive speed reduction problem caused by traditional congestion control, thereby significantly improving data transmission performance and optimizing the overall performance of hybrid expert model inference services.

[0024] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objects and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments of this disclosure will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic diagram of a system in this exemplary embodiment is shown; Figure 2 A schematic diagram of a network topology in this exemplary embodiment is shown; Figure 3 A flowchart illustrating a method for optimizing parallel communication among large-scale inference experts in this exemplary embodiment is shown. Figure 4 This illustration shows a process for determining the amount of traffic data sent by each network interface card in this exemplary embodiment. Figure 5 This illustration shows a process diagram of implementing a traffic scheduling mechanism in this exemplary embodiment; Figure 6 This illustration shows a process for determining the pre-allocated credits of each active connection by the receiving end in this exemplary embodiment; Figure 7 A schematic diagram of a credit scheduling mechanism in this exemplary embodiment is shown; Figure 8 This diagram illustrates the structure of a large-model inference expert parallel communication optimization device according to this exemplary embodiment. Figure 9 A schematic diagram of the structure of an electronic device in this exemplary embodiment is shown. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure. Unless otherwise specified, the embodiments and features in the embodiments of this disclosure can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0028] The term "comprising" and any variations thereof in the specification and claims of this disclosure are intended to cover non-exclusive protection. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0029] In this disclosure, there are one or more embodiments; "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0030] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, which are schematic illustrations of this disclosure and are not necessarily drawn to scale. Some block diagrams shown in the drawings may be functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in hardware modules or integrated circuits, or in networks, processors, or microcontrollers. Implementations can be carried out in various forms and should not be construed as limited to the examples set forth herein. The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough description of embodiments of this disclosure. However, those skilled in the art will recognize that one or more specific details may be omitted when implementing the technical solutions of this disclosure, or other methods, components, apparatuses, steps, etc., may be used to replace one or more specific details.

[0031] Currently, the existing Mixture of Experts (MoE) large-scale model inference expert parallel all-to-all communication scheme has the following problems: (1) Insufficient utilization of idle network card resources. Due to the inconsistent data transmission volume of each graphics processing unit (GPU) in the inference scenario, some GPU destinations are located on the machine. Furthermore, the traditional point-to-point transmission single GPU-single network card binding mode of the Collective Communication Library (CCL) will cause some network cards to be idle and not fully utilized at certain times, resulting in a decrease in overall transmission efficiency.

[0032] (2) Congestion within the cross-track communication network. Existing expert parallel all-to-all communication schemes for MoE large model inference generally adopt a multi-track network topology. Frequent cross-track communication in expert parallel all-to-all communication causes congestion within the network and reduces the all-to-all transmission performance.

[0033] (3) Lagging response of end-side congestion control. Congestion is inevitable in all-to-all communication at the end side. Traditional heuristic congestion control algorithms use coarse-grained rate control after congestion occurs, which leads to severe throughput loss in 400G high-speed networks.

[0034] In view of one or more of the above-mentioned problems, an exemplary embodiment of this disclosure provides a method for optimizing parallel communication of large model inference experts. This method receives all-to-all transmission tasks, adds the traffic to be transmitted to an incomplete transmission traffic queue according to the all-to-all transmission tasks, and performs a traffic scheduling mechanism on the incomplete transmission traffic queue to determine the amount of traffic data to be sent by each network interface card (NIC). The traffic scheduling mechanism can be understood as a mechanism for unified scheduling of all-to-all communication traffic and NIC resources, including mechanisms such as dynamic traffic priority scheduling, pre-allocation of internal NIC resources, and load-balanced data transmission. Therefore, this disclosure achieves efficient utilization of internal NIC resources in all-to-all communication through the traffic scheduling mechanism.

[0035] Furthermore, the receiver can determine the pre-allocated credits for each active connection and send these pre-allocated credits to the sender, enabling each network interface card (NIC) to transmit traffic data according to its corresponding pre-allocated credits. The pre-allocated credits are determined by the receiver dynamically adjusting credit allocation based on the historical credit usage information of each sender. In other words, a credit scheduling mechanism is introduced, achieving zero-congestion transmission at the receiver, overcoming the passive speed reduction problem caused by traditional congestion control, thereby significantly improving data transmission performance and optimizing the overall performance of hybrid expert model inference services.

[0036] To better understand the technical solutions provided in the embodiments of this disclosure, the following describes the systems to which the technical solutions provided in the embodiments of this disclosure are applicable. It should be noted that the application scenarios described below are only for illustrating the embodiments of this disclosure and are not intended to limit the scope. In specific implementations, the technical solutions provided in the embodiments of this disclosure can be flexibly applied according to actual needs.

[0037] In order to more clearly describe the technical solutions provided in this disclosure, some technical terms involved in this disclosure are introduced below: Mixture-of-Experts (MoE) is an advanced neural network architecture. Its core components include experts and a gating network. Experts are sub-models specifically designed for particular tasks; each expert excels at handling a specific type of data or task. In large language models, different experts can be responsible for different linguistic phenomena and tasks. The gating network acts as a selector, deciding which tokens are sent to which expert. MoE employs a sparse activation strategy, where each input activates only a small subset of expert parameters in the model. MoE inference refers to the process by which the MoE model dynamically selects specific expert sub-networks for computation when faced with new input data, using the gating mechanism to arrive at the output result.

[0038] In the MoE (Hybrid Expert Model) architecture, "Expert Parallelism" is one of the core distributed strategies supporting efficient model training and inference. Its core idea is to distribute the large number of "expert" sub-models in MoE across different computing devices (such as GPUs). Through collaborative communication and task scheduling between devices, it solves the problem of single devices being unable to handle large-scale expert parameters and computational resource bottlenecks, while maximizing the parallel computing capabilities of multiple devices.

[0039] Collective Communication Library (CCL) is a high-performance collective communication library designed specifically for GPU clusters, primarily used in distributed training and inference scenarios involving multiple GPUs and nodes.

[0040] Graphics Processing Unit (GPU): A processor architecture designed for massively parallel computing, featuring thousands of stream processing cores and a high-bandwidth memory system. It is used to accelerate deep learning training, scientific computing, and graphics rendering tasks, and supports the SIMT (Single Instruction Multithreaded) execution model and the PCIe / NVLink high-speed interconnect protocol.

[0041] All-to-all (ATU) is a distributed system communication mode that requires each node in the cluster to establish a bidirectional data transmission channel with all other nodes. It is mainly used for multi-GPU parameter synchronization and distributed model training scenarios.

[0042] Available Window: This is a core mechanism in the Transmission Control Protocol (TCP) (or similar reliable transmission protocols) used for flow control and congestion control. Essentially, it is the "upper limit of the amount of data that the receiver can currently receive," determining how much data the sender can continuously send without receiving an acknowledgment (ACK).

[0043] The sender end is the logical entity that needs to send data. It can be a software process (such as the process responsible for expert data transmission in MoE inference), a hardware computing unit (such as a GPU), or a combination of both. The core functions of the sender end are to generate the data to be sent and to decide on the sending logic (such as priority and destination).

[0044] The receiver end is the logical entity that needs to receive and process data. It can be a software module (such as the module responsible for receiving and integrating expert data in MoE inference), a hardware computing unit (such as a GPU), or a combination of both. The core functions of the receiver end are to receive data transmitted by the sender end, parse the data format (such as restoring the valid information encapsulated by the protocol), process the data (such as performing calculations and storing data), and make decision feedback logic (such as allocating receiving credits to the sender end based on its own resource status and providing feedback on whether the data reception result was successful).

[0045] Network Interface Card (NIC): This is the hardware interface for interaction between the server / compute node and the network. It is responsible for converting data in the host memory into electrical / optical signals that can be transmitted over the network (transmitting direction), or converting network signals into data and storing them in memory (receiving direction).

[0046] In this embodiment of the disclosure, a single transmitting end (such as a GPU) can simultaneously use multiple network interface cards (NICs) to transmit data. For example, in MoE inference, if a GPU needs to send data to 10 remote GPUs, the data can be split into two NICs and transmitted to different destinations respectively, thereby improving the overall throughput.

[0047] It is evident that the sending end is the source and decision-maker of the data, while the network interface card (NIC) is the executor of data transmission. The two interact through drivers, kernel protocol stacks, or dedicated hardware logic, and the transmission needs of the sending end must be realized through the physical capabilities of the NIC.

[0048] Flow: A core concept describing a set of data packets with the same characteristics, it is the basic management unit for network scheduling, congestion control, and resource allocation. For example, multiple data packets that meet the "same five-tuple (or some characteristics)" will be grouped into the same flow.

[0049] Credit: can be understood as a resource, determined by hardware capabilities (such as network card bandwidth) and network conditions (such as link congestion).

[0050] Active Connection: The "channel / link" (logical or physical layer) for data transmission. It usually exists as a "connection identifier" (such as the TCP four-tuple: source IP / port, destination IP / port). An active connection corresponds to a "sender-receiver pair". In this way, the credit assigned to the sender can be determined based on the credit assigned to the active connection, and thus the credit assigned to the network card can be determined.

[0051] In this embodiment, the receiving end (such as the network card of a server hosting a GPU) needs to complete the credit calculation and allocation for all sending end network cards before each round of transmission. Specifically, the receiving end allocates a "transmission credit" (i.e., the amount of data / data packets allowed to be received) to each sending end network card in advance based on its own processing capacity (such as buffer space and computing load) and network status. The sending end network cards strictly perform transmission according to this credit, thereby avoiding receiver overflow and network congestion caused by the sending end blindly pushing streams. This is especially suitable for fine-grained traffic management in high-parallelism, high-bandwidth (such as 400G) scenarios.

[0052] In this embodiment of the disclosure, the parallel communication optimization technology for large model inference experts can be applied to scenarios such as intelligent computing clusters and MoE large model all-to-all communication, but this embodiment of the disclosure does not limit it.

[0053] Please see Figure 1 As shown, Figure 1 This is a schematic diagram of the framework of an expert parallel all-to-all communication system. The system includes a scheduling layer and a transmission layer.

[0054] In this embodiment, the scheduling layer includes a traffic priority scheduling module, a traffic status management module, and a network interface card (NIC) resource management module. The traffic priority scheduling module prioritizes the transmission of traffic with the smallest remaining data volume at the current moment, based on a shortest task priority scheduling strategy, to minimize completion time. The NIC resource management module distributes traffic transmission tasks to all available NICs within the machine and establishes same-track communication, eliminating cross-track communication. Simultaneously, it dynamically adjusts the traffic transmission ratio undertaken by each NIC link based on its real-time load status, fully aggregating internal network bandwidth resources to achieve the shortest overall traffic transmission time. The traffic status management module maintains an active traffic bitmap to support ordered reordering of multi-path transmission traffic, ensuring the integrity and orderliness of data transmission.

[0055] It is evident that the scheduling layer can achieve unified management and scheduling of all traffic and internal network card resources, thereby achieving optimal traffic completion time, efficient utilization of internal network card resources, and avoiding congestion within the cross-rail communication network.

[0056] In this embodiment, the transport layer includes a credit transmission control module and a dynamic credit allocation module. The credit transmission control module transmits traffic based on the credit limit allocated to the receiver, achieving zero congestion at the receiver. The dynamic credit allocation module dynamically allocates credit limits based on the weighted least-connection scheduling algorithm, according to the actual transmission capacity of each sender, maximizing the utilization of bandwidth resources at the receiver.

[0057] It is evident that the transport layer can maximize the utilization of the receiver's bandwidth resources and achieve zero congestion and efficient transmission of traffic.

[0058] In this embodiment of the disclosure, the aforementioned expert parallel all-to-all communication system can be integrated into an electronic device (or a computing node if the electronic device is in a cluster), thereby enabling communication optimization of the hybrid expert model based on the electronic device.

[0059] In this embodiment, the CCL plugin can be integrated into the CCL as an expert parallel all-to-all transmission backend. That is, the solution provided in this embodiment does not require modification of existing large-model inference cluster equipment; it is integrated and deployed through a collection of communication library plugins, resulting in low deployment costs.

[0060] See Figure 2 The diagram shows a network topology for a large model inference cluster provided in an embodiment of this disclosure. The aforementioned topology provided in this embodiment of the disclosure... Figure 1 The system shown can be applied to the network topology of this large model inference cluster.

[0061] exist Figure 2 The network topology shown adopts a typical "Spine-Leaf" topology and includes a multi-Pod (a logical unit of a data center, which can be understood as a collection of servers and related network devices) design.

[0062] in, Figure 2 The Spine layer, located at the top of the architecture, contains multiple spine switches from Spine0 to SpineN. As the core switching layer, it connects Leaf switches across different Pods, enabling communication between Pods and providing global connectivity within the data center. All Leaf switches are fully interconnected with the Spine switches, ensuring that traffic between any Leaf can be forwarded through the Spine layer.

[0063] Figure 2 The leaf layer in the Pod acts as an access layer switch, connecting the server nodes within the Pod. It is responsible for aggregating server traffic and forwarding it to the Spine layer. It also receives cross-Pod traffic from the Spine layer and distributes it to the corresponding servers. Each Pod has multiple leaf switches, from Leaf0 to Leaf7.

[0064] Figure 2 In this context, Node represents a server node in the data center (such as Node0 or NodeX), which is the core entity that runs business applications and stores data. Figure 2Each Node in the system contains multiple remote network interface cards (RNICs) from RNIC0 to RNIC7. The RNICs are responsible for the network connection between the server and the Leaf switch, enabling the server to access the network and communicate with other servers or external networks through the Leaf and Spine layers.

[0065] Figure 2 The logical unit (Pod) in the data center is a logical partition. Each Pod contains a set of Leaf switches, server nodes and a matching RNIC.

[0066] In this embodiment of the disclosure, based on the foregoing Figure 2 The network topology shown can support the high bandwidth and low latency network requirements of scenarios such as cloud computing and big data.

[0067] To further illustrate the technical solutions provided by the embodiments of this disclosure, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this disclosure provide method operation steps as shown in the following embodiments or drawings, the method may include more or fewer operation steps based on conventional or non-inventive methods. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided by the embodiments of this disclosure. In actual processing or when the device executes the method, it may be executed sequentially or in parallel according to the method shown in the embodiments or drawings.

[0068] Please see Figure 3 , Figure 3 This is a flowchart illustrating a method for optimizing parallel communication among large-scale model inference experts, as described in an embodiment of this disclosure. The specific implementation process of this method is as follows: Step 301: Receive the full-to-full transmission task.

[0069] In this embodiment of the disclosure, during MoE inference, the gating network selects experts distributed across nodes for each token. For example, the token of node A needs to be processed by experts at nodes B and C, while the experts at node A also need to process the tokens of nodes D and E.

[0070] At this point, the inference framework needs to issue a full-to-full transmission task, so that each node can send its local token to the node where the target expert is located, and also receive tokens from other nodes to the local expert, ultimately achieving accurate matching of all tokens with the target expert, so as to realize cross-node collaboration of experts in MoE inference.

[0071] As can be seen, in this embodiment of the disclosure, the inference framework can... Figure 2The cluster shown in the diagram initiates a collaborative task in which each node needs to transmit data bidirectionally to all other nodes. In other words, the inference framework issues a full-to-full transmission task, so that the scheduling layer of the system deployed in the electronic device can receive the full-to-full transmission task issued by the inference framework.

[0072] Step 302: Add the traffic to be transmitted to the incomplete transmission traffic queue according to the full-to-full transmission task, and perform a traffic scheduling mechanism on the incomplete transmission traffic queue to determine the amount of traffic data to be sent by each network card.

[0073] In this disclosure embodiment, see Figure 4 As shown, the scheduling layer can determine the amount of traffic data sent by each network interface card using, but is not limited to, the following steps: Step 401: Select the first traffic with the smallest remaining data volume in the incomplete transmission traffic queue; In this embodiment of the disclosure, the shortest task priority scheduling strategy can be used to select the first traffic with the smallest remaining data volume from the unfinished transmission traffic queue.

[0074] The shortest task first scheduling strategy is a scheduling mechanism aimed at optimizing data transmission efficiency. Its core logic is to minimize the overall completion time of all flows by prioritizing the transmission tasks (flows) with the least remaining data. The first flow can be understood as an independent transmission task (flow) with the smallest remaining data and a complete lifecycle.

[0075] In other words, when multiple transmission tasks compete for the same bandwidth resource (such as the same network card or the same link), the scheduling layer always prioritizes the transmission task (flow) with the least amount of remaining data to occupy bandwidth for transmission. This minimizes the total time for all concurrent flows to be transmitted from start to finish. In other words, by quickly clearing smaller flows to reduce the backlog of tasks, the system avoids situations where large flows occupy bandwidth for extended periods, causing smaller flows to be unable to complete, ultimately reducing the overall long-tail latency of the transmission.

[0076] Step 402: Determine if there is an available window to send the first traffic.

[0077] In this embodiment, the available window of an onboard network interface card (such as a server's RNIC or a GPU's dedicated communication network card) can be understood as the upper limit of the amount of data that can be immediately sent, jointly maintained by the network card hardware and the driver. It can also be understood as the maximum amount of data that the receiving end can currently buffer (i.e., how many more bytes of data can be received without overflowing). Thus, the scheduling layer can compare the upper limit of the data amount (or the maximum data amount) with the size of the first flow, i.e., the transmission task (flow). If it is determined that the upper limit of the data amount is greater than the size of the first flow, then it is determined that there is an available window to send the first flow.

[0078] Step 403: When it is determined that there is an available window to send the first traffic, establish a traffic transmission co-track communication connection, and determine the data volume of the first traffic and the load information of each network card.

[0079] In this embodiment of the disclosure, to avoid multiple transmission tasks being transmitted across tracks, for example across... Figure 2 The Spine-Leaf architecture shown reduces the high latency and congestion risks of cross-track communication by connecting different Pods across different physical links, thereby improving transmission efficiency and stability. It can establish same-track communication connections for traffic transmission, so that all transmission links of the same batch of traffic to be transmitted are restricted to the same network track to complete data interaction.

[0080] In this embodiment, establishing a co-track communication connection for traffic transmission, i.e., establishing a co-track traffic transmission channel on all network interface cards (NICs), can be divided into the following steps: Step 1: Static binding of network tracks and NICs. Step 2: Dividing traffic according to the track where the target receiving end is located. Step 3: Allocating intra-track NIC channels for co-track traffic. Step 4: Transmission verification and dynamic maintenance.

[0081] For example, deploying the same group of experts on GPU servers within the same track ensures that the token transfer tasks (to which tokens need to be transferred) for these experts can be transmitted through the same track channel; when the gating network selects experts for tokens, it prioritizes allocating experts within the same track, reducing the number of cross-track transfer tasks, thereby improving the overall speed of MoE inference.

[0082] Step 404: Distribute the first traffic data volume to each network card in proportion to the load information of each network card to determine the traffic data volume sent by each network card.

[0083] In this embodiment of the disclosure, after determining the load information of the network card, the data volume of the first traffic can be distributed to each network card in proportion to the load information of each network card, so as to determine the traffic data volume sent by each network card.

[0084] In this embodiment of the disclosure, to avoid system blockage caused by infinite waiting and to ensure that the communication or scheduling process can recover from abnormal states, a timer can be set and a timer can be waited for to expire. Waiting for the timer to expire can be understood as the scheduling layer triggering a preset timeout processing logic after a pre-defined waiting time window for a certain operation (such as data transmission confirmation) has expired and no expected response signal (such as confirmation from the receiving end) has been received.

[0085] Optionally, the timer timeout period is set to the ideal transmission time for this traffic, i.e., T = data / all_NIC_BW. If there is no traffic transmission this time, T = 1 / 2. RTT. RRT stands for Round-Trip Time.

[0086] In this embodiment of the disclosure, step 401 can be understood as the scheduling layer performing dynamic scheduling of traffic priorities, steps 402-403 can be understood as the scheduling layer performing pre-allocation of network card resources within the machine, and step 404 can be understood as the scheduling layer performing data transmission balancing.

[0087] As can be seen, the embodiments of this disclosure design a unified scheduling scheme for intra-machine all-to-all communication traffic and network cards, thereby solving the problems of frequent cross-homing transmission and low communication efficiency using a single network card in existing all-to-all communication solutions. Specifically, based on the shortest task priority scheduling strategy, intra-machine traffic transmission can be uniformly scheduled to achieve optimal traffic completion time; through network card resource pre-allocation processing, intra-machine network card same-track transmission and zero cross-track communication can be achieved; and traffic transmission fully utilizes all intra-machine network card resources, significantly improving the efficiency of all-to-all transmission.

[0088] In this disclosed embodiment, in conjunction with the foregoing Figure 1 The system shown includes modules in its scheduling layer. A description of the full-to-full traffic transmission scheduling process in the scheduling layer is provided. (See [link to documentation]). Figure 5 , Figure 5 This is a flowchart of the full-to-full traffic transmission scheduling process of the scheduling layer in this embodiment of the disclosure.

[0089] exist Figure 5In this process, the traffic priority scheduling module first selects the flow with the smallest remaining data volume from the queue of currently incomplete transmission flows, based on the shortest task priority scheduling strategy, to achieve optimal flow completion time. The network interface card (NIC) resource management module establishes a co-transmission channel for all NICs and determines the amount of data (d) that the flow can transmit at the current moment based on the available NIC window, and feeds back the load of each NIC to the traffic priority scheduling module. Therefore, the traffic priority scheduling module, based on the transmission information fed back by the NIC resource management module, distributes the data volume (d) of the flow to each NIC for transmission proportionally according to the NIC load, achieving efficient utilization of the internal NIC resources. Furthermore, a scheduling timer timeout period T = d / all_NIC_BW can be set, and the above process is repeated when the timer is triggered.

[0090] As can be seen, the embodiments disclosed herein adopt a unified scheduling scheme for all-to-all traffic and network card resources within the machine to achieve near-theoretically optimal all-to-all traffic transmission scheduling.

[0091] Step 303: Determine the pre-allocated credits for each active connection at the receiving end, and send the pre-allocated credits for each active connection to the sending end so that each network card can transmit traffic data according to the corresponding pre-allocated credits.

[0092] In this embodiment of the disclosure, the receiving end can adopt a dynamic credit allocation algorithm to dynamically adjust the credit allocation of the sending end according to the actual sending capacity of the sending end, thereby avoiding bandwidth waste in the traditional equal-sharing algorithm.

[0093] In this disclosure embodiment, see Figure 6 As shown, the following steps can be used, but are not limited to, to determine the pre-allocated credits for each active connection at the receiving end: Step 601: Determine the basic credit allocation for each active connection in this round based on the number of current active connections; Step 602: Determine the previous round's transmission credit utilization rate for each active connection in the currently active connections; Step 603: Determine the pre-allocated credits for each active connection in this round based on the previous round's transmission credit utilization rate and basic allocated credits.

[0094] In this embodiment of the disclosure, a preset threshold can be determined; the transmission credit utilization rate of each active connection in the previous round is compared with the preset threshold to obtain multiple comparison results; based on the multiple comparison results and the basic allocated credit, the pre-allocated credit for each active connection in the current round is determined.

[0095] Optionally, if the comparison result of the first active connection indicates that the credit utilization rate is less than a preset threshold, the basic allocated credit is weighted based on the credit utilization rate to determine the pre-allocated credit for the first active connection in this round, and the excess credit is recycled to the credit pool.

[0096] Optionally, if the comparison result of the second active connection indicates that the credit utilization rate is not less than a preset threshold, then the total credit of the second active connection is calculated; based on the total credit of the second active connection, the basic allocated credit, and the credit recovered from the credit pool, the total amount of credit to be allocated is determined; the total amount of credit to be allocated is distributed proportionally to the second active connection to determine the pre-allocated credit of the second active connection in this round.

[0097] As can be seen, in this embodiment, a credit-based scheduling mechanism is used to achieve zero congestion at the receiver in all-to-all communication. Specifically, the credit scheduling mechanism can be understood as the receiver dynamically adjusting the credit allocation of the transmitter based on the transmitter's actual transmitting capacity, thus avoiding bandwidth waste in traditional equal-sharing algorithms.

[0098] In this disclosed embodiment, in conjunction with the foregoing Figure 1 The system shown includes modules in the transport layer. The dynamic credit allocation process at the transport layer receiver is described in the following section. Figure 7 , Figure 7 This is a flowchart of the dynamic credit allocation process at the transport layer in this embodiment. In this embodiment, the algorithm timer timeout T is set to the link RTT; the threshold γ is set to 0.9.

[0099] exist Figure 7 In this context, the basic credit allocation for the sender in round (i) can be calculated based on the current number of active connections N. The basic credit allocation is described using `credit_allocate`, and the total credit is described using `all_credit`, thus determining that `credit_allocate(i) = all_credit / N`. Therefore, the basic credit allocation can be determined by evenly distributing the total credit based on the current number of active connections.

[0100] exist Figure 7 In the process, the receiving end calculates the transmission credit utilization rate α of each active connection(s) in the previous round (i-1). The credit used by each active connection(s) in the previous round (i-1) is described by credit_use(i-1,s), and the basic allocated credit of each active connection(s) in the previous round (i-1) is described by credit_allocate(i-1,s). Therefore, the transmission credit utilization rate α(i-1,s) of each active connection(s) in the previous round (i-1) can be determined as α(i-1,s) = credit_use(i-1,s) / credit_allocate(i-1,s).

[0101] Please continue reading Figure 7 Once the credit utilization rate and basic credit allocation are obtained, the credit allocation can be dynamically adjusted.

[0102] Optionally, the receiver iterates through each active connection. For connections (y) with a credit utilization rate α < γ, it calculates the actual credit allocation for this round (i) based on α weighting: credit_allocate(i, y). Here, credit_allocate(i, y) = credit_allocate(i). α(i-1, y). The connection (y) can be understood as the first active connection mentioned above. The pre-allocated credit of the first active connection in this round is determined by weighting the basic allocated credit based on the credit utilization rate.

[0103] In addition, excess credit (i.e., the credit remaining after subtracting pre-allocated credit from the base credit) can be returned to the credit pool. Specifically, the credit added to the credit pool (i.e., the aforementioned excess credit) is credit_pool += credit_allocate(i). (1-α(i-1, y)).

[0104] Optionally, the receiver iterates through each active connection, and for connections (z) with a credit usage rate α>=γ, it calculates the total credit usage of these connections, credit_sum. Connection (z) can be understood as the aforementioned second active connection, so the total credit usage is credit_sum+=credit_use(i-1,z).

[0105] Additionally, the receiving end iterates through each active connection. For the connection (z) with a credit usage rate α>=γ, it allocates additional credit pool credit proportionally based on the credit used in the previous round (i-1) and uses it as the second active connection credit_allocate(i,z) in this round (i).

[0106] Among them, credit_allocate(i, z)=credit_allocate(i)+credit_pool (credit_use(i-1, z) / credit_sum).

[0107] In addition, a timeout period T is set for the scheduling timer, and the above process is repeated after the timer is triggered.

[0108] As can be seen, the embodiments of this disclosure provide a credit scheduling mechanism that can solve the problems of poor speed control and bandwidth waste in traditional passive congestion control. Specifically, in the credit adjustment mechanism, the receiving end dynamically adjusts the credit allocation in real time based on the historical credit usage of each sending end, thereby solving the bandwidth waste problem of traditional credit-sharing active transmission algorithms.

[0109] Exemplary embodiments of this disclosure also provide a parallel communication optimization apparatus for large model inference experts. (Reference) Figure 8 As shown, the large model inference expert parallel communication optimization device 800 includes the following program modules: The receiving unit 801 is used to receive all-to-all transmission tasks; The scheduling unit 802 is used to add the traffic to be transmitted to the queue of incomplete transmission traffic according to the full-to-full transmission task, and to perform a traffic scheduling mechanism on the queue of incomplete transmission traffic to determine the amount of traffic data sent by each network card. The transmission unit 803 is used to determine the pre-allocated credits of each active connection by the receiving end, and send the pre-allocated credits of each active connection to the sending end so that each network card transmits traffic data according to the corresponding pre-allocated credits.

[0110] In one possible implementation, the scheduling unit 802 is configured to: Select the first traffic with the smallest remaining data volume in the incomplete transmission traffic queue; When it is determined that there is an available window to send the first traffic, a traffic transmission co-track communication connection is established, and the data volume of the first traffic and the load information of each network card are determined; The data volume of the first traffic is distributed to each network card in proportion to the load information of each network card, so as to determine the traffic data volume sent by each network card.

[0111] In one possible implementation, the scheduling unit 802 is configured to: Based on the shortest task priority scheduling strategy, the first traffic with the smallest remaining data volume is selected from the queue of incomplete transmission traffic.

[0112] In one possible implementation, the transmission unit 803 is used for: The basic credit allocation for each active connection in this round is determined based on the number of active connections currently. Determine the transmission credit utilization rate of each active connection in the previous round among the currently active connections; The pre-allocated credit for each active connection in the current round is determined based on the previous round's transmission credit utilization rate and the basic allocated credit.

[0113] In one possible implementation, the transmission unit 803 is used for: Determine the preset threshold; The previous round transmission credit utilization rate of each active connection is compared with the preset threshold to obtain multiple comparison results; Based on the obtained multiple comparison results and the basic allocation credit, the pre-allocated credit for each activity connection in this round is determined.

[0114] In one possible implementation, the transmission unit 803 is used for: If the comparison result of the first active connection indicates that the credit utilization rate is less than the preset threshold, then the basic allocated credit is weighted based on the credit utilization rate to determine the pre-allocated credit of the first active connection in this round, and the excess credit is recycled to the credit pool.

[0115] In one possible implementation, the transmission unit 803 is used for: If the comparison result of the second active connection indicates that the credit utilization rate is not less than the preset threshold, then the total credit of the second active connection is calculated. The total amount of credit to be allocated is determined based on the total credit in the second activity connection, the basic allocated credit, and the credit recovered from the credit pool; The total amount of credit to be allocated is distributed proportionally to the second active connection to determine the pre-allocated credit for the second active connection in this round.

[0116] The specific details of each part of the above-mentioned device have been described in detail in the method section of the implementation plan. For any undisclosed details, please refer to the implementation plan of the method section, and therefore will not be repeated here.

[0117] In this embodiment of the disclosure, Figure 8 The scheduling unit 802 shown is Figure 1 The scheduling layer shown performs the same operation. Figure 8 The transmission unit 803 shown is Figure 1 The transport layer shown performs the same operation.

[0118] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0119] Exemplary embodiments of this disclosure also provide a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the above-described parallel communication optimization method for large model inference experts.

[0120] In one implementation, the computer program product can be a tangible product containing a computer program, such as a computer-readable storage medium storing the computer program. The readable storage medium can be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. For example, the computer program product can be implemented as a non-volatile storage medium storing a computer program, such as read-only memory, NAND flash memory, etc.

[0121] In one implementation, the computer program product can be an intangible product containing a computer program. For example, the computer program product can be implemented as a virtual digital product, such as an executable file, installation package, or other digital file storing the computer program.

[0122] Computer program code can be written in one or more programming languages. Examples of programming languages ​​include C, Java, and C++. Program code can execute entirely on the user's computing device, partially on the user's computing device, or as a standalone software package. It can also execute partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via an internet connection provided by a mobile network operator).

[0123] Computer programs can be carried or transmitted via signals such as electricity, magnetism, light, electromagnetic fields, and infrared radiation. Electronic devices can convert signals carrying computer programs into digital signals, thereby running the computer programs. When a computer program runs on an electronic device, its code is used to cause the electronic device to execute (more specifically, the processor of the electronic device to execute) the method steps of various exemplary embodiments of this disclosure, such as the above-described large model inference expert parallel communication optimization method, which includes the following steps: Step 301: Receive a full-to-full transmission task; Step 302: Add the traffic to be transmitted to the incomplete transmission traffic queue according to the full-to-full transmission task, and perform a traffic scheduling mechanism on the incomplete transmission traffic queue to determine the amount of traffic data to be sent by each network card; Step 303: Determine the pre-allocated credits of each active connection by the receiving end, and send the pre-allocated credits of each active connection to the sending end, so that each network card transmits the traffic data according to the corresponding pre-allocated credits; wherein, the pre-allocated credits are determined by the receiving end based on the historical credit usage information of each sending end by dynamically adjusting the credit allocation.

[0124] By implementing the above method steps through a computer program, a full-to-full transmission task can be received. Then, based on the full-to-full transmission task, the traffic to be transmitted is added to the incomplete transmission traffic queue. A traffic scheduling mechanism is then executed on the incomplete transmission traffic queue to determine the amount of traffic data to be sent by each network interface card (NIC). The traffic scheduling mechanism can be understood as a mechanism for unified scheduling of full-to-full communication traffic and NIC resources, including mechanisms such as dynamic traffic priority scheduling, pre-allocation of internal NIC resources, and load-balanced data transmission. Therefore, this disclosure achieves efficient utilization of internal NIC resources in full-to-full communication through a traffic scheduling mechanism.

[0125] Furthermore, the receiver can determine the pre-allocated credits for each active connection and send these pre-allocated credits to the sender, enabling each network interface card (NIC) to transmit traffic data according to its corresponding pre-allocated credits. The pre-allocated credits are determined by the receiver dynamically adjusting credit allocation based on the historical credit usage information of each sender. In other words, a credit scheduling mechanism is introduced, achieving zero-congestion transmission at the receiver, overcoming the passive speed reduction problem caused by traditional congestion control, thereby significantly improving data transmission performance and optimizing the overall performance of hybrid expert model inference services.

[0126] Exemplary embodiments of this disclosure also provide an electronic device, which may include a processor and a memory. The memory stores executable instructions for the processor, such as computer programs. The processor executes the executable instructions to perform the method steps of various exemplary embodiments of this disclosure.

[0127] The following is for reference. Figure 9 The electronic device is illustrated by way of a general-purpose computing device. It should be understood that... Figure 9 The electronic device 900 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0128] like Figure 9 As shown, the electronic device 900 may include: a processor 910, a memory 920, a bus 930, an I / O (input / output) interface 940, and a network adapter 950.

[0129] The memory 920 may include volatile memory, such as RAM 921 and cache unit 922, and may also include non-volatile memory, such as ROM 923. The memory 920 may also include one or more program modules 924, including but not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. For example, program module 924 may include the modules described above.

[0130] The processor 910 may include one or more processing units, such as an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor, and / or an NPU (Neural-Network Processing Unit).

[0131] The processor 910 can be used to execute executable instructions stored in the memory 920, such as the above-mentioned large model inference expert parallel communication optimization method, which includes the following steps: Step 301: Receive the full-to-full transmission task; Step 302: Add the traffic to be transmitted to the incomplete transmission traffic queue according to the full-to-full transmission task, and perform a traffic scheduling mechanism on the incomplete transmission traffic queue to determine the amount of traffic data sent by each network card; Step 303: Determine the pre-allocated credit of each active connection by the receiving end, and send the pre-allocated credit of each active connection to the sending end so that each network card transmits the traffic data according to the corresponding pre-allocated credit; wherein, the pre-allocated credit is determined by the receiving end dynamically adjusting the credit allocation based on the historical credit usage information of each sending end.

[0132] By executing the above method steps through processor 910, a full-to-full transmission task can be received. Then, based on the full-to-full transmission task, the traffic to be transmitted is added to the incomplete transmission traffic queue. A traffic scheduling mechanism is then executed on the incomplete transmission traffic queue to determine the amount of traffic data to be sent by each network interface card (NIC). The traffic scheduling mechanism can be understood as a mechanism for unified scheduling of full-to-full communication traffic and NIC resources, including mechanisms such as dynamic traffic priority scheduling, pre-allocation of internal NIC resources, and load-balanced data transmission. Therefore, this disclosure achieves efficient utilization of internal NIC resources for full-to-full communication through the traffic scheduling mechanism.

[0133] Furthermore, the receiver can determine the pre-allocated credits for each active connection and send these pre-allocated credits to the sender, enabling each network interface card (NIC) to transmit traffic data according to its corresponding pre-allocated credits. The pre-allocated credits are determined by the receiver dynamically adjusting credit allocation based on the historical credit usage information of each sender. In other words, a credit scheduling mechanism is introduced, achieving zero-congestion transmission at the receiver, overcoming the passive speed reduction problem caused by traditional congestion control, thereby significantly improving data transmission performance and optimizing the overall integrity of the hybrid expert model's inference services.

[0134] Bus 930 is used to connect different components of electronic device 900 and may include data bus, address bus and control bus.

[0135] Electronic device 900 can communicate with one or more external devices 1000 (such as keyboard, mouse, external controller, etc.) through I / O interface 940.

[0136] Electronic device 900 can communicate with one or more networks via network adapter 950. For example, network adapter 9500 can provide mobile communication solutions such as 3G / 4G / 5G / 6G, or wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication. Network adapter 950 can communicate with other modules of electronic device 900 via bus 930.

[0137] although Figure 9 As not shown in the diagram, other hardware and / or software modules may also be configured in the electronic device 900, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0138] As can be seen from the above, the technical solutions disclosed herein can be implemented as methods, apparatus, systems, computer program products, storage media, electronic devices, etc. Those skilled in the art will understand that various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which may be referred to as "circuit," "module," or "system," respectively.

[0139] It should be understood that this disclosure is not limited to the specific methods, steps, or structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. Those skilled in the art will readily conceive of other embodiments based on the specific implementations provided in this disclosure. Therefore, the specific implementations provided in this disclosure are merely exemplary, and the scope and spirit of this disclosure are indicated by the claims, and should cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary technical means in the art not disclosed in this disclosure.

Claims

1. A large model inference expert parallel communication optimization method, characterized in that, The method comprises: receiving a full-to-full transmission task; adding to-be-transmitted traffic to an unfinished transmission traffic queue according to the full-to-full transmission task, and performing a traffic scheduling mechanism on the unfinished transmission traffic queue to determine the amount of traffic data transmitted to each network card; determining pre-allocated credits of the receiving end for each active connection, and sending the pre-allocated credits of the active connection to the sending end, so that the amount of traffic data is transmitted to each network card according to the corresponding pre-allocated credit; wherein the pre-allocated credit is determined by dynamically adjusting the credit allocation of the receiving end based on the historical credit usage information of each sending end.

2. The method of claim 1, wherein, adding to-be-transmitted traffic to an unfinished transmission traffic queue according to the full-to-full transmission task, and performing a preset scheduling mechanism on the unfinished transmission traffic queue to determine the amount of traffic data transmitted to each network card, comprising: selecting a first traffic with the smallest remaining data amount in the unfinished transmission traffic queue; when it is determined that there is an available window to send the first traffic, establishing a traffic transmission same-rail communication connection, and determining the data amount of the first traffic and the load information of each network card; distributing the data amount of the first traffic to each network card in a proportion according to the load information of each network card to determine the amount of traffic data transmitted to each network card.

3. The method of claim 2, wherein, selecting a first traffic with the smallest remaining data amount in the unfinished transmission traffic queue, comprising: based on the shortest task first scheduling strategy, selecting a first traffic with the smallest remaining data amount in the unfinished transmission traffic queue from the unfinished transmission traffic queue.

4. The method according to any of claims 1 to 3, characterized in that, determining pre-allocated credits of the receiving end for each active connection, comprising: determining a basic allocation credit of each active connection in the current round according to the number of connections of the current active connection; determining a last round transmission credit usage rate of each active connection in the current active connection; determining the pre-allocated credit of each active connection in the current round according to the last round transmission credit usage rate of each active connection and the basic allocation credit.

5. The method of claim 4, wherein, determining the pre-allocated credit of each active connection in the current round according to the last round transmission credit usage rate of each active connection and the basic allocation credit, comprising: determining a preset threshold; comparing the last round transmission credit usage rate of each active connection with the preset threshold to obtain a plurality of comparison results; determining the pre-allocated credit of each active connection in the current round according to the plurality of comparison results and the basic allocation credit.

6. The method of claim 5, wherein, determining the pre-allocated credit of each active connection in the current round according to the plurality of comparison results and the basic allocation credit, comprising: if the comparison result of a first active connection indicates that the credit usage rate is less than the preset threshold, then the basic allocation credit is weighted calculated based on the credit usage rate to determine the pre-allocated credit of the first active connection in the current round, and the excess credit is recycled to a credit pool.

7. The method of claim 6, wherein, determining the pre-allocated credit of each active connection in the current round according to the plurality of comparison results and the basic allocation credit, comprising: if the comparison result of a second active connection indicates that the credit usage rate is not less than the preset threshold, then the total credit of the second active connection is counted; Determine a total amount of credits to be allocated according to the second active connection credit sum, the base allocated credit, and the credit pool recycled credit; Determine a pre-allocation credit of the second active connection in the current round by allocating the total amount of credits to be allocated to the second active connection in equal proportions.

8. A large model inference expert parallel communication optimization apparatus, characterized in that, The apparatus comprises: A receiving unit configured to receive a full-to-full transmission task; A scheduling unit configured to add to-be-transmitted traffic to an unfinished transmission traffic queue according to the full-to-full transmission task, and perform a traffic scheduling mechanism on the unfinished transmission traffic queue to determine an amount of traffic data to be transmitted for each network card; A transmitting unit configured to determine pre-allocation credits of each active connection for a receiving end, and transmit the pre-allocation credits of each active connection to a sending end, so that each network card transmits the amount of traffic data according to the corresponding pre-allocation credit; wherein the pre-allocation credit is determined by the receiving end based on historical credit usage information of each sending end to dynamically adjust credit allocation.

9. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method of any one of claims 1 to 7.

10. An electronic device, comprising: Comprise: A processor; A memory configured to store executable instructions of the processor; Wherein the processor is configured to implement the method of any one of claims 1 to 7 by executing the executable instructions.

Images