Distributed computing communication method and device, distributed computing method and system

Abstract

The embodiments of the present disclosure disclose a distributed computing communication method and device, a distributed computing method and system. The distributed computing communication method includes: receiving an operation instruction; in response to the operation instruction, identifying the type of the processing unit of the target device, and loading the target communication library to the target device according to the type of the processing unit of the target device; based on the type of the processing unit of the target device and the target communication library, selecting the first control instruction of the target device, and realizing local communication between the processing units of the target device by executing the first control instruction; controlling the processing unit of the target device to transmit the execution result to the first network switching device; controlling the communication between the first network switching device and the second network switching device, wherein the second network switching device is a network switching device that connects the target device to the heterogeneous device. This method can solve the problem of communication difficulties between each heterogeneous device, help multiple types of devices to work together, and improve the efficiency of distributed computing.

Description

Technical Field

[0001] The present disclosure relates to the field of distributed computing technology, and in particular to a distributed computing communication method and device, and a distributed computing method and system. Background Art

[0002] With the development of artificial intelligence and the expansion of model scale, distributed computing mode has become the mainstream of model training. This mode requires distributing computing tasks to the GPUs (graphics processing units) of several devices to accelerate the model training process. These GPUs can be located in different slots of the same machine or distributed on different physical machines, communicating and coordinating through the network. This mode can significantly improve computing efficiency, especially when dealing with data-intensive and computing-intensive tasks, such as image recognition and natural language processing.

[0003] However, with the diversification of computing hardware, especially the emergence of different types of GPUs (such as NVIDIA, AMD, etc.), the model training process is no longer limited to the use of a single type of GPU. These GPUs have significant differences in architecture, performance, memory bandwidth, etc., leading to the emergence of a heterogeneous device environment. However, in a heterogeneous device environment, communication and data exchange between different types of GPUs become particularly complex, and the lack of a unified communication strategy has become a bottleneck for distributed computing efficiency. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure provide a distributed computing communication method and apparatus, a distributed computing method and system, which can solve the problem of communication difficulties between each heterogeneous device during the distributed computing process, help multiple types of devices to work together, and improve the efficiency of distributed computing.

[0005] In a first aspect, an embodiment of the present disclosure provides a distributed computing communication method, which adopts the following technical solution:

[0006] Receive operation instructions;

[0007] In response to the operation instruction, identifying a processing unit type of a target device, and loading a target communication library into the target device according to the processing unit type of the target device;

[0008] Selecting a first control instruction of the target device based on the type of the processing unit of the target device and the target communication library, and implementing local communication between the processing units of the target device by executing the first control instruction;

[0009] Controlling the processing unit of the target device to transmit the execution result to the first network switching device;

[0010] Controlling communication between the first network switching device and a second network switching device, wherein the second network switching device is a network switching device that connects the target device to a heterogeneous device.

[0011] In a second aspect, the embodiments of the present disclosure further provide a distributed computing and communication device, which adopts the following technical solution:

[0012] The distributed computing communication device includes an interface unit, a communication library loading unit, a local communication unit, an execution result transmission unit, and a global communication unit;

[0013] The interface unit is used to receive operation instructions;

[0014] The communication library loading unit is used to identify the processing unit type of the target device in response to the operation instruction, and load the target communication library to the target device according to the processing unit type of the target device;

[0015] The local communication unit is used to select a first control instruction of the target device based on the type of the processing unit of the target device and the target communication library, and implement local communication between the processing units of the target device by executing the first control instruction;

[0016] The execution result transmission unit is used to control the processing unit of the target device to transmit the execution result to the first network switching device;

[0017] The global communication unit is used to control the communication between the first network switching device and the second network switching device, wherein the second network switching device is a network switching device that connects the target device with a heterogeneous device.

[0018] In a third aspect, the present disclosure provides a distributed computing method, which adopts the following technical solution:

[0019] Receive operation instructions;

[0020] In response to the operation instruction, identifying a processing unit type of a target device, and loading a target communication library into the target device according to the processing unit type of the target device;

[0021] Selecting a first control instruction of the target device based on a processing unit type of the target device and the target communication library;

[0022] The processing unit of the target device executes the first control instruction, and local communication is achieved between the processing units;

[0023] Sending a second control instruction to control the processing unit of the target device to transmit the execution result to the first network switching device;

[0024] The processing unit of the target device executes the second control instruction and transmits the execution result to the first network switching device;

[0025] Sending a third control instruction to control communication between the first network switching device and a second network switching device, wherein the second network switching device is a network switching device that connects the target device and a heterogeneous device;

[0026] The third control instruction is executed, and the first network switching device and the second network switching device perform global communication.

[0027] In a fourth aspect, the embodiments of the present disclosure further provide a distributed computing system, which adopts the following technical solutions:

[0028] The distributed computing system includes: an interface unit, a communication library loading unit, a local communication unit, an execution result transmission unit, a global communication unit, a target device, a heterogeneous device, a first network switching device and a second network switching device; wherein,

[0029] The interface unit receives an operation instruction;

[0030] The communication library loading unit identifies the processing unit type of the target device in response to the operation instruction, and loads the target communication library to the target device according to the processing unit type of the target device;

[0031] The local communication unit selects a first control instruction of the target device based on the processing unit type of the target device and the target communication library;

[0032] The processing unit of the target device executes the first control instruction, and local communication is achieved between the processing units;

[0033] The execution result transmission unit sends a second control instruction to control the processing unit of the target device to transmit the execution result to the first network switching device;

[0034] The processing unit of the target device executes the second control instruction and transmits the execution result to the first network switching device;

[0035] The global communication unit sends a third control instruction to control the communication between the first network switching device and the second network switching device, wherein the second network switching device is a network switching device that connects the target device and the heterogeneous device;

[0036] The first network switching device and the second network switching device execute the third control instruction, and the first network switching device and the second network switching device perform global communication.

[0037] In a fifth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the above-mentioned distributed computing communication methods, or execute any of the above-mentioned distributed computing methods.

[0038] In a sixth aspect, an embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements any of the above-described distributed computing communication methods, or implements any of the above-described distributed computing methods.

[0039] The distributed computing method provided by the embodiment of the present disclosure can adapt to different types of devices by selecting a specific first control instruction according to the processing unit type of the target device to automatically load the communication library suitable for the target device, so that the target device can obtain high-speed communication capabilities, optimize local communication, reduce unnecessary communication overhead, improve overall communication efficiency, reduce the workload of manual configuration and management, and reduce the probability of error. By controlling the communication between the target device and the first network switching device and the second network switching device connected to the heterogeneous device, it can not only support effective data exchange between heterogeneous devices, expand the scope of application and application scenarios of distributed computing, and enhance compatibility and scalability, which is crucial for building a system in which multiple types of devices work together, but also facilitate the optimization of data transmission paths, reduce network bottlenecks, improve data transmission speed and stability, and ensure the timely delivery of calculation results. Through this centralized control and dynamic adjustment communication method, the stability and security of the communication process can be ensured, the security risks and system failures caused by device differences and improper communication configuration can be reduced, and the overall stability and security of the system can be improved.

[0040] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 A schematic diagram of a distributed computing communication method according to an embodiment of the present disclosure;

[0043] Figure 2A flow chart of a communication control method for a control module provided in an embodiment of the present disclosure;

[0044] Figure 3 A schematic diagram of local communication provided by an embodiment of the present disclosure;

[0045] Figure 4 A schematic diagram of global communication provided by an embodiment of the present disclosure;

[0046] Figure 5 A principle block diagram of a distributed computing communication device provided in an embodiment of the present disclosure;

[0047] Figure 6 A schematic diagram of a distributed computing method according to an embodiment of the present disclosure;

[0048] Figure 7 A flow chart of a communication control method provided by an embodiment of the present disclosure;

[0049] Figure 8 A block diagram of the distributed computing system provided by the embodiment of the present disclosure;

[0050] Fig. 9 A schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0051] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0052] It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in this disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this disclosure.

[0053] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0054] Reference Figure 1 The present disclosure provides a distributed computing communication method, comprising the following steps:

[0055] S1: receiving operation instructions;

[0056] S2: In response to the operation instruction, identifying the processing unit type of the target device, and loading the target communication library to the target device according to the processing unit type of the target device;

[0057] S3: based on the processing unit type of the target device and the target communication library, selecting a first control instruction of the target device, and implementing local communication between the processing units of the target device by executing the first control instruction;

[0058] S4: Control the processing unit of the target device to transmit the execution result to the first network switching device;

[0059] S5: Control the communication between the first network switching device and the second network switching device, wherein the second network switching device is a network switching device that connects the target device and the heterogeneous device.

[0060] The distributed computing communication method disclosed in the present invention can adapt to different types of devices by selecting a specific first control instruction according to the processing unit type of the target device to automatically load a communication library suitable for the target device, so that the target device can obtain high-speed communication capabilities, optimize local communication, reduce unnecessary communication overhead, improve overall communication efficiency, reduce the workload of manual configuration and management, and reduce the probability of error. By controlling the communication between the target device and the first network switching device and the second network switching device connected to the heterogeneous device, it can not only support effective data exchange between heterogeneous devices, expand the scope of application and application scenarios of distributed computing, and enhance compatibility and scalability, which is crucial for building a system in which multiple types of devices work together, but also facilitate the optimization of data transmission paths, reduce network bottlenecks, improve data transmission speed and stability, and ensure the timely delivery of calculation results. Through this centralized control and dynamic adjustment-friendly communication method, the stability and security of the communication process can be ensured, the security risks and system failures caused by device differences and improper communication configuration can be reduced, and the overall stability and security of the system can be improved.

[0061] Optionally, in S1, the operation instruction may be all gather, ReduceScatter, Broadcast, AllReduce operation, etc.

[0062] Optionally, in S1, an operation function for the application layer to call is constructed, the operation function is executed, and an operation instruction is received. The operation function includes a data pointer, a data size, and a data type. For example, the operation function is a predefined AllReduce function, and the AllReduce function is as follows:

[0063] void AllReduce(void* data, size_t size, DataType dtype, CommunicatorType communicator);

[0064] Among them, data represents the data pointer, specifically the data pointer that needs to execute AllReduce, which is used to point to the data for the operation and allow the data to be passed in. Size represents the size of the data, which is used to indicate the byte size of the data to be processed. Dtype represents the type of data (such as float32, float64, etc.), which is used to indicate the type of data to be processed, and the calculation accuracy of the operation is determined according to the type of data. Communicator represents the communication library type, which represents the communicator type of different devices (such as NCCL, RCCL, MCCL), and is used to encapsulate the communication implementation algorithm for the target device. It is worth noting that when receiving operation instructions, the communicator is in a masked state, so there is no need to consider the specific communication library type in the AllReduce function.

[0065] By using operation functions, the development and maintenance of upper-level frameworks or applications are simplified. Developers do not need to write code for different communication libraries (such as NCCL, RCCL, and MCCL), reducing code duplication and maintenance costs. The operation function implements a unified API interface, which has high flexibility and scalability, and can easily support multiple data types and underlying communication libraries, adapt to different application scenarios and hardware environments, so that the upper-level framework or application does not need to consider the differences in devices, and can communicate with devices of different processing unit types, solving the compatibility problem between devices of different processing unit types.

[0066] Based on the operation function, in addition to receiving the operation instruction, a preset topology structure is also received, and nodes of the preset topology structure are parsed to obtain each topology node in the preset topology structure, where each topology node represents a target device.

[0067] Among them, the preset topological structure can be a tree structure (such as Tree AllReduce) or a ring structure (such as Ring AllReduce). The establishment of the topological structure requires the collection of key information between each heterogeneous device, and the extraction of the computation graph (Computation Graph) of the distributed computing task by parsing the operation instructions. By analyzing the computation graph, the data flow direction (such as where the data comes from and where it goes) and the communication mode (such as point-to-point, broadcast, and collective communication) are determined. According to the execution stage of the task (such as training and reasoning), the communication requirements and data sharding strategies of different stages are determined. This information constitutes the task data traffic distribution. Key information and task data traffic distribution facilitate the accurate grasp of the data dependency and communication mode of distributed computing tasks, and provide data support for the construction of the topological structure. Heuristic algorithms (such as genetic algorithms, simulated annealing algorithms) or machine learning models (such as decision trees, neural networks) are used to determine multiple different topological schemes based on key information and the task data traffic distribution. These schemes may involve different node connection methods, path selection strategies, and data transmission protocols.

[0068] This application can use a network simulator (such as NS3, OMNeT++, etc.) to simulate the network environment and communication load in various scenarios, implement each topology scheme in the simulator, collect and analyze key performance indicators such as end-to-end delay, overall communication overhead and load balancing, and determine the topology scheme with the best performance in the current scenario based on the comparison and analysis of performance information as the topology structure actually used.

[0069] Among them, collecting key information between each heterogeneous device further includes: using network scanning tools (such as nmap, arp-scan) to detect the physical connection mode between different heterogeneous devices and generate a network topology map; obtaining the configuration information of switches and routers through network management protocols (such as SNMP) to identify the network path between each heterogeneous device (i.e., between heterogeneous devices), and using network visualization tools (such as Wireshark, Graphviz) to draw a network topology map to display the physical connection relationship between different heterogeneous devices. By analyzing the network topology map, network topology information is obtained, which includes the physical connection mode between each heterogeneous device. Use network monitoring tools (such as iftop, bmon) to obtain the actual bandwidth capacity and current bandwidth usage of each communication link in real time, and use network management protocols (such as SNMP) to regularly query the bandwidth utilization data of network devices and perform real-time monitoring. Use custom monitoring scripts or software (such as Prometheus, Grafana) to collect and store bandwidth utilization data in real time. Use GPU monitoring tools (such as nvidia-smi, rocm-smi) to collect the computing power indicators of the processing unit (GPU) of each heterogeneous device, including the number of cores, video memory size, computing power, etc. Obtain the real-time computing load status of the GPU, such as utilization, temperature, power consumption, etc., through the API provided by the operating system (such as Linux's sysfs, procfs), and use custom monitoring scripts or software (such as Prometheus, Grafana) to collect and store GPU performance data in real time. Use network delay test tools (such as ping, traceroute) to obtain the physical distance and network delay data between different heterogeneous devices. You can use network monitoring tools (such as iperf, fping) to regularly test the network delay and packet loss rate between different heterogeneous devices. You can use custom monitoring scripts or software (such as Prometheus, Grafana) to collect and store network delay data in real time. Network topology information, actual bandwidth capacity, bandwidth usage, computing load status, physical distance, and network delay data constitute key information.

[0070] In S2, when an operation instruction is received, the processing unit type of the target device is automatically identified, the target communication library is selected based on the graphics processing unit type, and the dlopen function is used to load the target communication library for the target device.

[0071] In a specific implementation, a pre-built conditional judgment function is called to identify the processing unit type of the target device. The conditional judgment function includes multiple conditions, each condition includes a preset graphics processing unit type, and the multiple conditions exist in the form of if statements or else if statements, and based on the characteristics of the if statements and the else if statements, the multiple conditions form an execution order, which is referred to as a conditional order.

[0072] After calling the conditional judgment function, the conditional information (referring to the information of the condition being executed) is automatically sent to the target node according to the conditional order. The target node executes the internal judgment function (referring to the hardware or software logic inside the target device) according to the conditional information and returns the execution result. If the return result is false, the next condition is executed in the conditional order and the next conditional information (referring to the information of the next condition) is sent to the target node. If the return result is true, the preset graphics processing unit type in the corresponding condition (referring to the condition being executed) is the processing unit type of the target device.

[0073] In another specific implementation, a detection request is sent to a target device by calling a device detection function, wherein the calling device detection function may be a command for calling a device detector or a custom code function. The target device executes internal detection logic according to the detection request, returns detected processing unit information, and parses the returned processing unit information to obtain the processing unit type of the target device.

[0074] For example, the processing unit type includes but is not limited to NVIDIA Corporation (NVIDIA), Advanced Micro Devices, Inc. (AMD), and Shanghai Muxinjie Semiconductor Co., Ltd. If the processing unit type is NVIDIA, then the NVIDIA Collective Communications Library (NCCL) is loaded to the target device; if the processing unit type is AMD, then the ROCm Collective Communications Library (RCCL) is loaded to the target device; if the processing unit type is Muxinjie, then the Muxinjie Collective Communications Library (MCCL) is loaded to the target device.

[0075] In S3, the target device loads the target communication library to realize its own high-speed communication, selects the first control instruction of the target device according to the processing unit type of the target device, and when executing the distributed computing task of the model, the target device realizes local communication between its internal processing units by executing the first control instruction.

[0076] The first control instruction refers to the specific implementation of the target device. For example, for the AllReduce operation in the distributed computing task, the first control instruction refers to the specific implementation of the AllReduce operation. Through this specific implementation, each processing unit in the target device executes the distributed computing task and shares the execution results internally.

[0077] In S4 and S5, the processing unit of the target device is controlled to transmit the execution result to the first network switching device, and the first network switching device communicates with the second network switching device to complete the global communication between the target device and the heterogeneous device. Specifically, by sending the second control instruction, the processing unit of the target device is controlled to transmit the execution result to the first network switching device, and the processing unit of the target device executes the second control instruction and transmits the execution result to the first network switching device. By sending a third control instruction, the communication between the first network switching device and the second network switching device is controlled, wherein the second network switching device is a network switching device connecting the target device and the heterogeneous device. The first network switching device and the second network switching device execute the third control instruction, and the first network switching device and the second network switching device communicate globally. Exemplarily, the first network switching device and the second network switching device can be switches, which can be switches respectively integrated in the target device and the heterogeneous device, such as PCIE switches, NVLink-based switches, or MetaXLink-based switches, and the CPUs in the target device and the heterogeneous device can serve as a communication bridge between the switches and the processing units.

[0078] Furthermore, the processing unit of the target device is controlled to transmit the execution result to the homogeneous device, completing the local communication between the target device and the homogeneous device. Specifically, by sending the fourth control instruction, the processing unit of the target device is controlled to transmit the execution result to the homogeneous device, and the processing unit of the target device executes the fourth control instruction and transmits the execution result to the homogeneous device.

[0079] Reference Figure 2 The flow chart of the control module performing the communication control method is shown, and the method of controlling local communication and global communication specifically includes the following steps:

[0080] S41: performing structural analysis on a preset topological structure, constructing multiple communication events based on the connection relationship between the topological nodes, and defining basic attributes of the multiple communication events;

[0081] S42: A directed acyclic graph is used to represent the dependency relationship between multiple communication events, each DAG node represents a communication event, and the edge represents the dependency relationship between the communication events;

[0082] S43: based on the directed acyclic graph and basic properties of the multiple communication events, dividing the multiple communication events into multiple time windows;

[0083] S44: In each time window, optimizing the execution order between communication events based on the dependency relationship and basic attributes between the communication events;

[0084] S45: defining a state machine for each communication event, the state machine including several states;

[0085] S46: Generate a communication instruction according to the start and end time of the time window, the execution order of the communication events in the time window, and the current state of the state machine, wherein the communication instruction includes at least one of the second control instruction, the third control instruction, and the fourth control instruction.

[0086] In S41, all topological nodes in the preset topological structure and the connection relationships between them, including direct connections and indirect connections, are identified, and multiple communication events are constructed according to the connection relationships between the topological nodes, each of which represents a data transmission task. For example, if A1 and A2 are directly connected via InfiniBand (IB), a communication event from A1 to A2 can be constructed, and if A1 and B1 are indirectly connected via a CPU and a switch, a communication event from A1 to B1 can be constructed.

[0087] Assign a unique event ID to each communication event for tracking and management. Determine the source node and target node of each communication event, as well as the processing unit type of the source node and target node. For example, for a communication event from A1 to A2, A1 is the source node and A2 is the target node. Define the communication type for each communication event, such as point-to-point, broadcast, aggregation, etc. For example, if the communication is between two devices A1 and A2, it is point-to-point communication. If A1 sends the execution result to multiple devices, it is broadcast communication. If multiple devices send the execution result to A1, it is aggregation communication. Based on the content and data size of the communication, estimate the amount of data for each communication, which is convenient for predicting and planning the required network bandwidth to make resource allocation decisions. When the data volume is greater than the preset data volume threshold, the communication event adopts the data transmission sharding strategy for communication. Assign a priority to each communication event according to business logic or preset rules to determine the urgency of the communication.

[0088] Based on the above, the basic attributes include at least one of event ID, source node, target node, source node processing unit type, target node processing unit type, communication type, data volume, and priority.

[0089] In S42, a DAG node is created for each communication event. The DAG node contains basic attributes of the communication event. The dependencies between the communication events are analyzed. According to the determined dependencies, edges are added to the DAG nodes. Each edge represents a dependency, indicating that a communication event must be executed after another communication event is completed. For example, if communication event B must wait for communication event A to be completed before it can start, then communication event A is a prerequisite for communication event B. Specifically, the DAG nodes are organized into horizontal layers, and each DAG node is located below its dependency, which helps to visualize the directed acyclic graph (DAG) and simplify the DAG node connection strategy. A fixed width and spacing are assigned to each DAG node to facilitate the location of nodes in the directed acyclic graph. According to the node connection strategy (horizontal connection or orthogonal line segment), a connection is drawn between the output and input ports of the DAG node to form an edge to represent the dependency, and an arrow is added to the edge in the directed acyclic graph to indicate the direction of the dependency. At the same time, labels can be added to illustrate the nature of the dependency, and the DAG is topologically sorted to determine the execution order of the communication events to ensure that all predecessor events have been completed before the current event is executed.

[0090] In S43, the basic properties of each DAG node in the directed acyclic graph are analyzed to determine the parameters of the time window, such as window size, overlap, sliding strategy, etc. These parameters will affect how to allocate communication events to different time windows. Estimate the processing time of each communication event based on basic properties (such as data volume), which can be estimated here by historical data or a preset model. Through a greedy algorithm, dynamic programming or other optimization algorithm, the communication event is allocated to a suitable time window according to the processing time of the communication event and the capacity of the time window. When allocating, consider the dependencies defined in the directed acyclic graph to ensure that all preceding events of a communication event have been processed in the time window before it. Use an optimization algorithm (such as linear programming, integer programming) to minimize the total communication delay, balance the load of each time window, and satisfy the dependencies of all communication events. If there is a conflict between time windows (that is, there are too many communication events in the same time window), adjust the allocation of communication events or adjust the parameters of the time window until the conflict is eliminated. Verify whether the allocation of time windows meets the dependencies of all events and the performance requirements of the system. If necessary, make adjustments to optimize the overall scheduling. Implement the scheduling of communication events according to the determined time window allocation to ensure that communication events are executed according to the predetermined time windows.

[0091] In S44, the dependency of each communication event in the time window is analyzed to construct a local dependency graph (LDG), where the LDG nodes represent the communication events in the time window and the edges represent the dependencies between the communication events in the time window. The local dependency graph is topologically sorted to determine the execution order of the communication events to ensure that all the preceding events have been completed before the current event is executed.

[0092] Topological sorting provides a linear order in which all predecessor events of each communication event are ranked before it. In this case, if there are multiple communication events with the same sorting position (i.e., they can be executed in parallel without dependency), the execution order of the communication events is adjusted according to the priority level, and the communication events with higher priority are executed first. When two or more events have the same priority, at least one of the strategies such as random processing, first-come-first-served (FCFS), shortest job first (SJN), and polling is used to determine their execution order. Among them, random processing refers to randomly selecting communication events for execution to avoid bias and provide fairness, first-come-first-served refers to executing them according to the order in which the communication events enter the ready queue, and shortest job first refers to executing communication events with smaller data volume first to reduce the average waiting time. Polling refers to executing each communication event in turn based on a preset polling strategy when it is determined that the communication event needs to be processed periodically according to the nature of the distributed computing task, to ensure that all communication events can be evenly served.

[0093] In S45, all possible states in the life cycle of the communication event are determined, and a state machine is defined for each communication event based on these states, for example, the state machine includes at least one of the states of Initialized, Waiting, InProgress, Completed, and Failed, wherein Initialization means that the communication event is in the initialization state before it is selected and prepared by the scheduler; Waiting means that the communication event is ready, but the communication event cannot be executed immediately because the execution conditions (for example, waiting for the completion of the preceding event or waiting for the availability of resources) have not been met; InProgress means that the communication event is in progress and the execution result is being transmitted; Completed means that the communication event has been successfully completed and the execution result has been successfully transmitted and received by the target node; Failure means that the communication event encounters an error or abnormal situation during the execution process, resulting in the communication event being unable to be completed normally, and it is necessary to adopt strategies such as retry and backup path switching, and record abnormal events, and use a fault diagnosis assistant to analyze abnormal events for later optimization. Determine the events that cause state transitions, which may be externally triggered, such as receiving a data packet, timeout, etc., or internally triggered, such as task completion. Determine the transition rules from one state to another, including defining under what events and conditions the state will change. Define the actions that need to be performed when the state transition occurs, which can be initialization operations when entering a new state, or cleanup operations when exiting the current state. Based on the events that cause the state transition, the transition rules, and the actions that need to be performed when the state transition occurs, use the programming language to implement the configuration of the state machine, initialize the state machine and set the starting state. The state machine for each communication event starts from its starting state, which can be the "initialization" or "waiting" state. Establish a listener to monitor the life cycle events of the state machine, such as state switching, transition triggering, etc., which helps to debug and optimize the behavior of the state machine, and can predict possible communication bottlenecks, adjust the scheduling strategy in advance, monitor communication performance indicators such as network status and load changes in real time, and dynamically adjust communication parameters.

[0094] In S46, based on the start and end time of the time window, the time window (referred to as the current time window) in which the communication task currently needs to be executed is determined, the execution order between the communication events in the current time window is obtained, and based on the execution order, the communication event (referred to as the current communication event) in which the communication task currently needs to be executed is determined, and the basic attributes of the current communication event are obtained. Based on the basic attributes of the current communication event, it is determined whether it is a global communication. If not, the current state of the current communication event state machine is obtained, and when the current state is waiting, a fourth control instruction is generated and sent to the target device; if so, the current state of the current communication event state machine is obtained, and when the current state is waiting, a second control instruction is generated and sent to the target device, and after the first network switching device receives the execution result sent by the processing unit of the target device, a third control instruction is generated and sent to the first network switching device and the second network switching device.

[0095] Among them, "determining whether it is global communication based on the basic attributes of the current communication event" further includes: obtaining the source node, target node, processing unit type of the source node and processing unit type of the target node in the basic attributes of the current communication event; if the processing unit type of the source node and the processing unit type of the target node are the same, then the source node is determined to be the target device, the target node is a homogeneous device, and the communication between the target device and the homogeneous device is local communication; if the processing unit type of the source node and the processing unit type of the target node are different, then the source node is determined to be the target device, the target node is a heterogeneous device, and the communication between the target device and the heterogeneous device is global communication.

[0096] Monitor the operation of the current communication event in real time and obtain the operation information, update the current state of the state machine based on the operation information, and re-execute the communication task when it is detected that the current communication event fails to execute the communication task based on the operation information. If it still fails, it will automatically switch to the backup path and automatically isolate the current abnormal component to ensure that when the CPU, switch, etc. fail inside the device, the device can still complete communication with other devices to ensure the normal progress of the task.

[0097] Optionally, the operation process of the operation instruction is optimized through one or more of mixed precision communication, data compression and decompression. Mixed precision communication refers to determining the precision requirement according to the nature of the distributed computing task. If it is a high-precision requirement, 32-bit floating point numbers (FP32) are selected to complete the data calculation operation to improve the calculation accuracy. If it is a low-precision requirement, 16-bit floating point numbers (FP64) are selected to complete the data calculation operation to reduce the amount of data communicated. Data compression and decompression refers to using data compression technology to reduce the size of the data packet before data transmission, such as the Huffman algorithm, Lempel-Ziv-Welch (LZW) algorithm, etc. At the receiving end, the compressed data is decompressed to restore it to the original data to increase the communication speed.

[0098] According to the above method, the devices participating in the distributed computing task can complete efficient communication and effectively save the completion time of the distributed computing task. This communication method is also applicable to the distributed training of the model, that is, when the distributed computing task is specifically a model training task, the devices participating in the training task are divided into multiple clusters according to the different types of processing units.

[0099] Reference Figure 3 The schematic diagram of local communication is shown. The target device includes multiple processing units (GPUs), a CPU, and a first network switching device. Multiple GPUs communicate with each other. A node includes multiple devices. The device that needs to perform a communication task is the target device. The remaining devices in the node are homogeneous devices of the target device. The target device and the homogeneous devices communicate with each other through IB.

[0100] Reference Figure 4 A schematic diagram of global communication is shown, in which the target device and the heterogeneous device communicate with each other through the first network switching device and the second network switching device.

[0101] If the training method is a data parallel mode, the training data and model structure are obtained, the training data is divided into multiple subsets equal to the number of clusters, each subset is assigned to a different cluster, the model structure is sent to each device, and each device performs a data calculation operation according to the received subset and model structure to generate an execution result. The data calculation operation includes at least one of forward propagation calculation, back-propagation calculation, and model parameter update, and the execution result includes at least one of forward propagation calculation data, back-propagation calculation data, and model update parameters.

[0102] If the training method is a model parallel mode, the training data and model structure are obtained, the model structure is divided into a plurality of model blocks equal to the number of clusters, each model block is assigned to a different cluster, the training data is sent to each device, and each device performs a data calculation operation according to the received model block and training data to generate an execution result. The data calculation operation includes at least one of forward propagation calculation, back-propagation calculation and model parameter update, and the execution result includes at least one of forward propagation calculation data, back-propagation calculation data and model update parameters.

[0103] For the scenario of distributed model training, the distributed computing communication method disclosed in the present invention can solve the differences in target devices of different processing unit types, simplify the integration and communication processes between heterogeneous devices, ensure that devices from different manufacturers can work together efficiently, improve the compatibility and scalability of the system, optimize the communication path, balance the calculation time and communication time of the data, shorten the overall training time, effectively improve the overall training efficiency, speed up the distributed training, and be able to process larger-scale data sets and models to meet the computing resource requirements of modern AI and deep learning.

[0104] Reference Figure 5 , the present disclosure provides a distributed computing communication device, including an interface unit 101, a communication library loading unit 102, a local communication unit 103, an execution result transmission unit 104, and a global communication unit 105;

[0105] The interface unit 101 is used to receive operation instructions;

[0106] The communication library loading unit 102 is used to identify the processing unit type of the target device in response to the operation instruction, and load the target communication library to the target device according to the processing unit type of the target device;

[0107] The local communication unit 103 is used to select a first control instruction of the target device based on the processing unit type of the target device and the target communication library, and implement local communication between the processing units of the target device by executing the first control instruction;

[0108] An execution result transmission unit 104, used to control the processing unit of the target device to transmit the execution result to the first network switching device;

[0109] The global communication unit 105 is used to control the communication between the first network switching device and the second network switching device, wherein the second network switching device is a network switching device that connects the target device and the heterogeneous device.

[0110] Among them, refer to Figure 5 The distributed computing communication device may include one or more communication modules 10, each of which includes a corresponding interface unit 101, a communication library loading unit 102, a local communication unit 103, an execution result transmission unit 104, and a global communication unit 105. When the distributed computing communication device includes one communication module 10, the communication module 10 implements the communication between all devices in the distributed computing. When the distributed computing communication device includes multiple communication modules 10, the communication modules 10 may correspond to all devices in the distributed computing one by one, and each communication module 10 is responsible for implementing the communication within a device and between the device and other devices.

[0111] The various variations and specific examples of the distributed computing communication method provided above are also applicable to the distributed computing communication device provided in the present disclosure. Through the above detailed description of the distributed computing communication method, those skilled in the art can clearly understand the implementation method of the distributed computing communication device. For the sake of brevity of the specification, it will not be described in detail here.

[0112] Reference Figure 6 The present disclosure provides a distributed computing method, comprising the following steps:

[0113] T1: Receive operation instructions;

[0114] T2: In response to the operation instruction, identifying the processing unit type of the target device, and loading the target communication library into the target device according to the processing unit type of the target device;

[0115] T3: Selecting a first control instruction of the target device based on the processing unit type of the target device and the target communication library;

[0116] T4: The processing unit of the target device executes the first control instruction, and local communication is achieved between the processing units;

[0117] T5: Sending a second control instruction to control the processing unit of the target device to transmit the execution result to the first network switching device;

[0118] T6: The processing unit of the target device executes the second control instruction and transmits the execution result to the first network switching device;

[0119] T7: Send a third control instruction to control the communication between the first network switching device and the second network switching device, wherein the second network switching device is a network switching device that connects the target device and the heterogeneous device;

[0120] T8: Execute the third control instruction, and the first network switching device and the second network switching device perform global communication.

[0121] The distributed computing method disclosed in the present invention can adapt to different types of devices by selecting a specific first control instruction according to the processing unit type of the target device to automatically load a communication library suitable for the target device, so that the target device can obtain high-speed communication capabilities, optimize local communication, reduce unnecessary communication overhead, improve overall communication efficiency, reduce the workload of manual configuration and management, and reduce the probability of error. The communication between the target device and the first network switching device and the second network switching device connected to the heterogeneous device is controlled by the second control instruction and the third control instruction, which can not only support effective data exchange between heterogeneous devices, expand the scope of application and application scenarios of distributed computing, and enhance compatibility and scalability, which is crucial for building a system in which multiple types of devices work together, but also facilitate the optimization of data transmission paths, reduce network bottlenecks, improve data transmission speed and stability, and ensure the timely delivery of calculation results. Through this centralized control and dynamic adjustment communication method, the stability and security of the communication process can be ensured, the security risks and system failures caused by device differences and improper communication configuration can be reduced, and the overall stability and security of the system can be improved.

[0122] In T1, an operation function for the application layer to call is constructed, the operation function is executed, and the operation instruction is received. The operation function includes a data pointer, data size, and data type. For example, the operation function is a predefined AllReduce function, and the AllReduce function is as follows:

[0123] void AllReduce(void* data, size_t size, DataType dtype, CommunicatorType communicator);

[0124] Among them, data represents the data pointer, specifically the data pointer that needs to execute AllReduce, which is used to point to the data for the operation and allow the data to be passed in. Size represents the size of the data, which is used to indicate the byte size of the data to be processed. Dtype represents the type of data (such as float32, float64, etc.), which is used to indicate the type of data to be processed, and the calculation accuracy of the operation is determined according to the type of data. Communicator represents the communication library type, which represents the communicator type of different devices (such as NCCL, RCCL, MCCL), and is used to encapsulate the communication implementation algorithm for the target device. It is worth noting that when receiving operation instructions, the communicator is in a masked state, so there is no need to consider the specific communication library type in the AllReduce function.

[0125] By using operation functions, the development and maintenance of upper-level frameworks or applications are simplified. Developers do not need to write code for different communication libraries (such as NCCL, RCCL, and MCCL), reducing code duplication and maintenance costs. The operation function implements a unified API interface, which has high flexibility and scalability, and can easily support multiple data types and underlying communication libraries, adapt to different application scenarios and hardware environments, so that the upper-level framework or application does not need to consider the differences in devices, and can communicate with devices of different processing unit types, solving the compatibility problem between devices of different processing unit types.

[0126] Based on the operation function, in addition to receiving the operation instruction, a preset topology structure is also received, and nodes of the preset topology structure are parsed to obtain each topology node in the preset topology structure, where each topology node represents a target device.

[0127] Among them, the preset topological structure can be a tree structure (such as Tree AllReduce) or a ring structure (such as Ring AllReduce). The establishment of the topological structure requires the collection of key information between each heterogeneous device, and the extraction of the computation graph (Computation Graph) of the distributed computing task by parsing the operation instructions. By analyzing the computation graph, the data flow direction (such as where the data comes from and where it goes) and the communication mode (such as point-to-point, broadcast, and collective communication) are determined. According to the execution stage of the task (such as training and reasoning), the communication requirements and data sharding strategies of different stages are determined. This information constitutes the task data traffic distribution. Key information and task data traffic distribution facilitate the accurate grasp of the data dependency and communication mode of distributed computing tasks, and provide data support for the construction of the topological structure. Heuristic algorithms (such as genetic algorithms, simulated annealing algorithms) or machine learning models (such as decision trees, neural networks) are used to determine multiple different topological schemes based on key information and the task data traffic distribution. These schemes may involve different node connection methods, path selection strategies, and data transmission protocols.

[0128] Network simulators (such as NS3, OMNeT++, etc.) can be used to simulate network environments and communication loads in various scenarios, implement each topology scheme in the simulator, collect and analyze key performance indicators such as end-to-end delay, overall communication overhead, and load balancing, and determine the topology scheme with the best performance in the current scenario as the actual topology structure based on comparison and analysis of performance information.

[0129] Among them, collecting key information between each heterogeneous device further includes: using network scanning tools (such as nmap, arp-scan) to detect the physical connection mode between different heterogeneous devices and generate a network topology map; obtaining the configuration information of switches and routers through network management protocols (such as SNMP) to identify the network path between each heterogeneous device (i.e., between heterogeneous devices), and using network visualization tools (such as Wireshark, Graphviz) to draw a network topology map to display the physical connection relationship between different heterogeneous devices. By analyzing the network topology map, network topology information is obtained, which includes the physical connection mode between each heterogeneous device. Use network monitoring tools (such as iftop, bmon) to obtain the actual bandwidth capacity and current bandwidth usage of each communication link in real time, and use network management protocols (such as SNMP) to regularly query the bandwidth utilization data of network devices and perform real-time monitoring. Use custom monitoring scripts or software (such as Prometheus, Grafana) to collect and store bandwidth utilization data in real time. Use GPU monitoring tools (such as nvidia-smi, rocm-smi) to collect the computing power indicators of the processing unit (GPU) of each heterogeneous device, including the number of cores, video memory size, computing power, etc. Obtain the real-time computing load status of the GPU, such as utilization, temperature, power consumption, etc., through the API provided by the operating system (such as Linux's sysfs, procfs), and use custom monitoring scripts or software (such as Prometheus, Grafana) to collect and store GPU performance data in real time. Use network delay test tools (such as ping, traceroute) to obtain the physical distance and network delay data between different heterogeneous devices. You can use network monitoring tools (such as iperf, fping) to regularly test the network delay and packet loss rate between different heterogeneous devices. You can use custom monitoring scripts or software (such as Prometheus, Grafana) to collect and store network delay data in real time. Network topology information, actual bandwidth capacity, bandwidth usage, computing load status, physical distance, and network delay data constitute key information.

[0130] In T2, when an operation instruction is received, the processing unit type of the target device is automatically identified, the target communication library is selected based on the graphics processing unit type, and the dlopen function is used to load the target communication library for the target device.

[0131] In a specific implementation, a pre-built conditional judgment function is called to identify the processing unit type of the target device. The conditional judgment function includes multiple conditions, each condition includes a preset graphics processing unit type, and the multiple conditions exist in the form of if statements or else if statements, and based on the characteristics of the if statements and the else if statements, the multiple conditions form an execution order, which is referred to as a conditional order.

[0132] After calling the conditional judgment function, the conditional information (referring to the information of the condition being executed) is automatically sent to the target node according to the conditional order. The target node executes the internal judgment function (referring to the hardware or software logic inside the target device) according to the conditional information and returns the execution result. If the return result is false, the next condition is executed in the conditional order and the next conditional information (referring to the information of the next condition) is sent to the target node. If the return result is true, the preset graphics processing unit type in the corresponding condition (referring to the condition being executed) is the processing unit type of the target device.

[0133] In another specific implementation, a detection request is sent to a target device by calling a device detection function, wherein the calling device detection function may be a command for calling a device detector or a custom code function. The target device executes internal detection logic according to the detection request, returns detected processing unit information, and parses the returned processing unit information to obtain the processing unit type of the target device.

[0134] For example, the processing unit type includes but is not limited to NVIDIA Corporation (NVIDIA), Advanced Micro Devices, Inc. (AMD), and Shanghai Muxinjie Semiconductor Co., Ltd. If the processing unit type is NVIDIA, then the NVIDIA Collective Communications Library (NCCL) is loaded to the target device; if the processing unit type is AMD, then the ROCm Collective Communications Library (RCCL) is loaded to the target device; if the processing unit type is Muxinjie, then the Muxinjie Collective Communications Library (MCCL) is loaded to the target device.

[0135] In T3 and T4, the target device loads the target communication library to achieve its own high-speed communication, and selects the first control instruction of the target device according to the processing unit type of the target device. When executing the distributed computing task of the model, the target device achieves local communication between its internal processing units by executing the first control instruction.

[0136] The first control instruction refers to the specific implementation of the target device. For example, for the AllReduce operation in the distributed computing task, the first control instruction refers to the specific implementation of the AllReduce operation. Through this specific implementation, each processing unit in the target device executes the distributed computing task and shares the execution results internally.

[0137] In T5-T8, the processing unit controlling the target device transmits the execution result to the first network switching device, and the first network switching device communicates with the second network switching device to complete the global communication between the target device and the heterogeneous device. Specifically, by sending a second control instruction, the processing unit controlling the target device transmits the execution result to the first network switching device, and the processing unit of the target device executes the second control instruction and transmits the execution result to the first network switching device. By sending a third control instruction, the communication between the first network switching device and the second network switching device is controlled, wherein the second network switching device is a network switching device connecting the target device and the heterogeneous device. Exemplarily, the first network switching device and the second network switching device can be switches, which can be switches respectively integrated in the target device and the heterogeneous device, such as a PCIE switch, an NVLink-based switch, or a MetaXLink-based switch, and the CPU in the target device and the heterogeneous device can serve as a communication bridge between the switch and the processing unit.

[0138] Furthermore, the processing unit of the target device is controlled to transmit the execution result to the homogeneous device, completing the local communication between the target device and the homogeneous device. Specifically, by sending the fourth control instruction, the processing unit of the target device is controlled to transmit the execution result to the homogeneous device, and the processing unit of the target device executes the fourth control instruction and transmits the execution result to the homogeneous device.

[0139] Reference Figure 7 The flow chart of the communication control method shown in the figure, the method for controlling local communication and global communication specifically includes the following steps:

[0140] T41: Perform structural analysis on the preset topological structure, build multiple communication events based on the connection relationship between topological nodes, and define the basic attributes of multiple communication events;

[0141] T42: A directed acyclic graph is used to represent the dependency relationship between multiple communication events. Each DAG node represents a communication event, and the edge represents the dependency relationship between communication events.

[0142] T43: Based on the directed acyclic graph and the basic properties of multiple communication events, multiple communication events are divided into multiple time windows;

[0143] T44: In each time window, optimize the execution order between communication events based on the dependencies and basic properties between communication events;

[0144] T45: define a state machine for each communication event, the state machine includes several states;

[0145] T46: Generate a communication instruction according to the start and end time of the time window, the execution order of the communication events within the time window, and the current state of the state machine, wherein the communication instruction includes at least one of the fourth control instruction, the second control instruction, and the third control instruction.

[0146] In T41, all topological nodes in the preset topological structure and the connection relationships between them, including direct connections and indirect connections, are identified. According to the connection relationships between the topological nodes, multiple communication events are constructed, and each communication event represents a data transmission task. For example, if A1 and A2 are directly connected through InfiniBand (IB), a communication event from A1 to A2 can be constructed. If A1 and B1 are indirectly connected through the CPU and the switch, a communication event from A1 to B1 can be constructed.

[0147] Assign a unique event ID to each communication event for tracking and management. Determine the source node and target node of each communication event, as well as the processing unit type of the source node and target node. For example, for a communication event from A1 to A2, A1 is the source node and A2 is the target node. Define the communication type for each communication event, such as point-to-point, broadcast, aggregation, etc. For example, if the communication is between two devices A1 and A2, it is point-to-point communication. If A1 sends the execution result to multiple devices, it is broadcast communication. If multiple devices send the execution result to A1, it is aggregation communication. Based on the content and data size of the communication, estimate the amount of data for each communication, which is convenient for predicting and planning the required network bandwidth to make resource allocation decisions. When the data volume is greater than the preset data volume threshold, the communication event adopts the data transmission sharding strategy for communication. Assign a priority to each communication event according to business logic or preset rules to determine the urgency of the communication.

[0148] Based on the above, the basic attributes include at least one of event ID, source node, target node, source node processing unit type, target node processing unit type, communication type, data volume, and priority.

[0149] In T42, a DAG node is created for each communication event. The DAG node contains the basic properties of the communication event. The dependencies between the communication events are analyzed. According to the determined dependencies, edges are added to the DAG nodes. Each edge represents a dependency, indicating that one communication event must be executed after another communication event is completed. For example, if communication event B must wait for communication event A to be completed before it can start, then communication event A is a prerequisite for communication event B. Specifically, DAG nodes are organized into horizontal layers, and each DAG node is located below its dependency, which helps to visualize the directed acyclic graph (DAG) and simplify the DAG node connection strategy. A fixed width and spacing are assigned to each DAG node to facilitate the location of nodes in the directed acyclic graph. According to the node connection strategy (horizontal connection or orthogonal line segment), lines are drawn between the output and input ports of the DAG nodes to form edges to represent the dependency relationship. Arrows are added to the edges in the directed acyclic graph to indicate the direction of the dependency relationship. At the same time, labels can be added to illustrate the nature of the dependency relationship. The DAG is topologically sorted to determine the execution order of the communication events and ensure that all predecessor events have been completed before the current event is executed.

[0150] In T43, the basic properties of each DAG node in the directed acyclic graph are analyzed to determine the parameters of the time window, such as window size, overlap, sliding strategy, etc. These parameters will affect how to allocate communication events to different time windows. The processing time of each communication event is estimated based on the basic properties (such as data volume), which can be estimated here by historical data or a preset model. The communication events are allocated to the appropriate time windows based on the processing time of the communication events and the capacity of the time window through greedy algorithms, dynamic programming or other optimization algorithms. When allocating, the dependencies defined in the directed acyclic graph are considered to ensure that all the predecessor events of a communication event have been processed in the time window before it. Use optimization algorithms (such as linear programming and integer programming) to minimize the total communication delay, balance the load of each time window, and satisfy the dependencies of all communication events. If there is a conflict between time windows (that is, there are too many communication events in the same time window), adjust the allocation of communication events or adjust the parameters of the time window until the conflict is eliminated. Verify whether the allocation of time windows meets the dependencies of all events and the performance requirements of the system. If necessary, make adjustments to optimize the overall scheduling. Implement the scheduling of communication events according to the determined time window allocation to ensure that communication events are executed according to the predetermined time windows.

[0151] In T44, the dependencies of each communication event in the time window are analyzed to construct a local dependency graph (LDG), where LDG nodes represent communication events in the time window and edges represent dependencies between communication events in the time window. The local dependency graph is topologically sorted to determine the execution order of communication events and ensure that all predecessor events have been completed before the current event is executed.

[0152] Topological sorting provides a linear order in which all predecessor events of each communication event are ranked before it. In this case, if there are multiple communication events with the same sorting position (i.e., they can be executed in parallel without dependency), the execution order of the communication events is adjusted according to the priority level, and the communication events with higher priority are executed first. When two or more events have the same priority, at least one of the strategies such as random processing, first-come-first-served (FCFS), shortest job first (SJN), and polling is used to determine their execution order. Among them, random processing refers to randomly selecting communication events for execution to avoid bias and provide fairness, first-come-first-served refers to executing them according to the order in which the communication events enter the ready queue, and shortest job first refers to executing communication events with smaller data volume first to reduce the average waiting time. Polling refers to executing each communication event in turn based on a preset polling strategy when it is determined that the communication event needs to be processed periodically according to the nature of the distributed computing task, to ensure that all communication events can be evenly served.

[0153] In T45, all possible states in the life cycle of the communication event are determined, and a state machine is defined for each communication event based on these states. For example, the state machine includes at least one of the states of Initialized, Waiting, InProgress, Completed, and Failed, where Initialization means that the communication event is in the initialization state before it is selected and prepared by the scheduler; Waiting means that the communication event is ready, but the communication event cannot be executed immediately because the execution conditions (for example, waiting for the completion of the preceding event or waiting for the availability of resources) have not been met; InProgress means that the communication event is in progress and the execution result is being transmitted; Completed means that the communication event has been successfully completed and the execution result has been successfully transmitted and received by the target node; Failure means that the communication event encounters an error or abnormal situation during the execution process, resulting in the communication event being unable to complete normally, and it is necessary to adopt strategies such as retry and backup path switching, and record abnormal events, and use a fault diagnosis assistant to analyze abnormal events for later optimization. Determine the events that cause state transitions, which may be externally triggered, such as receiving a data packet, timeout, etc., or internally triggered, such as task completion. Determine the transition rules from one state to another, including defining under what events and conditions the state will change. Define the actions that need to be performed when the state transition occurs, which can be initialization operations when entering a new state, or cleanup operations when exiting the current state. Based on the events that cause the state transition, the transition rules, and the actions that need to be performed when the state transition occurs, use the programming language to implement the configuration of the state machine, initialize the state machine and set the starting state. The state machine for each communication event starts from its starting state, which can be the "initialization" or "waiting" state. Establish a listener to monitor the life cycle events of the state machine, such as state switching, transition triggering, etc., which helps to debug and optimize the behavior of the state machine, and can predict possible communication bottlenecks, adjust the scheduling strategy in advance, monitor communication performance indicators such as network status and load changes in real time, and dynamically adjust communication parameters.

[0154] In T46, based on the start and end time of the time window, the time window (referred to as the current time window) in which the communication task currently needs to be executed is determined, the execution order between the communication events in the current time window is obtained, and based on the execution order, the communication event (referred to as the current communication event) in which the communication task currently needs to be executed is determined, and the basic attributes of the current communication event are obtained. Based on the basic attributes of the current communication event, it is determined whether it is a global communication. If not, the current state of the current communication event state machine is obtained, and when the current state is waiting, a fourth control instruction is generated and sent to the target device; if so, the current state of the current communication event state machine is obtained, and when the current state is waiting, a second control instruction is generated and sent to the target device, and after the first network switching device receives the execution result sent by the processing unit of the target device, a third control instruction is generated and sent to the first network switching device and the second network switching device.

[0155] Among them, "determining whether it is global communication based on the basic attributes of the current communication event" further includes: obtaining the source node, target node, processing unit type of the source node and processing unit type of the target node in the basic attributes of the current communication event; if the processing unit type of the source node and the processing unit type of the target node are the same, then the source node is determined to be the target device, the target node is a homogeneous device, and the communication between the target device and the homogeneous device is local communication; if the processing unit type of the source node and the processing unit type of the target node are different, then the source node is determined to be the target device, the target node is a heterogeneous device, and the communication between the target device and the heterogeneous device is global communication.

[0156] Monitor the operation of the current communication event in real time and obtain the operation information, update the current state of the state machine based on the operation information, and re-execute the communication task when it is detected that the current communication event fails to execute the communication task based on the operation information. If it still fails, it will automatically switch to the backup path and automatically isolate the current abnormal component to ensure that when the CPU, switch, etc. fail inside the device, the device can still complete communication with other devices to ensure the normal progress of the task.

[0157] Optionally, the operation process of the operation instruction is optimized through one or more of mixed precision communication, data compression and decompression. Mixed precision communication refers to determining the precision requirement according to the nature of the distributed computing task. If it is a high-precision requirement, 32-bit floating point numbers (FP32) are selected to complete the data calculation operation to improve the calculation accuracy. If it is a low-precision requirement, 16-bit floating point numbers (FP64) are selected to complete the data calculation operation to reduce the amount of data communicated. Data compression and decompression refers to using data compression technology to reduce the size of the data packet before data transmission, such as the Huffman algorithm, Lempel-Ziv-Welch (LZW) algorithm, etc. At the receiving end, the compressed data is decompressed to restore it to the original data to increase the communication speed.

[0158] According to the above method, the devices participating in the distributed computing task can complete efficient communication and effectively save the completion time of the distributed computing task. This communication method is also applicable to the distributed training of the model, that is, when the distributed computing task is specifically a model training task, the devices participating in the training task are divided into multiple clusters according to the different types of processing units.

[0159] Reference Figure 3 The schematic diagram of local communication is shown. The target device includes multiple processing units (GPUs), a CPU, and a first network switching device. Multiple GPUs communicate with each other. A node includes multiple devices. The device that needs to perform a communication task is the target device. The remaining devices in the node are homogeneous devices of the target device. The target device and the homogeneous devices communicate with each other through IB.

[0160] Reference Figure 4 A schematic diagram of global communication is shown, in which the target device and the heterogeneous device communicate with each other through the first network switching device and the second network switching device.

[0161] If the training method is a data parallel mode, the training data and model structure are obtained, the training data is divided into multiple subsets equal to the number of clusters, each subset is assigned to a different cluster, the model structure is sent to each device, and each device performs a data calculation operation according to the received subset and model structure to generate an execution result. The data calculation operation includes at least one of forward propagation calculation, back-propagation calculation, and model parameter update, and the execution result includes at least one of forward propagation calculation data, back-propagation calculation data, and model update parameters.

[0162] If the training method is a model parallel mode, the training data and model structure are obtained, the model structure is divided into a plurality of model blocks equal to the number of clusters, each model block is assigned to a different cluster, the training data is sent to each device, and each device performs a data calculation operation according to the received model block and training data to generate an execution result. The data calculation operation includes at least one of forward propagation calculation, back-propagation calculation and model parameter update, and the execution result includes at least one of forward propagation calculation data, back-propagation calculation data and model update parameters.

[0163] For the scenario of distributed model training, the distributed computing communication method disclosed in the present invention can solve the differences in target devices of different processing unit types, simplify the integration and communication processes between heterogeneous devices, ensure that devices from different manufacturers can work together efficiently, improve the compatibility and scalability of the system, optimize the communication path, balance the calculation time and communication time of the data, shorten the overall training time, effectively improve the overall training efficiency, speed up the distributed training, and be able to process larger-scale data sets and models to meet the computing resource requirements of modern AI and deep learning.

[0164] Reference Figure 8 The present disclosure provides a distributed computing system, including: a communication module 10, a target device, a heterogeneous device, a first network switching device and a second network switching device, the communication module includes: an interface unit, a communication library loading unit, a local communication unit, an execution result transmission unit and a global communication unit; wherein,

[0165] The interface unit receives an operation instruction;

[0166] The communication library loading unit identifies the type of the processing unit of the target device 20 in response to the operation instruction, and loads the target communication library to the target device 20 according to the type of the processing unit of the target device 20;

[0167] The local communication unit selects a first control instruction of the target device 20 based on the processing unit type of the target device 20 and the target communication library;

[0168] The processing unit of the target device 20 executes the first control instruction, and local communication is achieved between the processing units;

[0169] The execution result transmission unit sends a second control instruction to control the processing unit of the target device 20 to transmit the execution result to the first network switching device 40;

[0170] The processing unit of the target device 20 executes the second control instruction and transmits the execution result to the first network switching device 40;

[0171] The global communication unit sends a third control instruction to control the communication between the first network switching device 40 and the second network switching device 50, wherein the second network switching device 50 is a network switching device connecting the target device 20 and the heterogeneous device 60;

[0172] The first network switching device 40 and the second network switching device 50 execute the third control instruction, and the first network switching device 40 and the second network switching device 50 perform global communication.

[0173] Among them, the global communication unit can also be used to send a fourth control instruction to control the processing unit of the target device 20 to transmit the execution result to the homogeneous device 30; the processing unit of the target device 20 executes the fourth control instruction and transmits the execution result to the homogeneous device 30.

[0174] The various variations and specific examples of the distributed computing method provided above are also applicable to the distributed computing system provided in the present disclosure. Through the above detailed description of the distributed computing method, those skilled in the art can clearly understand the implementation method of the distributed computing system. For the sake of brevity of the specification, it will not be described in detail here.

[0175] The computer device according to the embodiment of the present disclosure includes a memory and a processor. The memory is used to store non-temporary computer-readable instructions. Specifically, the memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc.

[0176] The processor may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the computer device to perform desired functions. In one embodiment of the present disclosure, the processor is used to run the computer-readable instructions stored in the memory, so that the computer device performs all or part of the steps of the distributed computing communication method or distributed computing method of each embodiment of the present disclosure.

[0177] Those skilled in the art should be able to understand that in order to solve the technical problem of how to obtain a good user experience, the present embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the protection scope of the present disclosure.

[0178] like Fig. 9 A schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure is shown, which is a schematic diagram of the structure of a computer device suitable for implementing the embodiment of the present disclosure. Fig. 9 The computer device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0179] like Fig. 9As shown, the computer device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the computer device are also stored. The processor, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0180] Typically, the following devices can be connected to the I / O interface: input devices such as sensors or visual information acquisition devices; output devices such as display screens; storage devices such as tapes, hard disks, etc.; and communication devices. The communication device can allow the computer device to communicate with other devices (such as edge computing devices) wirelessly or by wire to exchange data. Fig. 9 A computer device having various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0181] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processor, all or part of the steps of the distributed computing communication method or distributed computing method of the embodiment of the present disclosure are executed.

[0182] According to the computer-readable storage medium of the embodiment of the present disclosure, non-transitory computer-readable instructions are stored thereon. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the distributed computing communication method or distributed computing method of each embodiment of the present disclosure are executed.

[0183] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A distributed computing communication method, characterized in that: include: Receive operation instructions; In response to the operation instruction, identifying a processing unit type of a target device, and loading a target communication library into the target device according to the processing unit type of the target device; Selecting a first control instruction of the target device based on the type of the processing unit of the target device and the target communication library, and implementing local communication between the processing units of the target device by executing the first control instruction; Controlling the processing unit of the target device to transmit the execution result to the first network switching device; Controlling communication between the first network switching device and a second network switching device, wherein the second network switching device is a network switching device that connects the target device with a heterogeneous device; The receiving operation instruction includes: Constructing an operation function for the application layer to call, wherein the operation function includes a data pointer, a data size and a data type; Execute the operation function and receive the operation instruction; The step of identifying the processing unit type of the target device includes: Constructing a conditional judgment function, wherein the conditional judgment function includes a plurality of conditions, and each condition includes a preset graphics processing unit type; Calling the conditional judgment function to judge, if the return result of a condition is true, then the preset graphics processing unit type in the condition is the processing unit type of the target device; or, Calling a device detection function to detect the processing unit type of the target device; The topological structure preset in the distributed computing includes: the target device, a heterogeneous device heterogeneous with the target device, and a homogeneous device homogeneous with the target device; the distributed computing communication method further includes: The processing unit of the target device is controlled to transmit the execution result to the homogeneous device.

2. The distributed computing communication method according to claim 1, characterized in that: Also includes: Performing structural analysis on a preset topological structure, constructing multiple communication events based on the connection relationship between topological nodes, and defining basic attributes of multiple communication events, wherein the topological nodes are nodes of the preset topological structure; A directed acyclic graph is used to represent the dependency relationship between multiple communication events, each DAG node represents a communication event, and an edge represents the dependency relationship between the communication events, wherein the DAG node is a node of the directed acyclic graph; Based on the directed acyclic graph and basic properties of the multiple communication events, dividing the multiple communication events into multiple time windows; In each time window, the execution order between communication events is optimized based on the dependencies and basic properties between communication events; defining a state machine for each communication event, the state machine comprising a plurality of states; Generate communication instructions according to the start and end time of the time window, the execution order between communication events in the time window, and the current state of the state machine; Based on the communication instruction, the processing unit of the target device is controlled to transmit the execution result to the isomorphic device, the processing unit of the target device is controlled to transmit the execution result to the first network switching device, and the communication between the first network switching device and the second network switching device is controlled.

3. The distributed computing communication method according to claim 2, characterized in that: The generating of the communication instruction according to the start and end time of the time window, the execution order of the communication events in the time window, and the current state of the state machine includes: Based on the start and end time of the time window, determine the time window in which the communication task needs to be executed; Obtain the execution order between communication events in the current time window, and determine the communication event that currently needs to execute the communication task based on the execution order; Based on the basic attributes of the current communication event, determine whether it is global communication; If not, when the current state of the current communication event state machine is waiting, generating a fourth control instruction and sending it to the target device; If so, when the current state of the current communication event state machine is waiting, generating a second control instruction and sending it to the target device; After the first network switching device receives the execution result sent by the processing unit of the target device, a third control instruction is generated and sent to the first network switching device and the second network switching device.

4. The distributed computing communication method according to claim 3, characterized in that: Also includes: The current communication event is monitored in real time, and when it is monitored that the communication task fails to be executed, the current state of the state machine is abnormal; Re-execute the communication task. If it still fails, automatically switch to the backup path and isolate the current abnormal component.

5. A distributed computing communication device, applied to the distributed computing communication method according to any one of claims 1 to 4, characterized in that: It includes an interface unit, a communication library loading unit, a local communication unit, an execution result transmission unit, and a global communication unit; The interface unit is used to receive operation instructions; The communication library loading unit is used to identify the processing unit type of the target device in response to the operation instruction, and load the target communication library to the target device according to the processing unit type of the target device; The local communication unit is used to select a first control instruction of the target device based on the type of the processing unit of the target device and the target communication library, and implement local communication between the processing units of the target device by executing the first control instruction; The execution result transmission unit is used to control the processing unit of the target device to transmit the execution result to the first network switching device; The global communication unit is used to control the communication between the first network switching device and the second network switching device, wherein the second network switching device is a network switching device that connects the target device with a heterogeneous device.

6. A distributed computing method, characterized in that: include: Receive operation instructions; In response to the operation instruction, identifying a processing unit type of a target device, and loading a target communication library into the target device according to the processing unit type of the target device; Selecting a first control instruction of the target device based on a processing unit type of the target device and the target communication library; The processing unit of the target device executes the first control instruction, and local communication is achieved between the processing units; Sending a second control instruction to control the processing unit of the target device to transmit the execution result to the first network switching device; The processing unit of the target device executes the second control instruction and transmits the execution result to the first network switching device; Sending a third control instruction to control communication between the first network switching device and a second network switching device, wherein the second network switching device is a network switching device that connects the target device and a heterogeneous device; Executing the third control instruction, the first network switching device and the second network switching device perform global communication; The topological structure preset in the distributed computing includes: the target device, a heterogeneous device heterogeneous with the target device, and a homogeneous device homogeneous with the target device; the distributed computing method further includes: Sending a fourth control instruction to control the processing unit of the target device to transmit the execution result to the homogeneous device; The processing unit of the target device executes the fourth control instruction and transmits the execution result to the homogeneous device.

7. The distributed computing method according to claim 6, characterized in that: Also includes: Collect key information between each heterogeneous device; Based on the operation instruction, the task data traffic distribution is obtained, wherein the task data traffic distribution includes data flow direction, communication mode, communication requirements at different stages and data sharding strategy; Determine a plurality of different topology solutions based on the key information and the task data flow distribution; The simulator is used to obtain performance information of each topology solution in various scenarios, and the topology structure in the corresponding scenario is determined based on the performance information.

8. The distributed computing method according to claim 7, characterized in that: The key information collected between each heterogeneous device includes: Collecting network topology information, wherein the network topology information includes a physical connection mode between each heterogeneous device; Obtain the actual bandwidth capacity of each communication link in real time, and monitor the bandwidth usage of the current communication link in real time; Collect the computing power indicators of the processing units of each heterogeneous device and monitor the computing load status of each heterogeneous device in real time; Get the physical distance and network latency data between each heterogeneous device.

9. A distributed computing system, applied to the distributed computing method according to any one of claims 6 to 8, characterized in that: include: An interface unit, a communication library loading unit, a local communication unit, an execution result transmission unit, a global communication unit, a target device, a heterogeneous device, a first network switching device, and a second network switching device; wherein, The interface unit receives an operation instruction; The communication library loading unit identifies the processing unit type of the target device in response to the operation instruction, and loads the target communication library to the target device according to the processing unit type of the target device; The local communication unit selects a first control instruction of the target device based on the processing unit type of the target device and the target communication library; The processing unit of the target device executes the first control instruction, and local communication is achieved between the processing units; The execution result transmission unit sends a second control instruction to control the processing unit of the target device to transmit the execution result to the first network switching device; The processing unit of the target device executes the second control instruction and transmits the execution result to the first network switching device; The global communication unit sends a third control instruction to control the communication between the first network switching device and the second network switching device, wherein the second network switching device is a network switching device that connects the target device and the heterogeneous device; The first network switching device and the second network switching device execute the third control instruction, and the first network switching device and the second network switching device perform global communication.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the distributed computing communication method described in any one of claims 1-4, or to execute the distributed computing method described in any one of claims 6-8.

11. A computer program product comprising computer instructions, characterized in that: When the computer instruction is executed by a processor, the steps of the distributed computing communication method described in any one of claims 1-4 are implemented, or when executed, the steps of the distributed computing method described in any one of claims 6-8 are implemented.

Images