Data processing method and related device

Abstract

A data processing method and related device, the method comprising: obtaining a computing instruction, the computing instruction being used to indicate execution of a computing task, the computing task comprising a first collective communication operation. Selecting a first algorithm from a configuration pool according to a first performance requirement of the first collective communication operation. The configuration pool is used to indicate algorithms associated with performance requirements of collective communication operations, and the first algorithm is an algorithm associated with the first performance requirement in the configuration pool. Executing the first collective communication operation by using the first algorithm to obtain an operation result. Thus, the algorithm used by the collective communication operation is determined efficiently, and the data processing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to a data processing method, apparatus, computing cluster, chip, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of computer technology, more and more fields are using high-performance computing (HPC) systems to solve the computing needs of computationally intensive and massive data processing tasks. For example, in weather forecasting and image processing scenarios, high-performance computing systems can shorten computing time and improve computing accuracy.

[0003] High-performance computing systems often employ message passing interfaces (MPI) for information communication, supporting one-to-many or many-to-many collective communication operations between processes. In practice, a collective communication operation can be implemented using various algorithms. Taking the MPI_Allreduce operation, which implements reduction computation, as an example, it can be implemented using one of the following six algorithms: Basic_linear, Non-overlapping, Recursive_doubling, Ring, Segmented_ring, and Rabenseifner.

[0004] Therefore, during the operation of high-performance computing systems, technicians often need to determine the appropriate set communication algorithm based on the specific application scenario. This approach is not only inefficient but may also result in algorithms that fail to meet the performance requirements of the application. Therefore, determining the appropriate set communication algorithm more efficiently has become a pressing technical problem. Summary of the Invention

[0005] This application provides a data processing method and related equipment that can efficiently determine the algorithm used in set communication operations in high-performance computing.

[0006] Firstly, a data processing method is provided, comprising: obtaining a computation instruction for instructing the execution of a computation task, the computation task including a first set communication operation; selecting a first algorithm from a configuration pool according to a first performance requirement of the first set communication operation; the configuration pool indicating algorithms associated with the performance requirement of the set communication operation, wherein the first algorithm is an algorithm associated with the first performance requirement in the configuration pool; and executing the first set communication operation using the first algorithm to obtain an operation result. As described above, by constructing a configuration pool to indicate the relationship between performance requirements and algorithms, when selecting an algorithm for a set communication operation, the algorithm associated with the first performance requirement can be found by querying the configuration pool, and then the first set communication operation can be executed using that algorithm. This allows for efficient determination of the algorithm used in the set communication operation. Furthermore, since the configuration pool contains various relationships between different performance requirements and set communication algorithms in the technical solution provided above in this embodiment, when applied to different HPC systems, as long as the parameters of the environment attributes of the HPC system match the performance requirements in the configuration pool, the set communication operation algorithm associated with the current performance requirement scenario can be found using the configuration pool.

[0007] In one possible design, the configuration pool includes a mapping between the performance requirements of a set communication operation and its associated algorithms, where each set communication operation's performance requirement is associated with at least one algorithm. With this design, when executing a set communication operation (or first set communication operation) under a certain performance requirement (or first performance requirement), the configuration pool can be queried to determine the first performance requirement. Based on the mapping between the first performance requirement and its associated algorithms, at least one algorithm corresponding to the first set communication operation can be determined; specifically, this can be at least one algorithm with superior performance under the first performance requirement. Then, one algorithm can be selected from these at least one algorithms for execution, thus achieving the effect of efficiently determining the algorithm used in the set communication operation.

[0008] In one possible design, the aforementioned performance requirements may include at least one of the following: the number of parallel processes in the collective communication operation, the length of messages transmitted in the collective communication operation, the network topology, the network hardware type, the transmission type between network computing nodes (e.g., connection-oriented or datagram-oriented), the point-to-point communication protocol used by the network (e.g., the applicable rules for point-to-point communication transmission protocols), and the network's RDMA support capability (e.g., whether the current network supports RDMA or not). This design better reflects the performance requirements of the collective communication operation, thereby determining an algorithm for the collective communication operation that is more suited to the current performance requirements.

[0009] In one possible design, the aforementioned performance requirements may include a hash digest of the runtime parameters. Furthermore, the selection of a first algorithm from the configuration pool based on the first performance requirements of the first set of communication operations includes: selecting the first algorithm from the configuration pool based on the hash digest of the runtime parameters included in the first performance requirements. In this design, by incorporating the hash digest of the runtime parameters as part (or all) of the performance requirements, the amount of data required to record the performance requirements can be reduced, the size of the configuration pool can be reduced, and the query efficiency of the configuration pool can be improved.

[0010] In one possible design, before obtaining the computation instructions, the method further includes: obtaining a second algorithm from a first algorithm set. The first algorithm set includes multiple algorithms corresponding to the first set communication operation. The first set communication operation is executed using the second algorithm. Performance data of the first set communication operation performed using the second algorithm is recorded in a performance data set. The performance data set records the algorithms that have performed the first set communication operation under the first performance requirement and their corresponding performance data. After the performance data set includes the performance data corresponding to the multiple algorithms in the first algorithm set, the algorithm with the best performance is selected from the multiple algorithms based on the performance data corresponding to each algorithm, and recorded as the algorithm associated with the first performance requirement in a configuration pool. In the above design, the first set communication operation is first executed using multiple algorithms in the first computation set under the first performance requirement to obtain the performance data corresponding to each algorithm. Then, the algorithm with the best performance is selected based on the performance data corresponding to each algorithm, and this algorithm is recorded as the algorithm associated with the aforementioned performance requirement in the configuration pool. Therefore, when the aforementioned set communication operation needs to be executed again under the aforementioned performance requirement, the algorithm can be directly selected from the configuration pool for execution.

[0011] In one possible design, obtaining the second algorithm from the first algorithm set includes: if the performance data set is empty, then according to preset rules, obtaining the third algorithm from the first algorithm set that best matches the first performance requirement, as indicated by the preset rules, as the second algorithm. In this design, considering the case where the performance data set does not record any algorithm that has performed the first set communication operation under the first performance requirement (which can also be understood as the system not having previously performed the first set communication operation under the first performance requirement), the algorithm that best matches the first set communication operation can be selected from N algorithms according to some preset rules for execution. This ensures that the algorithm executed initially has better performance, facilitating the analysis and comparison of the corresponding performance data when selecting algorithms to perform set communication operations again.

[0012] In one possible design, obtaining the second algorithm from the first algorithm set includes: selecting an algorithm from the first algorithm set that is not included in the performance data set, as the second algorithm. In the above design, by selecting an algorithm from the performance data set as the second algorithm, the algorithms that have not yet been executed are iterated over, so that all algorithms in the first algorithm set are eventually executed.

[0013] In one possible design, obtaining the second algorithm from the first algorithm set includes: selecting the second algorithm from the first algorithm set based on the current system performance. This design reduces the impact on system performance caused by iterating through all algorithms in the first algorithm set.

[0014] In one possible design, obtaining the second algorithm from the first algorithm set includes: selecting the second algorithm from the first algorithm set based on the most recent p algorithms executed under the first performance requirement, as found in the performance data set. In this design, considering that different algorithms have different performance characteristics when performing set communication operations, the second algorithm for this set communication operation is determined based on the most recent P executed algorithms. This avoids the problem of severe system performance degradation caused by repeatedly using inferior algorithms to perform set communication operations.

[0015] In one possible design, the second algorithm is obtained from the first algorithm set based on the most recent p executions of the first set communication operation under the first performance requirement, according to the performance data set. Specifically, if all the algorithms in the most recent p executions of the first set communication operation under the first performance requirement are the third algorithm, then an algorithm other than those in the performance data set is selected as the second algorithm. Otherwise, the third algorithm is selected as the second algorithm. Here, the third algorithm is the algorithm that best matches the first performance requirement, as indicated by a preset rule. This design achieves the effect of using the third algorithm to execute the first set communication operation under the first performance requirement every p times, and then selecting an algorithm not found in the performance data set to execute the first set communication operation. This achieves the goal of traversing the algorithms in the first algorithm set while maintaining relatively stable system performance.

[0016] In one possible design, the second algorithm is obtained from the first algorithm set based on the algorithm that has performed the first set communication operation most recently (p times) under the first performance requirement, from the performance data set. Specifically, if the algorithm that has performed the first set communication operation most recently (p times) under the first performance requirement does not include the third algorithm, then the third algorithm is used as the second algorithm. Otherwise, an algorithm other than the algorithm in the performance data set is selected from the first algorithm set as the second algorithm. The third algorithm is the algorithm that best matches the first performance requirement, as indicated by a preset rule. Through this design, after selecting an algorithm other than the algorithm in the performance data set from the first algorithm set every p times to perform the first set communication operation under the first performance requirement, the third algorithm is used to perform the first set communication operation once under the first performance requirement. This achieves the goal of traversing the algorithms in the first algorithm set while maintaining relatively stable system performance.

[0017] Secondly, this application provides a data processing apparatus, including an acquisition unit and a processing unit. The acquisition unit and the processing unit are respectively configured to perform the operational steps of the first aspect or any design within the first aspect, so that the data processing apparatus performs the method as described in the first aspect or any design within the first aspect.

[0018] Thirdly, this application provides a data processing apparatus, including a processor and an interface. The processor receives or transmits data through the interface, and the processor is used to implement the method as described in the first aspect or any of the designs in the first aspect.

[0019] Fourthly, this application provides a computing cluster comprising: multiple servers. At least one of the multiple servers is used to execute the method as described in the first aspect or any of the designs in the first aspect.

[0020] Fifthly, this application provides a chip including a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to retrieve and execute the computer instructions from the memory to implement the method as described in the first aspect or any of the designs in the first aspect.

[0021] In a sixth aspect, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described in the first aspect or any of the designs in the first aspect.

[0022] In a seventh aspect, this application provides a computer program product including instructions that, when executed on a processor, implement the method as described in the first aspect or any of the designs in the first aspect.

[0023] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0024] Figure 1 A schematic diagram of a network architecture for a computing cluster is provided in this application;

[0025] Figure 2 This application provides a flowchart illustrating an MPI_Allreduce operation performed between four processes using the Ring algorithm.

[0026] Figure 3 A flowchart illustrating an MPI_Allreduce operation performed across 8 processes using the Recursive_doubling algorithm, provided for this application;

[0027] Figure 4 A flowchart illustrating an algorithm for determining set communication operations provided in this application;

[0028] Figure 5 One of the flowcharts illustrating a data processing method provided in this application;

[0029] Figure 6 A second flowchart illustrating a data processing method provided in this application;

[0030] Figure 7 One of the flowcharts provided in this application for obtaining a second algorithm from a first set of algorithms;

[0031] Figure 8 A schematic diagram of a decision tree structure provided in this application;

[0032] Figure 9 A second schematic diagram of a process for obtaining a second algorithm from a first set of algorithms provided in this application;

[0033] Figure 10 The third schematic diagram of a process for obtaining a second algorithm from a first set of algorithms provided in this application;

[0034] Figure 11 This is one of the structural schematic diagrams of a data processing device provided in this application;

[0035] Figure 12 This is a second schematic diagram of the structure of a data processing device provided in this application. Detailed Implementation

[0036] To facilitate understanding of this embodiment, some technical terms involved in this embodiment will be introduced first:

[0037] High-performance computing (HPC) systems are computer cluster systems that connect multiple computer systems through various interconnection technologies. They leverage the combined computing power of all connected systems to handle large-scale computational problems; therefore, HPC systems are also known as high-performance computing clusters. HPC provides solutions with ultra-high floating-point computing capabilities, which can be used to address the computational needs of computationally intensive, massive data processing applications such as scientific research, weather forecasting, computational simulation, military research, computer-aided design (CAD) / computer-aided engineering (CAE), biopharmaceuticals, gene sequencing, and image processing. Utilizing high-performance computing systems to handle computational problems can significantly reduce computation time and improve computational accuracy.

[0038] The Message Passing Interface (MPI) is an application programming interface for message passing, including protocols and semantic specifications. It is a messaging standard in the HPC field, supporting point-to-point and broadcast communication, and features high performance, large scale, portability, and scalability.

[0039] MPI includes two communication protocols: Eagar and RNDZ (Rendezvous). The Eagar protocol sends the message to the target in a single transmission and is often used for small to medium-sized packets. The RNDZ protocol, on the other hand, involves the process dividing the message into multiple data blocks, sending description and control information to the target, then using RDMA GET / PUT instructions to send each data block separately to the target, and finally sending a completion signal. This protocol is typically used for large packets.

[0040] The system structure involved in this embodiment is described below:

[0041] like Figure 1 The diagram shown illustrates a network architecture for a computing cluster provided in this embodiment. The computing cluster 100 can be used to implement the technical solution provided in this embodiment. Specifically, the computing cluster 100 can be a high-performance computing (HPC) system.

[0042] Specifically, the computing cluster includes multiple servers (four servers are shown in the figure as an example: server 101, server 102, server 103 and server 104), management device 106 and storage device 107.

[0043] In some application scenarios, each server in the multiple servers can be a single computing device, meaning the server's functionality is implemented by a single computing device. In other application scenarios, each server in the multiple servers can also comprise a device cluster consisting of multiple computing devices, where the server's functionality can be implemented by virtual machines running on the device cluster. Each server in the multiple servers can serve as a computing node (or simply a node) in the computing cluster 100, providing computing resources for the computing cluster 100. The management device 106 is used to manage the computing cluster 100. The storage device 107 is used to store the data generated by the computing cluster 100 during runtime.

[0044] In addition, multiple servers, management devices 106, and storage devices 107 are connected to the computing cluster through switching devices in the access switching layer. The access switching layer may include one or more switching devices (such as…). Figure 1 (Switching equipment 111-114).

[0045] The switching devices in the access switching layer are connected to the switching devices in the aggregation switching layer. The aggregation switching layer may include one or more switching devices (such as...). Figure 1 (Intermediate switching equipment 121 and switching equipment 122). The aggregation switching layer is used to aggregate access layer traffic and connects to the core switching layer.

[0046] The core switching layer includes one or more switching devices (such as...) Figure 1 Switching devices 131 and 132). The core switching layer is used to forward data packets within the computing cluster and provides an external network interface. Examples include... Figure 1 In this configuration, the core switch 130 connects to external devices (such as user equipment) via an egress router. Additionally, the core switching layer may include a firewall 133 for system security and a service gateway 134 for providing external network interfaces.

[0047] It should be noted that the above is only an exemplary description of the computing cluster structure applicable to this embodiment. In actual application, the technical solution provided in this embodiment can also be applied to other computing clusters with simpler or more complex structures. This embodiment does not limit the system structure of the computing cluster.

[0048] Specifically, when a computing cluster 100 is used to process a computing problem, after a technician inputs computing instructions through a user device to instruct the computing cluster 100 to process the computing problem, the management device 106 in the computing cluster 100 can decompose the computing problem into a series of discrete parts that can be executed concurrently, and execute these discrete parts through multiple parallel processes in multiple servers.

[0049] For example, during the operation of computing cluster 100, such as Figure 1 The diagram shows processes P1-P8 running in parallel on servers 101-104, each responsible for a portion of the computational tasks in a computational problem. Specifically, P1 and P2 run on server 101, P3 and P4 on server 102, P5 and P6 on server 103, and P7 and P8 on server 104. This example illustrates two processes running on each server; in practice, more or fewer processes can run on a single server, and this embodiment does not impose any limitations on this.

[0050] When multiple processes are executed in parallel across multiple servers, the processes can send and receive messages using a message passing interface (MPI).

[0051] Furthermore, message sending and receiving between two processes is called point-to-point communication, such as the MPI_Send and MPI_Recv operations defined in the MPI standard; one process sending or receiving messages to multiple processes is called one-to-many communication, such as the MPI_Bcast operation defined in the MPI standard; and multiple processes sending or receiving messages to multiple processes is called many-to-many communication, such as the MPI_Alltoall and MPI_Allreduce operations defined in the MPI standard. Both one-to-many and many-to-many communication can be referred to as collective communication.

[0052] A collection communication operation can typically be implemented using multiple algorithms. In practical applications, depending on the specific application scenario, a more suitable algorithm can be selected to achieve better performance.

[0053] The following section uses MPI_Allreduce as an example to introduce several algorithms for a set communication operation:

[0054] MPI_Allreduce is an operation that reduces data from all processes to a single result and returns that result to all processes. Figure 1Taking processes P1-P8 as an example, each of these eight processes has one set of data (a total of eight sets of data). By executing MPI_Allreduce, the result of the reduction calculation on these eight sets of data can be returned to all eight processes P1-P8. The operation method used in the reduction calculation can be determined according to actual needs. For example, the reduction calculation on the eight sets of data can be performed by summing, multiplying, or other operations to obtain the result data.

[0055] In related technologies, one of the following six algorithms can generally be selected to implement MPI_Allreduce: Basic linear algorithm, Non-overlapping algorithm, Recursive doubling algorithm, Ring algorithm, Segmented ring algorithm, and Rabenseifner algorithm.

[0056] The following section introduces the commonly used Ring algorithm and the Recursive Doubling algorithm:

[0057] For example, in a scenario where the Ring algorithm is used to perform MPI_Allreduce operations across four processes, such as... Figure 2 As shown:

[0058] First, the four processes are sorted, and the data in the four processes is divided into four data blocks, such as... Figure 2 The data on process P1 is divided into a1, b1, c1, and d1; the data on process P2 is divided into a2, b2, c2, and d2; the data on process P3 is divided into a3, b3, c3, and d3; and the data on process P4 is divided into a4, b4, c4, and d4. Then, in each operation, process Pj (j takes values ​​1-4) sends its [(ji-1)%4]th data block to the neighboring process P(j+1) and receives the [(ji-2)%4]th data block from the neighboring process P(j-1), performing a reduce operation on the received data; after... Figure 2 (a) to Figure 2 (b) Figure 2 (b) to Figure 2 (c) Figure 2 (c) to Figure 2 After 3 operations in (d), such as Figure 2 As shown in (d), at this point, the [(j+1)%4]th data block in process Pj has collected the [(j+1)%4]th data blocks from all processes. Afterwards, the algorithm can be completed using the allgather operation, as shown below. Figure 2 As shown in (e).

[0059] For another example, in a scenario where the Recursive Doubling algorithm is used to perform MPI Allreduce operations across 8 processes, such as... Figure 3 As shown:

[0060] First, sort the 8 processes. Then, exchange data between adjacent processes, such as... Figure 3 In the first step, processes P1 and P2 exchange data, processes P3 and P4 exchange data, and so on. Each process then obtains data from two processes; for example, processes P1 and P2 can obtain their own data and the data of the other process, processes P3 and P4 can obtain their own data and the data of the other process, and so on. Next, processes P1 and P3 exchange data, processes P2 and P4 exchange data, and so on. This allows each process to obtain data from four processes; for example, processes P1 through P4 can all obtain data from processes P1 through P4. Then, processes P1 and P5 exchange data, processes P2 and P6 exchange data, and so on. In this way, each process can obtain data from all processes.

[0061] Compared to the Recursive Doubling algorithm, the Ring algorithm is more suitable for situations with limited system bandwidth and large message lengths, while the Recursive Doubling algorithm is more suitable for situations where the number of processes is a power of 2 and the parallel scale is relatively small.

[0062] The above examples, using the Ring and Recursive Doubling algorithms as illustrations, demonstrate the applicable scenarios for different algorithms. In practical applications, many factors influence set communication operations. These include the parallel scale of processes, the length of transmitted messages, network topology, transmission method, storage method, and the type of computing nodes. These runtime parameters all affect the choice of algorithm for set communication operations. Therefore, selecting a more efficient set communication algorithm is a problem that needs to be solved.

[0063] In related technologies, convolutional neural networks (CNNs) can be used to construct a performance model. In specific application scenarios, this performance model can be used to determine the best-performing algorithm for that scenario, enabling the algorithm to perform ensemble communication operations. For example, ... Figure 4As shown, taking Open MPI as an example, in constructing this performance model using a CNN, a large number of set communication operations with different operating parameters (e.g., process parallelism, message length, network topology, transmission method, storage method, and type of computing nodes, etc.) are first constructed as training set samples. Then, the algorithms and performance results of these set communication operations implemented in Open MPI are recorded. Next, using these training set samples and the corresponding algorithms and performance results, a convolutional neural network is trained to obtain a predictor, which is the performance model. Then, in actual use, the current operating parameters are input into the predictor, and the predictor outputs the corresponding algorithm based on the current operating parameters, so that the set communication operation can be executed using that algorithm.

[0064] As can be seen, in order to ensure that the algorithm output by the performance model is the optimal algorithm, the above scheme requires a large amount of training data, a significant amount of time to train the neural network, and substantial resources to analyze the model's fitting and convergence performance, resulting in high model construction costs. Furthermore, due to the significant differences in configurations among different HPC systems, the performance model needs to be retrained for each system to achieve good accuracy. For example, different HPC systems have different bandwidths and topologies, which significantly impact model accuracy. Therefore, when applying the performance model to HPC systems with different bandwidths and topologies, retraining is necessary to maintain model accuracy.

[0065] In this embodiment, it is considered that: for a set communication operation, the algorithm of the set communication operation with the best performance corresponding to different performance requirements can be predetermined under different performance requirements (the performance requirements may include various operating parameters under the scenario, such as various parameters reflecting the system environment attributes, various parameters reflecting the attributes of this set communication operation, etc.), and the correlation between the performance requirements and the algorithm of the set communication operation is recorded in a configuration pool.

[0066] For example, for the MPI_Allreduce operation, under a specific performance requirement scenario, the MPI_Allreduce operation can be pre-executed using six different algorithms under that scenario. Then, based on the execution results of the six algorithms, the algorithm with the best performance is selected, thus determining the optimal MPI_Allreduce algorithm associated with that performance requirement. Similarly, for different performance requirement scenarios, the association between the performance requirement and the MPI_Allreduce algorithm is determined in the same way, and this relationship is recorded in the configuration pool.

[0067] In this way, when performing a collection communication operation in a scenario that requires meeting specific performance requirements, the algorithm associated with that performance requirement can be directly selected from the configuration pool to perform the collection communication operation using that algorithm.

[0068] Compared to related technologies that determine algorithms by constructing performance models, the technical solution provided in this embodiment does not require constructing a performance model, thus saving the overhead associated with such construction. Furthermore, since the configuration pool in the technical solution provided in this embodiment contains various relationships between different performance requirements and collective communication algorithms, when applied to different HPC systems, as long as the environmental parameters of the HPC system match the performance requirements in the configuration pool, the optimal collective communication operation algorithm for the current scenario can be found using the configuration pool. Compared to selecting algorithms by constructing performance models, the technical solution provided in this embodiment can more efficiently and conveniently determine the collective communication operation algorithm associated with performance requirements.

[0069] The following description, in conjunction with the accompanying drawings, further illustrates a data processing method provided in this embodiment. This method can be applied to, for example... Figure 1 In the computing cluster shown. For example... Figure 5 The data processing method shown may include:

[0070] S201, Obtain calculation instructions.

[0071] This computation instruction can be understood as one of the various instructions that trigger the computing cluster to execute computation tasks.

[0072] For example, the calculation instruction can be Figure 1 The user equipment sends instructions to the computing cluster 100. Upon receiving the instructions, the computing cluster 100 executes the corresponding computing tasks.

[0073] Specifically, after receiving the computing instruction, the computing cluster 100 can trigger multiple servers in the computing cluster to execute multiple discrete parts of the computing task through multiple processes executed in parallel.

[0074] During the execution of the aforementioned computational task, multiple processes on multiple servers can send and receive messages between processes through a set communication operation. Specifically, when the set communication operation to be performed includes multiple algorithms, the method provided in this embodiment can select the algorithm in the following manner, S202:

[0075] S202. Based on the first performance requirement of the first set of communication operations, select a first algorithm associated with the first performance requirement from the configuration pool.

[0076] The first set communication operation can be any set communication operation during the execution of a computing task by the computing cluster. For example, the first set communication operation can be any one of the MPI_Bcast operation, MPI_Alltoall operation, or MPI_Allreduce operation.

[0077] In practical applications, the performance requirements of the collective communication operation referred to in this embodiment can be expressed as one or more operating parameters in the scenario corresponding to the current execution of the collective communication operation. These one or more operating parameters can uniquely determine the performance of the current system.

[0078] In one implementation, the performance requirements of the collective communication operation in this embodiment include parameters reflecting the attributes of the current system environment and parameters reflecting the attributes of the collective communication operation. Specifically, the aforementioned first performance requirement may include parameters reflecting the attributes of the current system environment corresponding to the first collective communication operation and parameters reflecting the attributes of the first collective communication operation.

[0079] One aspect of the parameters reflecting the attributes of the collective communication operation can specifically include: the type of the collective communication operation (e.g., whether it is an MPI_Bcast operation, an MPI_Alltoall operation, or an MPI_Allreduce operation), and the number of parallel processes for the collective communication operation (e.g., using...). Figure 1 Taking the scenario shown as an example, if this set communication operation needs to complete communication between 8 processes, then the number of parallel processes for this set communication operation is 8, and the number of computing nodes for this set communication operation (for example, taking...) Figure 1 Taking the scenario shown as an example, the 8 processes run on 4 servers, i.e., 4 compute nodes, respectively; the number of processes within each compute node in this set communication operation (e.g., Figure 1 The parameters to be considered include: the number of processes within a computing node; the length of the message transmitted by the set communication operation; the commutativity of the set communication operation's reduction operation (for example, when performing a summation operation via MPI_Allreduce, A+B=B+A, so the summation operation via MPI_Allreduce is commutative; when performing a difference operation via MPI_Allreduce, the minuend and subtrahend cannot be arbitrarily interchanged, so the difference operation via MPI_Allreduce is not commutative); whether the set communication operation supports blocking operations; and whether the set communication operation is communication within the same communication subnet in the network.

[0080] On the other hand, parameters reflecting the attributes of the current system environment can specifically include: parameters reflecting network topology characteristics, parameters reflecting the characteristics of transmission methods between computing nodes in the network, and parameters reflecting the characteristics of shared memory methods among computing nodes.

[0081] Among them, parameters reflecting network topology characteristics may include: network topology, type of hardware used in the network (such as switches or routers), and whether the network supports shared storage between computing nodes, one or more of these parameters.

[0082] Parameters reflecting the characteristics of the transmission method between computing nodes in the network may include: the transmission method between computing nodes in the network (e.g., whether the transmission between computing nodes in the network is connection-oriented or datagram-oriented (unreliable connection / unreliable datagram, UC / UD)); the point-to-point communication protocol used (e.g., the specific rules applicable to the point-to-point communication transmission protocol in the network, which may include thresholds for the Eagar protocol and the RNDZ protocol, which indicate that the Eagar protocol is used when the length of the message to be transmitted is less than the threshold, and the RNDZ protocol is used when the length of the message to be transmitted is greater than the threshold); and the network's support capability for remote direct memory access (RDMA) (e.g., whether the current network supports RDMA or not), one or more of these parameters.

[0083] Parameters reflecting the characteristics of shared memory methods within compute nodes can include: the shared memory level within compute nodes in the network. Specifically, the shared memory level within compute nodes can be categorized into levels such as disallowing sharing, supporting shared memory, and supporting shared CPU L1 cache (L1 cache), etc.

[0084] In one possible design, in order to better reflect the performance requirements of the collective communication operation, the performance requirements of the collective communication operation in actual application shall include at least the following: the number of parallel processes of the collective communication operation, the length of the message transmitted by the collective communication operation, parameters reflecting the network topology, the type of hardware used by the network, the transmission type between computing nodes of the network (e.g., connection-oriented or datagram-oriented), the point-to-point communication protocol used by the network (e.g., the applicable rules of the network for point-to-point communication transmission protocols), and the network's RDMA support capability (e.g., whether the current network supports RDMA or not).

[0085] In addition, in this embodiment, the configuration pool is used to indicate the algorithms of the collection communication operations associated with different performance requirements.

[0086] Specifically, the configuration pool includes the correspondence between the different performance requirements of various set communication operations and their associated algorithms. Each set communication operation's performance requirement can be associated with at least one algorithm.

[0087] For example, taking the MPI_Allreduce operation as an example, in this embodiment, a list can be built in the configuration pool to store the algorithms of the MPI_Allreduce operation associated with different performance requirements. For example, the list in the configuration pool is shown in Table 1 below:

[0088] Table 1

[0089] Key Value Performance Requirement 1 Recursive Doubling Algorithm Performance Requirement 2 Non-overlapping algorithm Performance requirement 3 Recursive Doubling Algorithm Performance requirement 4 Ring algorithm Performance requirement 5 Segmented_ring algorithm Performance requirement 6 Rabenseifner algorithm Performance requirement 7 Ring algorithm

[0090] Table 1 records the algorithms associated with seven performance requirements (Performance Requirement 1 to Performance Requirement 7). Each performance requirement and its associated algorithm is recorded in Table 1 as key-value pairs. Specifically, the key portion of Table 1 for each performance requirement includes parameters reflecting the current system environment attributes and parameters reflecting the attributes of the set communication operation, as mentioned above. The value portion for each performance requirement records the algorithm, which can be a pre-determined algorithm that performs optimally when performing set communication operations under that performance requirement. It should be noted that the above example only uses a list of key-value pairs to describe the relationship between performance requirements and algorithms indicated by the configuration pool. In actual implementation, other forms can be used to indicate this relationship, and this embodiment does not impose any restrictions on this.

[0091] Additionally, it should be noted that the above example illustrates the configuration pool by associating a performance requirement for a set communication operation with a specific set communication algorithm. In other application scenarios, the performance requirement for a set communication operation in the configuration pool can also be associated with multiple algorithms. In such scenarios, after determining the multiple algorithms associated with the performance requirement of the set communication operation, an algorithm can be further selected from these algorithms to execute the current set communication operation.

[0092] Optionally, the configuration pool may also record a corresponding software version number to indicate the software version of the computing cluster to which the configuration pool is applicable, so as to facilitate the subsequent expansion and upgrading of the configuration pool.

[0093] S203. Execute the first set communication operation using the first algorithm and obtain the operation result.

[0094] Specifically, after determining the first algorithm, the first algorithm can be used to execute the first set communication operation, obtain the operation result, and complete the calculation task indicated by the calculation instruction.

[0095] It should be noted that in practical applications, when a master process (such as the root process) exists among the multiple processes participating in the first set communication operation, the above-mentioned S202 content can be executed by the master process. After determining the first algorithm, the master process sends the first algorithm to the other processes, so that all processes participating in the first set communication operation can use the first algorithm to execute the first set communication operation. Alternatively, when the multiple processes participating in the first set communication operation do not distinguish between master and slave processes, multiple processes can execute the above-mentioned S202 separately, so that each process can use S202 to determine the first algorithm, and then all processes can use the first algorithm to execute the first set communication operation. In this embodiment, there are no restrictions on the process executing the above-mentioned S202 content.

[0096] In the method described in this embodiment, after obtaining the first performance requirement of the first collective communication operation, the algorithm associated with the first performance requirement can be found by querying the configuration pool, and then the first collective communication operation can be executed using the algorithm. Compared with the related technology that uses the construction of a performance model to determine the algorithm of the collective communication operation, the method described in this embodiment can save the various overheads incurred in constructing the performance model. In addition, since the configuration pool contains various associations between different performance requirements and collective communication algorithms in the technical solution provided in the above aspects of this embodiment, when applied to different HPC systems, as long as the parameters of the environmental attributes of the HPC system match the performance requirements in the configuration pool, the collective communication operation algorithm associated with the current performance requirement scenario can be found using the configuration pool. Furthermore, based on the construction of the configuration pool, this embodiment can also adopt the method that the algorithm associated with the performance requirement of the collective communication operation indicated in the configuration pool is the pre-determined algorithm with the best performance associated with the collective communication operation under the performance requirement, thereby achieving the effect of selecting the algorithm with the best performance under the current performance requirement to execute the collective communication operation according to the different current performance requirements.

[0097] In one implementation, to better represent the performance requirements of a collection communication operation, a number of runtime parameters are typically needed. That is, for each performance requirement, the configuration pool needs to store the contents of all runtime parameters included in that performance requirement (specifically, this can include multiple parameters from both the parameters reflecting the current system environment attributes and the parameters reflecting the collection communication operation attributes). This requires a significant amount of storage space to record the multiple runtime parameters corresponding to the performance requirements.

[0098] Therefore, in this embodiment, the performance requirement may include a hash digest of the operating parameters. For example, the parameters reflecting the current system environment attributes and the parameters reflecting the set communication operation attributes corresponding to the performance requirement can be integrated into a document, and then the hash digest of the document can be calculated to obtain the hash digest of the performance requirement containing the operating parameters.

[0099] Furthermore, the configuration pool can specifically indicate the association between the hash digests of the runtime parameters included in the performance requirements and the algorithm. Therefore, S202 above can include:

[0100] A first algorithm is selected from the configuration pool based on the hash digest of the running parameters included in the first performance requirement of the first set of communication operations.

[0101] For example, a performance requirement can be represented as an 8-byte hash digest by hashing the runtime parameters included in the performance requirement.

[0102] Furthermore, considering that the number of algorithms for a set communication operation is typically less than 10, one byte can be used to represent each algorithm. Therefore, in the configuration pool, the association between the hash digest of each performance requirement and the algorithm can be represented by 9 bytes. Thus, 1MB of space can store more than 11,000 associations. This reduces the storage cost of the configuration pool.

[0103] In another implementation, if the configuration pool does not record the algorithm associated with the set communication operation under the current performance requirements, an algorithm can be selected from among several algorithms capable of performing the set communication operation, and its performance data recorded. Subsequently, when the set communication operation needs to be performed again under the same performance requirements, another algorithm can be selected until the performance data for each algorithm under the same performance requirements is obtained. This allows the algorithm with the best performance to be selected based on the performance data of each algorithm, and this algorithm is recorded in the configuration pool as the algorithm associated with the performance requirements. Therefore, the next time the set communication operation needs to be performed under the same performance requirements, the algorithm can be directly selected from the configuration pool.

[0104] Therefore, as Figure 6 As shown, before executing S202, the method provided in this embodiment can first determine whether the configuration pool indicates an algorithm associated with the first performance requirement. If it is determined that the configuration pool does not indicate an algorithm associated with the first performance requirement, the above method may further include:

[0105] S204. Obtain the second algorithm from the first algorithm set.

[0106] The first algorithm set includes N algorithms corresponding to the first set of communication operations.

[0107] For example, when the first set of communication operations is the MPI_Allreduce operation, the first set of algorithms may include the Basic_linear algorithm, the Non-overlapping algorithm, the Recursive_doubling algorithm, the Ring algorithm, the Segmented_ring algorithm, and the Rabenseifner algorithm. The second algorithm can be one of the above six algorithms included in the first set of algorithms.

[0108] In one possible design, considering that in some scenarios there might be algorithms for the first set of communication operations that are clearly unsuitable for the current primary performance requirements, these unsuitable algorithms can be directly filtered out when selecting the second algorithm to avoid affecting system performance. That is, before executing S204, the method may also include:

[0109] From the various algorithms corresponding to the first set of communication operations, algorithms that are not suitable for the first performance requirement are filtered out, resulting in N algorithms in the aforementioned first algorithm set.

[0110] In this embodiment, the term "algorithm not applicable to the first performance requirement" can be understood as an algorithm determined according to a pre-defined rule that should be excluded from use in scenarios with the first performance requirement. This pre-defined rule can be a pre-configured decision tree, etc. It is understood that this embodiment does not impose restrictions on the specific conditions for "not applicable."

[0111] S205. Using the second algorithm, perform the first set communication operation and obtain the operation result.

[0112] Continuing with the example of the first set communication operation being the MPI_Allreduce operation, if the second algorithm is the Recursive_doubling algorithm, then the first set communication operation will be executed using the Recursive_doubling algorithm.

[0113] S206. Record the performance data of the first set of communication operations performed using the second algorithm in the performance data set.

[0114] The performance data set is used to record the algorithms that have performed the first set of communication operations under the first performance requirement, as well as the corresponding performance data of the algorithms.

[0115] The performance data corresponding to the algorithm includes various parameters indicating the performance of the algorithm in performing the first set communication operation under the first performance requirement. For example, the performance data corresponding to the algorithm may specifically include: the time spent performing the first set communication operation using the algorithm under the first performance requirement; or, for example, the performance data corresponding to the algorithm may specifically include: the amount of resources used by the algorithm in performing the first set communication operation under the first performance requirement, and so on.

[0116] Continuing with the example above, after performing the first set communication operation (MPI_Allreduce operation) using the Recursive_doubling algorithm, the performance dataset will record: the performance requirements of this set communication (such as the first performance requirement mentioned above), the algorithm used to perform the set communication operation (such as the second algorithm mentioned above), and the performance data corresponding to the algorithm (such as the time spent on this set communication operation).

[0117] S207. After including the performance data corresponding to the above N algorithms in the performance data set, select the algorithm with the best performance from the N algorithms according to the performance data corresponding to the N algorithms, and record it in the configuration pool as the algorithm associated with the first performance requirement.

[0118] For example, taking the MPI_Allreduce operation as an example, in each scenario where the first performance requirement is encountered, one of the above six algorithms can be selected to execute the MPI_Allreduce operation according to steps S204-S206, and the performance data corresponding to that algorithm can be obtained. This continues until the performance data set records the performance data corresponding to each of the above six algorithms. For example, the contents of the performance data set are shown in Table 2:

[0119] Table 2

[0120] Performance requirements algorithm Performance data (runtime) Performance Requirement 1 Basic_linear algorithm t1 Performance Requirement 1 Non-overlapping algorithm t2 Performance Requirement 1 Recursive Doubling Algorithm t3 Performance Requirement 1 Ring algorithm t4 Performance Requirement 1 Segmented_ring algorithm t5 Performance Requirement 1 Rabenseifner algorithm t6

[0121] This document records the runtime (t1-t6) of the set communication operations corresponding to the six algorithms for the MPI_Allreduce operation under performance requirement 1. The algorithm with the shortest runtime is then selected and recorded in the configuration pool as the algorithm associated with performance requirement 1. Therefore, when the MPI_Allreduce operation needs to be executed again under the first performance requirement (as shown in Table 2, performance requirement 1), the operation can be executed by selecting the algorithm associated with the first performance requirement from the configuration pool, thus achieving the effect of executing the MPI_Allreduce operation according to the algorithm with the shortest runtime and improving system performance.

[0122] Specifically, considering that the performance data corresponding to the aforementioned N algorithms may be affected by the current system load, for example, taking the runtime of the set communication operation as the performance data corresponding to the algorithm, under different current system load scenarios, the runtime of the set communication operation may be long or short when using the same algorithm to perform the same set communication operation with the same performance requirements. Generally, the higher the current system load, the longer the runtime; the lower the current system load, the shorter the runtime. Therefore, in this embodiment, the performance data corresponding to the N algorithms can specifically be the performance data when the system load is below the load threshold.

[0123] For example, if the system load is high (greater than the load threshold) when performing the first set communication operation using algorithm x, the performance data of this first set communication operation using algorithm x will not be recorded in the performance data set.

[0124] Alternatively, in another implementation, before executing S202, the optimal first algorithm corresponding to the first performance requirement can be determined according to the methods of S204-S207, and this first algorithm can be recorded in the configuration pool as the algorithm corresponding to the first performance requirement. This allows S202 to select the first algorithm corresponding to the first performance requirement from the configuration pool based on the first performance requirement. The process of determining the optimal first algorithm corresponding to the first performance requirement according to the methods of S204-S207 before executing S202 can be referred to the relevant descriptions of S204-S207 in this embodiment, and will not be repeated here.

[0125] In another possible design, such as Figure 7 As shown, in the process of obtaining the second algorithm from the first algorithm set via S204, the method may specifically include:

[0126] S2041. Determine whether the performance data set records an algorithm that has performed the first set communication operation under the first performance requirement.

[0127] If the performance data set is empty, then execute S2042; if the performance data set is not empty, then execute S2043a.

[0128] The performance data set is empty, which can be understood as the performance data set not recording any algorithms that have performed the first set communication operation under the first performance requirement.

[0129] S2042. According to the preset rules, obtain the third algorithm that best matches the first performance requirement from the first algorithm set, as indicated by the preset rules, and use it as the second algorithm.

[0130] In this design, considering the scenario where the performance dataset does not record any algorithm that has executed the first set communication operation under the first performance requirement (which can also be understood as the system not having previously executed the first set communication operation under the first performance requirement), we can first select the best-matching algorithm from N algorithms according to some preset rules. This ensures that the algorithm executed for the first time performs better, facilitating the analysis and comparison of the corresponding performance data when selecting algorithms again for set communication operations.

[0131] Specifically, the aforementioned preset rule can be a pre-determined decision tree, which is used to determine the algorithm associated with the performance requirements based on one or more operating parameters in the performance requirements of the set communication operation.

[0132] For example, as Figure 8 Taking the decision tree shown as an example, this decision tree uses five operating parameters: the interactivity of the reduction operation (i.e., whether the graph supports swapping operations; a positive result is represented as "≥0.5"; a negative result as "<0.5"), whether shared memory is supported (a positive result is represented as "≥0.5"; a negative result as "<0.5"), whether the process has CPU affinity (a positive result is represented as "≥0.5"; a negative result as "<0.5"), the number of parallel processes, and the length of the transmitted message. These parameters are used to construct a decision tree that selects an algorithm from eight algorithms (Algorithms 1-8 in the graph). Using this decision tree, the third algorithm that best matches the current primary performance requirement can be selected based on these five operating parameters.

[0133] S2043a. Select an algorithm other than the one in the performance data set from the first algorithm set as the second algorithm.

[0134] In other words, if the performance dataset already records the algorithms that have performed the first set of communication operations under the first performance requirement, then an algorithm other than the one in the performance dataset can be selected as the second algorithm. This allows the algorithms that have not yet been executed to be traversed, so that all the algorithms in the first set of algorithms can be executed once in the end.

[0135] Furthermore, considering that during system operation, when the first set of communication operations needs to be performed under the first performance requirement, selecting an algorithm other than the one in the performance data set for execution each time might impact system performance. For example, it's possible that multiple consecutive selected algorithms might have poor performance data, thus affecting system performance.

[0136] Therefore, in one possible design, when S2041 determines that the performance data set is not empty, such as Figure 9 As shown, the method also includes:

[0137] S2043b: Based on the current system performance, obtain the second algorithm from the first algorithm set.

[0138] For example, when the system performance is poor, the system performance can be improved by selecting an algorithm with better performance data from the first algorithm set; when the system performance is good, the algorithm in the first algorithm set can be continued by selecting an algorithm other than the one in the performance data set.

[0139] Specifically, S2043b can be implemented in the following ways: 1)-2)

[0140] 1) Detect the current system performance. For example, detect the current system load.

[0141] 2) On the one hand, if it is determined that the current system performance is worse than the system performance threshold, the third algorithm that best matches the first performance requirement as indicated by the aforementioned preset rule is obtained from the first algorithm set and used as the second algorithm. The specific process of obtaining the third algorithm can be referred to the content of S2042 above.

[0142] On the other hand, if it is determined that the current system performance is better than the system performance threshold, an algorithm other than the algorithms in the performance data set is selected from the first algorithm set as the second algorithm. The specific process of selecting the second algorithm from the first algorithm set (excluding the algorithms in the performance data set) can be found in section S2043a above.

[0143] By using the method described in S2043b, the impact on system performance caused by traversing each algorithm in the first algorithm set can be reduced.

[0144] Alternatively, in another possible design, to reduce the impact on system performance caused by traversing all algorithms in the first algorithm set, in scenarios where the first set of communication operations needs to be performed under the first performance requirement, the algorithm with better performance data (such as the third algorithm that best matches the first performance requirement as indicated by the aforementioned preset rules) can be periodically executed to maintain system performance. For example, after executing the first set of communication operations using x algorithms not found in the performance data set (i.e., algorithms that have not been executed before), the third algorithm can be used to execute the first set of communication operations once; or, after executing the first set of communication operations using the third algorithm x consecutive times, an algorithm not found in the performance data set can be selected to execute the first set of communication operations. In this way, the system performance can be kept in a relatively stable state.

[0145] Therefore, when S2041 determines that the performance data set is not empty, such as Figure 10 As shown, the method also includes:

[0146] S2043c: Based on the algorithm that has been used in the most recent p times to execute the first set communication operation under the first performance requirement in the performance data set, obtain the second algorithm from the first algorithm set.

[0147] Specifically, in one design, S2043c includes:

[0148] If the algorithm that performed the first set communication operation in the performance dataset in the most recent p instances under the first performance requirement was the third algorithm, then an algorithm other than the one in the performance dataset was selected as the second algorithm from the first algorithm set. Otherwise, the third algorithm was selected as the second algorithm.

[0149] The third algorithm is the one that best matches the first performance requirement, as indicated by the preset rules.

[0150] Regarding the content on preset rules, please refer to the relevant content in S2042 above. Additionally, the process of selecting an algorithm from the performance data set (excluding the algorithms in the first algorithm set) as the second algorithm can be referred to the content in S2043a above.

[0151] Through the above design, we can achieve the effect of using the third algorithm to execute the first set of communication operations under the first performance requirement every p times, and then selecting an algorithm not found in the performance data set to execute the first set of communication operations. In this way, we can achieve the goal of traversing the algorithms in the first algorithm set, and at the same time, we can keep the system performance in a relatively stable state.

[0152] In another design, S2043c specifically includes:

[0153] If the algorithm that performed the first set communication operation under the first performance requirement in the performance dataset most recently p times does not include the third algorithm, then the third algorithm is selected as the second algorithm. Otherwise, an algorithm other than the one in the performance dataset is selected from the first algorithm set as the second algorithm.

[0154] The process of selecting the second algorithm from the performance data set (excluding the first algorithm set) as the second algorithm, as described above, can be found in the relevant content and will not be repeated here.

[0155] With the above design, after selecting an algorithm other than the one in the performance data set from the first algorithm set every p times to perform the first set communication operation under the first performance requirement, the third algorithm is then used to perform the first set communication operation once under the first performance requirement. This achieves the goal of traversing the algorithms in the first algorithm set while maintaining relatively stable system performance.

[0156] It should be noted that the above Figure 7 , Figure 9 and Figure 10 The examples provided illustrate the process of selecting the second algorithm from the first algorithm set by prioritizing the execution of the third algorithm when the performance dataset is empty (i.e., the performance dataset does not contain records of any executed algorithms). In other implementations, S2041 may be omitted, and the second algorithm may be selected directly from the first algorithm set according to S2043a, S2043b, or S2043c. In other words, when the performance dataset is empty, it is not necessary to prioritize the execution of the third algorithm. This embodiment does not impose any restrictions on this.

[0157] In this embodiment, the method described above, on the one hand, employs a configuration pool to indicate the relationship between performance requirements and algorithms. When selecting an algorithm for a collective communication operation, the algorithm associated with the first performance requirement can be found by querying the configuration pool, and then the first collective communication operation can be executed using that algorithm. Compared to related technologies that use performance models to determine the algorithm for collective communication operations, this embodiment saves the overhead of constructing performance models. Furthermore, since the configuration pool contains various relationships between different performance requirements and collective communication algorithms, when applied to different HPC systems, as long as the environmental parameters of the HPC system match the performance requirements in the configuration pool, the collective communication operation algorithm associated with the current performance requirement scenario can be found using the configuration pool. Further, based on the constructed configuration pool, the algorithm associated with the performance requirement of the collective communication operation indicated in the configuration pool in this embodiment can be a pre-determined algorithm with the optimal performance associated with that performance requirement. This achieves the effect of selecting the optimal algorithm for executing the collective communication operation based on the different current performance requirements.

[0158] The above text combines Figures 1 to 10 The present invention describes in detail the data processing method provided in this embodiment of the present invention, which will be discussed below in conjunction with... Figures 11 to 12 This describes the data processing apparatus provided according to this embodiment of the present invention.

[0159] Figure 11This is a schematic diagram of the structure of a data processing apparatus provided herein. The data processing apparatus 30 includes an acquisition unit 301 and a processing unit 402. This data processing apparatus is used to implement the above-described... Figure 5 , Figure 6 , Figure 7 , Figure 9 and Figure 10 The function of some or all of the steps in the method described.

[0160] The acquisition unit 301 is used to acquire a calculation instruction, which is used to instruct the execution of a calculation task; the calculation task includes a first set communication operation.

[0161] Processing unit 302 is configured to select a first algorithm from a configuration pool based on a first performance requirement of the first aggregate communication operation; the configuration pool is used to indicate the algorithm associated with the performance requirement of the aggregate communication operation; the first algorithm is the algorithm associated with the first performance requirement in the configuration pool.

[0162] The processing unit 302 is further configured to use the first algorithm to perform the first set communication operation to obtain the operation result.

[0163] It should be understood that the data processing apparatus 30 of this embodiment of the present invention can be implemented by a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. It can also be implemented in software. Figures 5 to 10 In the data processing method shown, the data processing device and its various modules can also be software modules.

[0164] In one possible design, the configuration pool includes a mapping between the performance requirements of a set of communication operations and the associated algorithms, with each set of communication operation's performance requirement associated with at least one algorithm.

[0165] In one possible design, the performance requirements include at least one of the following: the number of parallel processes in the collective communication operation, the length of the messages transmitted in the collective communication operation, the network topology, the network hardware type, the transmission method between the network's computing nodes, the point-to-point communication protocol used by the network, and the network's support for Remote Direct Memory Access (RDMA).

[0166] Optionally, the performance requirements include a hash digest of the running parameters.

[0167] The processing unit 302 is configured to select a first algorithm from the configuration pool according to the first performance requirement of the first set communication operation, including: the processing unit 302 is configured to select the first algorithm from the configuration pool according to the hash digest of the running parameters included in the first performance requirement.

[0168] In one possible design, the acquisition unit 301 is further configured to acquire a second algorithm from a first algorithm set before acquiring the calculation instruction; wherein the first algorithm set includes multiple algorithms corresponding to the first set communication operation.

[0169] The processing unit 302 is further configured to execute the first set communication operation using the second algorithm.

[0170] The processing unit 302 is further configured to record the performance data of the first set communication operation performed using the second algorithm in a performance data set; wherein, the performance data set is used to record the algorithm that performed the first set communication operation under the first performance requirement and the corresponding performance data of the algorithm.

[0171] The processing unit 302 is further configured to, after including the performance data corresponding to the various algorithms in the performance data set, select the algorithm with the best performance from the various algorithms according to the performance data corresponding to the various algorithms, and record it in the configuration pool as the algorithm associated with the first performance requirement.

[0172] In one possible design, the acquisition unit 301 is further configured to acquire a second algorithm from the first algorithm set before acquiring the calculation instruction, including: if the performance data set is empty, the acquisition unit 301 is configured to acquire, according to a preset rule, a third algorithm that best matches the first performance requirement from the first algorithm set, as the second algorithm.

[0173] In one possible design, the acquisition unit 301 is further configured to acquire a second algorithm from the first algorithm set before acquiring the computation instructions, including: the acquisition unit 301 is configured to select an algorithm other than the algorithm in the performance data set from the first algorithm set as the second algorithm.

[0174] In one possible design, the acquisition unit 301 is further configured to acquire a second algorithm from the first algorithm set before acquiring the calculation instruction, including: the acquisition unit 301 is configured to acquire the second algorithm from the first algorithm set according to the current system performance.

[0175] In one possible design, the acquisition unit 301 is further configured to acquire a second algorithm from the first algorithm set before acquiring the computation instruction, including: the acquisition unit 301 is configured to acquire the second algorithm from the first algorithm set based on the algorithm that has been executed the first set communication operation most recently p times under the first performance requirement in the performance data set.

[0176] In one possible design, the acquisition unit 301 is used to acquire a second algorithm from a first algorithm set based on the algorithms used in the most recent p instances of executing the first set communication operation under the first performance requirement, as shown in the performance data set. Specifically, the acquisition unit 301 is used to select an algorithm other than those in the performance data set as the second algorithm if all the algorithms used in the most recent p instances of executing the first set communication operation under the first performance requirement are third algorithms. Otherwise, the third algorithm is selected as the second algorithm. The third algorithm is the algorithm that best matches the first performance requirement, as indicated by a preset rule. Through this design, the effect of using the third algorithm to execute the first set communication operation under the first performance requirement every p instances, followed by selecting an algorithm not found in the performance data set to execute the first set communication operation, can be achieved.

[0177] In one possible design, the acquisition unit 301 is used to acquire a second algorithm from a first algorithm set based on the algorithms that have performed the first set communication operation under the first performance requirement in the performance data set in the most recent p times. Specifically, the acquisition unit 301 is used to select the third algorithm as the second algorithm if the third algorithm is not included in the algorithms that have performed the first set communication operation under the first performance requirement in the performance data set in the most recent p times. Otherwise, an algorithm other than the algorithm in the performance data set is selected from the first algorithm set as the second algorithm. The third algorithm is the algorithm that best matches the first performance requirement as indicated by a preset rule.

[0178] For a more detailed description of the acquisition unit 301 and the processing unit 302 mentioned above, please refer to [link / reference]. Figure 5 , Figure 6 , Figure 7 , Figure 9 and Figure 10 The relevant descriptions of the methods shown will not be repeated here.

[0179] In this embodiment, the data processing device can also provide Figure 1The illustrated hardware device is used to perform aggregate communication operations in a computing cluster. For example, the data processing device can be a portion of the hardware device of one or more servers in a computing cluster.

[0180] Figure 12 This is a schematic diagram of another data processing device provided in this embodiment. The data processing device 40 can be a chip or a system-on-a-chip. The data processing device 40 may include some or all of the following components: a processor 401, a communication line 408, a memory 403, and at least one communication interface 402.

[0181] The processor 401 is used to execute the data processing method provided in this embodiment.

[0182] Specifically, processor 401 may include a general-purpose central processing unit (CPU), and processor 401 may also include a microprocessor, a field-programmable gate array (FPGA), a digital signal processor (DSP), or an application-specific integrated circuit (ASIC), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0183] In a specific implementation, as one example, processor 401 may include one or more CPUs, for example... Figure 12 CPU0 and CPU1 in the CPU.

[0184] In a specific implementation, as one embodiment, device 40 may include multiple processors, for example... Figure 12 Processors 401 and 407 are mentioned. Each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process, for example, data (computer program instructions).

[0185] Additionally, memory 403 can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). Memory 403 may exist independently and be connected to processor 401 via communication line 408. Memory 403 may also be integrated with processor 401.

[0186] Memory 403 stores computer instructions. For example, ... Figure 12 As shown, the computer instructions stored in memory 403 may include software modules for implementing the functions of the acquisition unit 301 and the processing unit 302 described above. The processor 401 can execute the data processing method provided in this embodiment by executing the computer instructions stored in memory 403.

[0187] Optionally, the computer execution instructions in this embodiment may also be referred to as application code, and this embodiment does not specifically limit this.

[0188] In addition, the communication interface 402 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0189] In addition, communication line 408 is used to connect the various components in data processing device 40. Specifically, communication line 408 may include data bus, power bus, control bus, and status signal bus, etc. However, for clarity, all buses are labeled as communication line 408 in the figure.

[0190] In a specific implementation, as one embodiment, the data processing device 40 may further include an output device 405 and an input device 406. The output device 405 can communicate with the processor 401 and can display information in various ways. For example, the output device 405 may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device 406 can communicate with the processor 401 and can receive user input in various ways. For example, the input device 406 may be a mouse, keyboard, touchscreen device, or sensing device, etc.

[0191] Additionally, the data processing apparatus 40 may also include a storage medium 404. The storage medium 404 stores computer instructions and various data for implementing the technical solutions of this embodiment. This allows the data processing apparatus 40 to load the computer instructions and various data stored in the storage medium 404 into the memory 403 when executing the data processing method described above in this embodiment, so that the processor 401 can execute the data processing method provided in this embodiment by executing the computer instructions stored in the memory 403.

[0192] It should be understood that the data processing apparatus 40 according to this embodiment may correspond to the data processing apparatus 30 in this embodiment, and may correspond to the corresponding subject executing the data processing method according to this embodiment, and the above and other operations and / or functions of each module in the data processing apparatus 40 are respectively for implementing Figures 5-7 as well as Figure 9 , Figure 10 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0193] As one possible embodiment, this application also provides a computing cluster composed of a plurality of the above-described data processing devices 40. Specifically, the plurality of the above-described data processing devices 40 can be... Figure 1 This refers to multiple servers or portions of hardware within those servers. Furthermore, the structure of the computing cluster provided in this embodiment can be as follows: Figure 1 As shown.

[0194] like Figure 1As shown, in practical applications, to enable communication between multiple data processing devices 40 (i.e., multiple servers in the diagram) within a computing cluster, the computing cluster can also include one or more switching devices to build a data exchange network, such as... Figure 1 The computing cluster includes switching devices in the access switching layer, aggregation switching layer, and core switching layer. In other applications, a simpler switching network can be used in the computing cluster. For example, multiple data processing devices 40 can communicate directly without switching devices, or a more complex switching network can be used in the computing cluster to achieve communication between multiple data processing devices 40. This embodiment does not impose any restrictions on this.

[0195] The method steps in this embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in RAM, flash memory, ROM, PROM, EPROM, EEPROM, registers, hard disk, portable hard disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a network device or terminal device. Of course, the processor and storage medium can also exist as discrete components in the network device or terminal device.

[0196] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in this embodiment are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as an SSD.

[0197] In this embodiment, unless otherwise specified or there is a logical conflict, the terms and / or descriptions of different implementations are consistent and can be referenced by each other. The technical features of different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0198] In this embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. Other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Furthermore, for elements appearing in the singular forms "a," "an," and "the," unless explicitly specified by the context, it does not mean "one or only one," but rather "one or more than one." For example, "a device" means one or more such devices. Moreover, "at least one of..." means one or any combination of subsequent related objects. For example, "at least one of A, B, and C" includes A, B, C, AB, AC, BC, or ABC. In the textual description of this embodiment, the character " / " generally indicates an "or" relationship between the preceding and following related objects; in the formula of this embodiment, the character " / " indicates a "division" relationship between the preceding and following related objects.

[0199] It is understood that the various numerical designations used in this embodiment are merely for descriptive convenience and are not intended to limit the scope of this embodiment. The order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

Claims

1. A data processing method, characterized in that, The method includes: Obtain computation instructions, which are used to instruct the execution of a computation task; the computation task includes a first set communication operation. A first algorithm is selected from a configuration pool based on a first performance requirement of the first set communication operation; the configuration pool is used to indicate the algorithms associated with the performance requirements of the set communication operation; the first algorithm is the algorithm associated with the first performance requirement in the configuration pool; the first set communication operation is executed using the first algorithm to obtain the operation result; Prior to obtaining the calculation instructions, the method further includes: Obtain the second algorithm from the first algorithm set; wherein the first algorithm set includes multiple algorithms corresponding to the communication operation of the first set; The first set of communication operations is executed using the second algorithm; The performance data of the first set communication operation performed using the second algorithm is recorded in a performance data set; wherein, the performance data set is used to record the algorithm that performed the first set communication operation under the first performance requirement and the corresponding performance data of the algorithm; If the performance data set is empty, then according to the preset rules, the third algorithm that best matches the first performance requirement as indicated by the preset rules is obtained from the first algorithm set and used as the second algorithm; When it is determined that the performance data set is not empty, the second algorithm is obtained from the first algorithm set based on the algorithm that was executed in the most recent p times under the first performance requirement in the performance data set. If all the algorithms in the most recent p instances of executing the first set communication operation under the first performance requirement in the performance data set are the third algorithm, then an algorithm other than the algorithm in the performance data set is selected from the first algorithm set as the second algorithm; otherwise, the third algorithm is selected as the second algorithm. The third algorithm is the algorithm that best matches the first performance requirement as indicated by a preset rule.

2. The method according to claim 1, characterized in that, The configuration pool includes the correspondence between the performance requirements of the set communication operations and the associated algorithms, with each set communication operation's performance requirements associated with at least one algorithm.

3. The method according to claim 1, characterized in that, The performance requirements include at least one of the following: the number of parallel processes in the collective communication operation, the length of the message transmitted in the collective communication operation, the network topology, the network hardware type, the transmission method between the network's computing nodes, the point-to-point communication protocol used by the network, and the network's support for Remote Direct Memory Access (RDMA).

4. The method according to any one of claims 1-3, characterized in that, The performance requirements include a hash digest of the runtime parameters; The step of selecting a first algorithm from the configuration pool based on the first performance requirement of the first set communication operation includes: selecting the first algorithm from the configuration pool based on the hash digest of the running parameters included in the first performance requirement.

5. The method according to any one of claims 1-3, characterized in that, The method further includes: After the performance data set includes the performance data corresponding to the various algorithms, the algorithm with the best performance is selected from the various algorithms based on the performance data corresponding to the various algorithms, and is used as the algorithm associated with the first performance requirement, and recorded in the configuration pool.

6. A data processing apparatus, characterized in that, include: An acquisition unit is used to acquire calculation instructions, which are used to instruct the execution of a calculation task. The computation task includes a first set of communication operations; A processing unit is configured to select a first algorithm from a configuration pool based on a first performance requirement of the first aggregate communication operation; the configuration pool is configured to indicate the algorithms associated with the performance requirements of the aggregate communication operation; the first algorithm is the algorithm associated with the first performance requirement in the configuration pool. The processing unit is further configured to use the first algorithm to perform the first set communication operation to obtain the operation result; The acquisition unit is further configured to acquire a second algorithm from a first algorithm set before acquiring the calculation instruction; wherein the first algorithm set includes multiple algorithms corresponding to the first set communication operation; The processing unit is further configured to execute the first set communication operation using the second algorithm; The processing unit is further configured to record the performance data of the first set communication operation performed using the second algorithm in a performance data set; wherein, the performance data set is configured to record the algorithm that performed the first set communication operation under the first performance requirement and the corresponding performance data of the algorithm; The processing unit is further configured to, when the performance data set is empty, obtain, according to a preset rule, a third algorithm that best matches the first performance requirement from the first algorithm set and use it as the second algorithm. The processing unit is further configured to, when determining that the performance data set is not empty, obtain the second algorithm from the first algorithm set based on the algorithm that performed the first set communication operation under the first performance requirement in the most recent p times in the performance data set; if all the algorithms that performed the first set communication operation under the first performance requirement in the most recent p times in the performance data set are the third algorithm, then select an algorithm other than the algorithm in the performance data set from the first algorithm set as the second algorithm; otherwise, use the third algorithm as the second algorithm; the third algorithm is the algorithm that best matches the first performance requirement as indicated by a preset rule.

7. The data processing apparatus according to claim 6, characterized in that, The configuration pool includes the correspondence between the performance requirements of the set communication operations and the associated algorithms, with each set communication operation's performance requirements associated with at least one algorithm.

8. The data processing apparatus according to claim 6, characterized in that, The performance requirements include at least one of the following: the number of parallel processes in the collective communication operation, the length of the message transmitted in the collective communication operation, the network topology, the network hardware type, the transmission method between the network's computing nodes, the point-to-point communication protocol used by the network, and the network's support for Remote Direct Memory Access (RDMA).

9. The data processing apparatus according to any one of claims 6-8, characterized in that, The performance requirements include a hash digest of the runtime parameters; The processing unit is configured to select a first algorithm from the configuration pool based on the first performance requirement of the first set communication operation, including: The processing unit is configured to select the first algorithm from the configuration pool based on the hash digest of the running parameters included in the first performance requirement.

10. The data processing apparatus according to any one of claims 6-8, characterized in that, The processing unit is further configured to, after including the performance data corresponding to the various algorithms in the performance data set, select the algorithm with the best performance from the various algorithms based on the performance data corresponding to the various algorithms, and record it in the configuration pool as the algorithm associated with the first performance requirement.

11. A data processing apparatus, characterized in that, include: A processor and an interface, wherein the processor receives or transmits data through the interface, and the processor is configured to implement the method as described in any one of claims 1-5.

12. A computing cluster, characterized in that, include: Multiple servers; wherein at least one of the multiple servers is used to perform the method according to any one of claims 1-5.

13. A chip, characterized in that, It includes a memory and a processor, the memory being used to store computer instructions, and the processor being used to retrieve and execute the computer instructions from the memory to implement the method as described in any one of claims 1-5.

14. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-5.

15. A computer program product, characterized in that, The computer program product includes instructions that, when executed on a processor, implement the method as described in any one of claims 1-5.

Images