Data processing method and device, equipment, storage medium and computer program product

Abstract

The invention discloses a data processing method and device, equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring at least one self-defined configuration file; wherein each self-defined configuration file corresponds to a self-defined first combination, the first combination represents a combination of a set communication algorithm and a set communication protocol, the set communication protocol is a protocol bound with the set communication algorithm, and each set communication algorithm corresponds to the same self-defined data size interval; according to the at least one self-defined configuration file, under the target data transmission size and the target system configuration, testing communication performance parameters for executing the set communication operation based on each first combination; the target data transmission size is selected from the data size interval; determining an optimal combination according to the target data transmission size and the communication performance parameter corresponding to each first combination; wherein the optimal combination is used for executing the set communication task.

Description

Technical Field

[0001] This application relates to the field of wireless communication technology, and in particular to a data processing method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] In related technologies, collective communication refers to a data model in distributed systems where multiple computing units collaborate to complete data interaction, primarily implemented using collective communication algorithms. Currently, the default collective communication algorithms and protocols in collective communication libraries are used to implement collective communication, which lacks flexibility, struggles to meet complex and diverse collective communication task scenarios, and cannot guarantee that the scheduled collective communication algorithm will achieve optimal communication performance. Summary of the Invention

[0003] In view of this, embodiments of this application aim to provide a data processing method, apparatus, device, storage medium, and computer program product.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a scheduling method, the method comprising: Obtain at least one custom configuration file; wherein each custom configuration file corresponds to a custom first combination, the first combination representing a combination of a set communication algorithm and a set communication protocol, the set communication protocol being a protocol bound to the set communication algorithm, and each set communication algorithm corresponding to the same custom data size range; Based on the at least one custom configuration file, under a target data transfer size and a target system configuration, test the communication performance parameters of performing set communication operations based on each of the first combinations; the target data transfer size is selected from the data size range. Based on the target data transmission size and the communication performance parameters corresponding to each of the first combinations, an optimal combination is determined; wherein the optimal combination is used to perform a collective communication task.

[0005] Furthermore, according to at least one embodiment of this application, obtaining at least one custom configuration file includes: Get the directory; Determine if the directory is empty; If the directory is determined to be non-empty, the directory is traversed to obtain at least one custom configuration file in the directory.

[0006] Furthermore, according to at least one embodiment of this application, the method further includes: If the directory is determined to be empty, the preset native algorithm for set communication is used for scheduling. The pre-defined native algorithm for set communication and the pre-defined native protocol for set communication are used to execute set communication tasks.

[0007] Furthermore, according to at least one embodiment of this application, determining the optimal combination based on the target data transmission size and the communication performance parameters corresponding to each of the first combinations includes: Based on the target data transmission size and the communication performance parameters corresponding to each first combination, determine the set communication time corresponding to each first combination; The optimal combination is determined based on the set communication time corresponding to each of the first combinations.

[0008] Furthermore, according to at least one embodiment of this application, the communication performance parameters include bandwidth parameters and latency parameters; determining the set communication time corresponding to each first combination based on the target data transmission size and the communication performance parameters corresponding to each first combination includes: For each of the first combinations, perform the following operations: The first value is obtained by quoting the target data transmission size with the bandwidth parameter, which is included in the communication performance parameters corresponding to the first combination. The second value is obtained by multiplying the delay parameter, which is included in the communication performance parameters corresponding to the first combination, with the preset delay weight. Summing the first value and the second value yields the third value; The third value is used as the set communication time corresponding to the first combination.

[0009] Furthermore, according to at least one embodiment of this application, the step of quotienting the target data transmission size with the bandwidth parameter, which is included in the communication performance parameters corresponding to the first combination, to obtain a first value includes: Determine whether the set communication algorithm included in the first combination is a first preset algorithm; If it is determined that the set communication algorithm included in the first combination is the first preset algorithm, then the bandwidth parameter included in the communication performance parameters corresponding to the first combination is multiplied by the first loss factor to obtain the fourth value; The first value is obtained by dividing the target data transmission size by the fourth value.

[0010] Furthermore, according to at least one embodiment of this application, the step of multiplying the delay parameter, which is included in the communication performance parameters corresponding to the first combination, with a preset delay weight to obtain a second value includes: Determine whether the collection communication algorithm included in the first combination is a second preset algorithm, and determine whether the collection communication protocol included in the first combination is a preset protocol; If it is determined that the first combination includes the second preset algorithm as the set communication algorithm, and the first combination includes the preset protocol as the set communication protocol, then the delay parameter included in the communication performance parameters corresponding to the first combination is multiplied by the second loss factor to obtain the fifth value; The second value is obtained by multiplying the fifth value with the preset delay weight.

[0011] Furthermore, according to at least one embodiment of this application, the number of the first combinations is N, where N is an integer greater than 1, and determining the optimal combination based on the set communication time corresponding to each of the first combinations includes: Determine the minimum value from the communication times of the sets corresponding to the N first combinations; If the minimum value is less than or equal to the set communication time corresponding to the preset set communication native algorithm, then the first combination corresponding to the minimum value is taken as the optimal combination.

[0012] This application provides a data processing apparatus, including: An acquisition module is used to acquire at least one custom configuration file; wherein each custom configuration file corresponds to a custom first combination, the first combination representing a combination of a set communication algorithm and a set communication protocol, the set communication protocol being a protocol bound to the set communication algorithm, and each set communication algorithm corresponding to the same custom data size range; A processing module is configured to test, based on the at least one custom configuration file, under a target data transmission size and a target system configuration, communication performance parameters for performing a collection communication operation based on each of the first combinations; the target data transmission size is selected from the data size range; and, based on the target data transmission size and the communication performance parameters corresponding to each of the first combinations, determine an optimal combination from at least one of the first combinations; wherein the optimal combination is used to perform the collection communication task.

[0013] This application provides a scheduler, including a processor and a memory for storing computer programs that can run on the processor. When the processor runs the computer program, it performs the steps of any of the methods described above.

[0014] At least one embodiment of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0015] At least one embodiment of this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the above-described embodiments.

[0016] The data processing method, apparatus, device, storage medium, and computer program product provided in this application embodiment include: obtaining at least one custom configuration file; wherein each custom configuration file corresponds to a custom first combination, the first combination representing a combination of a set communication algorithm and a set communication protocol, the set communication protocol being a protocol bound to the set communication algorithm, and each set communication algorithm corresponding to the same custom data size range; according to the at least one custom configuration file, under a target data transmission size and a target system configuration, testing communication performance parameters of performing set communication operations based on each first combination; the target data transmission size being selected from the data size range; and determining an optimal combination based on the target data transmission size and the communication performance parameters corresponding to each first combination; wherein the optimal combination is used to perform set communication tasks.

[0017] Using the technical solution provided in this application embodiment, the custom configuration file contains configuration information for testing the communication performance parameters corresponding to the first combination. Moreover, the first combination is customized by the user according to the actual collective communication task scenario. Thus, by determining the optimal combination based on the communication performance parameters corresponding to at least one first combination, and using the collective communication algorithm contained in the optimal combination and the collective communication protocol bound to the collective communication algorithm to execute the collective communication task, the best communication performance can be achieved in complex and diverse collective communication task scenarios. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating the implementation flow of the data processing method in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the specific implementation flow of the data processing method in the embodiments of this application; Figure 3 This is a schematic diagram of the composition structure of the data processing device according to an embodiment of this application; Figure 4 This is a schematic diagram of the composition structure of the scheduler in an embodiment of this application. Detailed Implementation

[0019] Before introducing the technical solutions of the embodiments of this application, the relevant technologies will be introduced first.

[0020] In related technologies, with the rapid development of deep learning, data analysis, and high-performance computing (HPC), parallel computing based on multiple graphics processing units (GPUs) has become a key technology for improving computing power and efficiency. Collective communication, as a common communication mode in multi-GPU parallel computing, directly affects the overall system efficiency. Against this backdrop, various Collective Communications Libraries (CCLs) have emerged, optimizing communication performance in multi-GPU environments through different communication modes and protocols.

[0021] In multi-GPU systems, aggregate communication operations, including AllReduce, AllGather, and Broadcast, require efficient data transfer between multiple GPUs. The bottleneck in communication efficiency primarily lies in the selection of the aggregate communication protocol and the rational scheduling of communication paths. Existing communication libraries optimize communication paths through aggregate communication algorithms such as Ring and Tree, and introduce aggregate communication protocols such as LL (Low Latency), LL128 (Low Latency 128), and Simple to further improve performance in data transfers of varying scales.

[0022] However, with the increasing prevalence of heterogeneous GPU architectures and the growing complexity of communication tasks, existing dynamic scheduling strategies often struggle to achieve optimal communication efficiency when faced with complex tasks. Static scheduling algorithms, due to their precise calculations and optimizations under predefined conditions, can provide efficient communication scheduling schemes for given GPU topologies and task loads. Therefore, static scheduling optimization has become a research hotspot for multi-GPU heterogeneous systems, as this optimization method can effectively improve overall communication efficiency and reduce latency and bandwidth bottlenecks.

[0023] Related technologies mainly include scheduling algorithms from collective communication libraries such as NCCL and MSCCL. NCCL is a widely used collective communication library specifically designed for collective communication in multi-GPU environments. NCCL's core algorithms include Ring and Tree, which effectively manage data transmission in multi-node environments. However, NCCL's scheduling algorithm limits users' ability to optimize according to specific task requirements, especially when facing complex and dynamic network topologies, failing to provide sufficient flexibility. The MSCCL scheduling algorithm lacks flexibility and practical feasibility; it defaults to matching the first communication algorithm that meets the running conditions, and cannot guarantee scheduling to a collective communication algorithm with optimal communication capabilities.

[0024] In related technologies, the scheduling of collective communication algorithms has the following technical shortcomings: First, the static scheduling strategy lacks flexibility.

[0025] Currently, most collective communication libraries such as NCCL and MSCCL rely on preset static scheduling strategies for selecting collective communication algorithms and protocols. While static scheduling strategies can demonstrate good performance in certain specific GPU topologies and communication scenarios, they lack sufficient flexibility for complex and diverse task scenarios. They cannot dynamically adjust according to actual communication needs, hardware architecture, cluster topology, etc., thus limiting communication performance.

[0026] Second, the limitations of protocol selection.

[0027] Current recursive communication libraries typically rely on simple combinations of recursive communication algorithms and protocols, lacking fine-grained optimization techniques. For example, while the LL protocol is suitable for low-latency, small data packet transmission, its efficiency decreases in medium-sized data transmissions, and the Simple protocol, while suitable for large data packet transmissions, is inefficient with small data. These technologies struggle to automatically select the optimal algorithm and protocol combination for various communication tasks, impacting the overall communication performance of the system.

[0028] Third, the impact of data size on performance is ignored.

[0029] Related technologies often fail to adequately consider the impact of varying data sizes on communication bandwidth and latency when executing scheduling algorithms. When processing large or small datasets, algorithm selection often falls short of optimal performance, leading to increased communication time. This is particularly problematic in multi-tasking scenarios, where they cannot flexibly adapt to changes in data size across different tasks.

[0030] In summary, the existing technologies that utilize the default collective communication algorithms and protocols in collective communication libraries lack flexibility, struggle to meet the demands of complex and diverse collective communication tasks, and cannot guarantee optimal communication performance from the scheduled algorithms. In other words, when the scheduler executes the scheduling algorithm, it cannot consider actual communication needs, hardware architecture, cluster topology, data scale, and other factors when scheduling the collective communication algorithm and protocol, potentially impacting the communication performance of collective communication tasks, such as communication time.

[0031] Based on this, in this embodiment of the application, at least one custom configuration file is obtained; wherein each custom configuration file corresponds to a custom first combination, the first combination representing a combination of a set communication algorithm and a set communication protocol, the set communication protocol being a protocol bound to the set communication algorithm, and each set communication algorithm corresponding to the same custom data size range; according to the at least one custom configuration file, under a target data transmission size and a target system configuration, the communication performance parameters of performing set communication operations based on each first combination are tested; the target data transmission size is selected from the data size range; based on the target data transmission size and the communication performance parameters corresponding to each first combination, an optimal combination is determined; wherein the optimal combination is used to perform set communication tasks. That is, the custom configuration file contains configuration information for testing the communication performance parameters corresponding to the first combination, and the first combination is customized by the user according to the actual set communication task scenario. Thus, by determining the optimal combination based on the communication performance parameters corresponding to at least one first combination, and using the set communication algorithm contained in the optimal combination and the set communication protocol bound to the set communication algorithm to perform set communication tasks, optimal communication performance can be achieved in complex and diverse set communication task scenarios.

[0032] See Figure 1 , Figure 1 This is a schematic diagram illustrating the implementation flow of the data processing method in an embodiment of this application, as shown below. Figure 1 As shown, the process applied to the scheduler includes steps 101 to 103: Step 101: Obtain at least one custom configuration file; wherein each custom configuration file corresponds to a custom first combination, the first combination representing a combination of a set communication algorithm and a set communication protocol, the set communication protocol being a protocol bound to the set communication algorithm, and each set communication algorithm corresponding to the same custom data size range.

[0033] Here, a custom configuration file can refer to a user-generated XML configuration file, or be described as an XML format file, XML file, etc.

[0034] Here, the custom configuration file may contain configuration information for testing the communication performance parameters corresponding to the first combination. This configuration information is used to test the communication performance parameters of performing set communication operations based on the first combination under the target data transmission size and the target system configuration.

[0035] Here, the custom first combination can be used to implement user-defined set communication operations through user-defined set communication algorithms and set communication protocols. For example, user-defined set communication operations such as AllReduce can be implemented through user-defined set communication algorithms such as the Ring algorithm and set communication protocols such as the Simple protocol. Here, a set communication operation can be implemented using different set communication algorithms.

[0036] Here, the set communication algorithm refers to the specific data transmission method used to implement set communication operations, such as the Ring algorithm and the Tree algorithm.

[0037] Here, a collection of communication protocols refers to a set of rules that define how data is packaged, sent, and received. Common protocols include LL (Low Latency), LL128, and Simple.

[0038] Here, users can define the same data size range for each of the collective communication algorithms based on the data size of the current collective communication task. For example, one first combination may include the Ring algorithm, and another first combination may include the Tree algorithm. The two collective communication algorithms may each correspond to the same custom data size range, assuming the range specified by the user is 1B-4GB. Here, a set communication algorithm can be bound to a set communication protocol. For example, a set communication algorithm such as the Ring algorithm can be bound to a set communication protocol such as the LL128 protocol to obtain a first combination, or a set communication algorithm such as the Ring algorithm can be bound to a set communication protocol such as the Simple protocol to obtain a first combination.

[0039] In this application, the scheduling algorithm used by the scheduler supports users in customizing the combination of algorithms and protocols in a multi-GPU cluster environment, thereby optimizing communication performance. This feature allows users to select the most suitable collective communication algorithm based on various factors such as the specific communication task, hardware architecture, and cluster topology. In practical applications, a flexible programming interface allows users to customize the collective communication algorithm and specify the data size range corresponding to the collective communication algorithm based on the data size of the current collective communication task. This enables users to not only specify different data partitioning strategies but also precisely control the order, path, and operation mode of data transmission.

[0040] Here, the flexibility of this application is reflected in supporting users to select different aggregate communication algorithms and protocols for different communication stages. Users can define a custom aggregate communication algorithm and bind it to different aggregate communication protocols (such as LL, LL128, and Simple) to obtain different first combinations, generating a feasibility XML file containing the execution steps of the aggregate communication operation, i.e., a custom configuration file. This custom configuration file can be used to test communication performance parameters, such as bandwidth and latency parameters, based on the first combination under a target data transmission size and target system configuration; the target data transmission size is selected from a custom data size range. Users can also customize optimized combination schemes for different data scales and communication modes. Users can define a custom AllReduce aggregate communication operation and implement it using a custom aggregate communication algorithm (Ring algorithm) and a aggregate communication protocol (Simple protocol). Through such flexible combinations, users can design algorithms to reduce inter-node conflicts for fully connected networks or design more efficient communication paths for sparse networks. Furthermore, the above design allows users to test different combinations with very low overhead, thereby enabling rapid iterative optimization.

[0041] Compared to traditional algorithms like Ring and Tree, this application allows users to customize set communication algorithms and protocols based on GPU topology, generating corresponding XML configuration files (custom configuration files). Users can place the generated XML files in a specified directory (algorithms) for the scheduling algorithm to select from. Compared to traditional set communication libraries, this approach effectively overcomes the limitations of static optimization for specific hardware and communication modes in existing libraries. Users can customize set communication algorithms and protocols to meet their specific needs for heterogeneous systems or specific topologies (such as GPU topologies).

[0042] In some embodiments, obtaining at least one custom configuration file includes: Get the directory; Determine if the directory is empty; If the directory is determined to be non-empty, the directory is traversed to obtain at least one custom configuration file in the directory.

[0043] Here, if the directory contains at least one custom configuration file, then the directory is not empty; otherwise, if the directory does not contain at least one custom configuration file, then the directory is empty.

[0044] In some embodiments, the method further includes: If the directory is determined to be empty, the preset native algorithm for set communication is used for scheduling. The pre-defined native algorithm for set communication and the pre-defined native protocol for set communication are used to execute set communication tasks.

[0045] Here, the preset native set communication algorithm can refer to a default set communication algorithm preset by the system, such as the Ring algorithm.

[0046] Here, the preset native protocol for aggregate communication can refer to a default aggregate communication protocol that is pre-set by the system and bound to the default aggregate communication algorithm, such as the LL128 protocol.

[0047] Step 102: Based on the at least one custom configuration file, under the target data transmission size and target system configuration, test the communication performance parameters of performing set communication operations based on each of the first combinations; the target data transmission size is selected from the data size range.

[0048] Here, the custom configuration file details all the information needed to test the communication performance parameters for performing set communication operations based on the first combination under the target data transfer size and target system configuration.

[0049] Here, the target data transfer size can be arbitrarily or randomly selected from the data size range. For example, if the user-defined data size range is 1GB-4GB, then the target data transfer size can be 1GB.

[0050] Here, the target system configuration can be specified by the user in a custom configuration file according to actual needs. For example, the target system configuration can be a multi-GPU architecture configuration, a heterogeneous configuration, etc.

[0051] Here, each of the first combinations performing a set communication operation can be understood as the set communication algorithm in each first combination performing a set communication operation using the bound set communication protocol.

[0052] Here, the communication performance parameters for the first combination to perform a set communication operation include bandwidth parameters and latency parameters. The bandwidth parameter represents the data transmission rate, and the bandwidth is used to determine the data transmission time during the set communication operation based on the first combination, in conjunction with the target data transmission size. The latency parameter represents the data processing time during the set communication operation based on the first combination, and data processing may include calculations on the data.

[0053] Step 103: Determine the optimal combination based on the target data transmission size and the communication performance parameters corresponding to each of the first combinations; wherein the optimal combination is used to perform the collective communication task.

[0054] Here, the method further includes: executing a set communication task using the optimal combination. Specifically, executing the set communication task using the optimal combination can be understood as executing the set communication task using the set communication algorithms included in the optimal combination and the set communication protocol bound to those algorithms.

[0055] In some embodiments, determining the optimal combination based on the target data transmission size and the communication performance parameters corresponding to each of the first combinations includes: Based on the target data transmission size and the communication performance parameters corresponding to each first combination, determine the set communication time corresponding to each first combination; The optimal combination is determined based on the set communication time corresponding to each of the first combinations.

[0056] In some embodiments, the communication performance parameters include bandwidth parameters and latency parameters; determining the set communication time corresponding to each first combination based on the target data transmission size and the communication performance parameters corresponding to each first combination includes: For each of the first combinations, perform the following operations: The first value is obtained by quoting the target data transmission size with the bandwidth parameter, which is included in the communication performance parameters corresponding to the first combination. The second value is obtained by multiplying the delay parameter, which is included in the communication performance parameters corresponding to the first combination, with the preset delay weight. Summing the first value and the second value yields the third value; The third value is used as the set communication time corresponding to the first combination.

[0057] Here, communication performance parameters include bandwidth parameters and delay parameters. The bandwidth parameter represents the data transmission rate, and this bandwidth is used to determine the data transmission time in conjunction with the target data transmission size. The delay parameter represents the data processing time during the execution of a set communication operation on the data. Data processing may include calculations on the data.

[0058] Here, the target data transmission size is divided by the bandwidth parameter, which is included in the communication performance parameters corresponding to the first combination, to obtain a first value. The first value represents the data transmission time during the process of performing a set communication operation on the data based on the first combination.

[0059] Here, the communication performance parameters corresponding to the first combination include the delay parameter, which represents the time required for data processing during the execution of a set communication operation based on the first combination.

[0060] Here, the preset delay weight is a numerical value used to measure the relative importance of the delay parameter, which is included in the overall communication performance parameters, within the overall communication performance. The preset delay weight can be adjusted according to factors such as task type and system configuration to highlight the impact of the delay parameter, which is included in the communication performance parameters, in certain scenarios.

[0061] Here, by introducing a preset delay weight, the system can flexibly adjust the proportion of delay parameters included in the overall performance evaluation. This method of introducing a preset delay weight makes the scheduling algorithm more adaptable and enables it to make optimal decisions under different task requirements.

[0062] Here, the first value and the second value are summed to obtain a third value, which is used as the set communication time corresponding to the first combination. It can be understood that the set communication time corresponding to the first combination may include the sum of the data transmission time and the data processing time during the set communication operation on the data based on the first combination.

[0063] In practical applications, it's also necessary to determine whether the user-defined or selected algorithm is the first preset algorithm, i.e., the Tree algorithm. If the user-defined or selected algorithm is the first preset algorithm (Tree algorithm), then a bandwidth loss factor needs to be considered. This is because the Tree algorithm improves communication efficiency by decomposing the data transmission task into multiple sub-tasks and using a hierarchical parallel processing structure. Considering that using the first preset algorithm may cause some performance loss during actual communication due to network card latency or network receive queue congestion, a bandwidth loss factor, i.e., the first loss factor, is introduced to make the actual data transmission time calculated based on the target data transmission size and bandwidth closer to reality.

[0064] Based on this, in some embodiments, the step of quoting the target data transmission size with the bandwidth parameter, which is included in the communication performance parameters corresponding to the first combination, to obtain the first value includes: Determine whether the set communication algorithm included in the first combination is a first preset algorithm; If it is determined that the set communication algorithm included in the first combination is the first preset algorithm, then the bandwidth parameter included in the communication performance parameters corresponding to the first combination is multiplied by the first loss factor to obtain the fourth value; The first value is obtained by dividing the target data transmission size by the fourth value.

[0065] Here, the first preset algorithm can be the Tree algorithm, etc.

[0066] Here, the bandwidth parameter, which is included in the communication performance parameters corresponding to the first combination, is multiplied by the first loss factor to obtain the fourth value, which can be understood as updating or correcting the bandwidth parameter.

[0067] Here, if it is determined that the set communication algorithm included in the first combination is not the first preset algorithm, then the bandwidth parameter is not updated.

[0068] Here, the first loss factor can be preset according to the actual situation.

[0069] Here, a first data interval containing the custom data size interval can be determined from the pre-divided data interval. Based on the preset correspondence between the pre-divided data interval and the bandwidth loss factor, the bandwidth loss factor corresponding to the first data interval can be determined, and the determined bandwidth loss factor can be used as the first loss factor.

[0070] For example, suppose the data intervals are divided into [1B, 128KB], [128KB, 32MB], and [32MB, 4GB], and the custom data size interval is [1B, 50KB]. From the pre-divided data intervals, the first data interval containing the custom data size interval is determined to be [1B, 128KB]. If the bandwidth loss factor set for the first data interval [1B, 128KB] is 0.3, then the first loss factor is 0.3.

[0071] In this embodiment of the application, by introducing a judgment mechanism based on a first preset algorithm and combining it with a bandwidth loss factor for bandwidth correction, communication performance can be evaluated more accurately.

[0072] In practical applications, it's also necessary to determine whether the user-defined or selected algorithm is the second preset algorithm, i.e., the Ring algorithm, and whether the bound protocol is the preset protocol, i.e., Simple. If both conditions are met, the latency parameter needs to be adjusted. This is because the Ring algorithm transmits data sequentially, with each node receiving and forwarding data in turn. This method is particularly sensitive to latency. Considering that the second preset algorithm may deviate from the ideal situation due to various reasons such as poor network conditions during actual communication, setting a second attenuation factor, i.e., the latency attenuation factor, can more accurately reflect the actual communication latency and ensure that the optimized algorithm adapts to network conditions. Although the Simple protocol has a relatively straightforward design, it is still affected by network latency. Introducing an attenuation factor helps to adjust performance.

[0073] Based on this, in some embodiments, the step of multiplying the delay parameter, which is included in the communication performance parameters corresponding to the first combination, with a preset delay weight to obtain the second value includes: Determine whether the collection communication algorithm included in the first combination is a second preset algorithm, and determine whether the collection communication protocol included in the first combination is a preset protocol; If it is determined that the first combination includes a set communication algorithm that is a second preset algorithm, and the first combination includes a set communication protocol that is the preset protocol, then the delay parameter included in the communication performance parameters corresponding to the first combination is multiplied by the second loss factor to obtain the fifth value; The second value is obtained by multiplying the fifth value with the preset delay weight.

[0074] Here, the second preset algorithm can be the Ring algorithm, etc., and the preset protocol can be the Simple protocol, etc.

[0075] Here, the delay parameter, which is included in the communication performance parameters corresponding to the first combination, is multiplied by the second loss factor to obtain the fifth value, which can be understood as updating or correcting the delay parameter.

[0076] Here, the latency parameter included in the communication performance parameters will not be updated in one of the following cases: The first combination includes a collection communication algorithm that is not the second preset algorithm, or the first combination includes a collection communication protocol that is not the preset protocol. The first combination includes a second preset algorithm for the collective communication algorithm, but the first combination includes a different preset protocol for the collective communication algorithm. The first combination includes a collection communication algorithm that is not the second preset algorithm, but the first combination includes a collection communication protocol that is the preset protocol. The first combination includes a collection communication algorithm that is not the second preset algorithm, and the first combination includes a collection communication protocol that is not the preset protocol.

[0077] Here, the second loss factor can be preset according to the actual situation.

[0078] Here, a first data interval containing the custom data size interval is determined from the pre-divided data interval. Based on the preset correspondence between the pre-divided data interval and the delay loss factor, a delay loss factor corresponding to the first data interval is determined, and the determined delay loss factor is used as the second loss factor.

[0079] For example, suppose the data intervals are divided into [1B, 128KB], [128KB, 32MB], and [32MB, 4GB], and the custom data size interval is [1B, 50KB]. From the pre-divided data intervals, the first data interval containing the custom data size interval is determined to be [1B, 128KB]. If the latency loss factor set for the first data interval [1B, 128KB] is 1.3, then the second loss factor is 1.3.

[0080] In this embodiment of the application, by introducing a judgment mechanism for the collective communication algorithm and protocol, and combining it with a delay loss factor for delay correction, the communication performance can be evaluated more accurately, which helps to improve the rationality of communication scheduling and improve the operating efficiency of the system.

[0081] In some embodiments, the number of the first combinations is N, where N is an integer greater than 1. Determining the optimal combination based on the set communication time corresponding to each of the first combinations includes: Determine the minimum value from the communication times of the sets corresponding to the N first combinations; If the minimum value is less than or equal to the set communication time corresponding to the preset set communication native algorithm, then the first combination corresponding to the minimum value is taken as the optimal combination.

[0082] Specifically, the process of determining the optimal combination may include: The set communication time corresponding to the preset set communication native algorithm is taken as the first time; Based on the communication time of the first time and the first set corresponding to the first combination, determine the minimum value and use the minimum value as the initial optimal value; Based on the optimal value and the communication time of the set corresponding to the second first combination, determine the minimum value and take the minimum value as the new optimal value; And so on, until the last first combination. If there is a first combination among the N first combinations whose set communication time is less than the first time, the first combination corresponding to the last determined optimal value shall be the optimal combination, where N is an integer greater than 1.

[0083] Here, if none of the N first combinations contains a first combination with a communication time shorter than the first time, then the preset native algorithm for collective communication and the preset protocol for collective communication are combined to form the optimal combination. The preset protocol for collective communication is pre-defined and bound to the preset native algorithm for collective communication.

[0084] Here, the set communication time corresponding to the preset set communication native algorithm can be pre-set.

[0085] Here, the following operations can also be performed to determine the co-communication time corresponding to the preset co-communication native algorithm: Test the performance parameters of performing co-communication operations based on the preset co-communication native algorithm and using the bound co-communication native protocol under the target data transmission size and the target system configuration; the performance parameters include bandwidth and latency; quotient the target data transmission size with the bandwidth to obtain a sixth value; multiply the latency with a preset latency weight to obtain a seventh value; sum the sixth value and the seventh value to obtain the co-communication time corresponding to the preset co-communication native algorithm. Here, the preset native algorithm for set communication and the preset native protocol for set communication are combined in an optimal way to perform set communication tasks by using the native algorithm for set communication and the native protocol for set communication bound to the native algorithm for set communication.

[0086] Here, the preset native set communication algorithm can refer to a default set communication algorithm preset by the system, such as the Ring algorithm.

[0087] Here, the preset native protocol for aggregate communication can refer to a default aggregate communication protocol that is pre-set by the system. The default aggregate communication protocol is bound to the default aggregate communication algorithm, such as the LL128 protocol.

[0088] The embodiments of this application have the following advantages: (1) The custom configuration file contains configuration information for testing the communication performance parameters corresponding to the first combination. Moreover, the first combination is customized by the user according to the actual collective communication task scenario. Thus, the optimal combination is determined based on the communication performance parameters corresponding to at least one first combination. The collective communication task is executed by the collective communication algorithm contained in the determined optimal combination and the collective communication protocol bound to the collective communication algorithm. This ensures that the best communication performance is achieved in complex and diverse collective communication task scenarios, and improves the efficiency, flexibility and adaptability of scheduling collective communication.

[0089] This paper proposes a novel static scheduling algorithm that aims to improve the efficiency of ensemble communication by optimizing the combination of ensemble communication algorithms and protocols, and to provide optimal scheduling schemes for different data scales and communication tasks. It supports user-defined algorithm and protocol combinations, allowing users to define suitable ensemble communication algorithms and bind corresponding ensemble communication protocols based on specific communication tasks, hardware environments, and data transmission scales, thereby determining the optimal combination scheme. This not only expands upon the limitations of related technologies in terms of functionality but also provides significant improvements in algorithm flexibility and communication efficiency, offering a new solution for ensemble communication optimization in multi-GPU cluster environments.

[0090] This paper addresses the limitations of existing collective communication libraries in static scheduling and protocol selection by proposing a flexible and efficient static scheduling optimization algorithm. By considering communication bandwidth, latency, and data transmission scale, the algorithm selects the optimal combination of algorithm and protocol before the collective communication task begins, significantly improving communication efficiency, reducing communication latency, and increasing bandwidth utilization.

[0091] Here, addressing the lack of flexibility and adaptability in related technologies' collective communication libraries, a suitable collective communication algorithm can be defined and bound to the corresponding collective communication protocol before the collective communication task is executed, based on the characteristics of different tasks, hardware environment, data transmission scale, etc. Then, combined with the communication bandwidth, latency and other parameters obtained from testing, the optimal combination of collective communication algorithm and protocol can be selected. This can maximize the efficiency of collective communication and ensure that the best performance can be achieved in various scenarios.

[0092] Here, users are allowed to customize combinations of algorithms and protocols in a multi-GPU cluster environment to generate executable XML files, i.e., custom configuration files. This feature enables users to design the most suitable aggregate communication strategy based on the specific hardware architecture, network topology, and characteristics of the communication task. By supporting custom configuration, the system enhances its adaptability and can adjust according to real-time needs during operation, thereby effectively handling complex computing scenarios.

[0093] Here, an XML configuration file mechanism is introduced, allowing users to generate XML files containing communication performance parameters for testing combined algorithmic and protocol-based communication operations under target data transfer sizes and system configurations. The scheduler parses and executes this file to obtain communication performance parameters, including bandwidth and latency information, and then makes optimization choices based on these parameters. This mechanism overcomes the limitations of static optimization in traditional libraries, ensuring that the algorithm's performance can be quickly adjusted to specific environmental changes.

[0094] Here, to improve the flexibility and accuracy of algorithm selection, the scheduler sets a depreciation factor value for bandwidth and delay calculations based on a pre-defined data interval during the static scheduling process. This depreciation factor setting effectively compensates for potential delay and bandwidth losses in actual communication, thereby helping the algorithm maintain high performance under complex network conditions.

[0095] This allows for improved communication efficiency across multi-GPU clusters, enabling the system to optimize resource utilization to meet diverse computing demands and ultimately achieve efficient task scheduling and execution.

[0096] This allows users to customize combinations of aggregated communication algorithms and protocols based on specific communication tasks, hardware architectures, and cluster topologies. This flexibility enables users to design optimal communication strategies for specific application scenarios, overcoming the limitations of fixed algorithm selection in related technologies.

[0097] Here, the characteristics of the communication task are analyzed before execution, and the optimal combination of algorithm and communication protocol is selected. Compared with the default algorithms of commonly used libraries such as Tree and Ring, this scheduling mechanism can more effectively cope with complex data transmission requirements, thereby significantly improving communication efficiency.

[0098] Here, multiple performance metrics, such as bandwidth, latency, and target data transmission size (BufferSize), are comprehensively considered in the selection of the optimal combination. The scheduling algorithm has comprehensive evaluation capabilities during the selection process. This capability enables the algorithm and protocol combination selected by the scheduling algorithm to achieve higher communication performance in practical applications, reducing the performance loss caused by ignoring specific parameters.

[0099] Here, different combinations of algorithms and protocols can be tested with low overhead; that is, the communication performance of algorithm and protocol combinations can be tested through a custom configuration file.

[0100] Here, for applications in multi-GPU and heterogeneous systems, the bandwidth and latency parameters obtained from the test can be dynamically adjusted to ensure that the calculated communication time is more in line with the actual situation, thereby ensuring that the best communication efficiency can be achieved under different set communication task requirements, and effectively cope with complex computing environments in practical applications.

[0101] Here, the scheduling of collective communication outperforms related technologies in terms of efficiency, flexibility, and adaptability, providing a new solution for efficient communication in multi-GPU cluster environments.

[0102] See Figure 2 , Figure 2 This is a schematic diagram illustrating the specific implementation flow of the data processing method in the embodiments of this application, as shown below. Figure 2 As shown, steps 201 to 212 are included: Step 201: The scheduler obtains the directory and determines whether the directory is empty. If the directory is determined to be empty, proceed to step 202; otherwise, proceed to step 203.

[0103] Here, if the directory contains at least one custom configuration file, then the directory is not empty; otherwise, if the directory does not contain at least one custom configuration file, then the directory is empty.

[0104] Here, a custom configuration file can refer to a user-generated XML configuration file, or be described as an XML format file, XML file, etc.

[0105] Here, each of the custom configuration files corresponds to a custom first combination. The first combination represents a combination of a set communication algorithm and a set communication protocol. The set communication protocol is a protocol bound to the set communication algorithm. Each set communication algorithm corresponds to the same custom data size range.

[0106] Here, the custom first combination can be used to implement user-defined collection communication operations. For example, a custom collection communication operation, AllReduce, can be implemented using a custom collection communication algorithm, namely the Ring algorithm, and a collection communication protocol, namely the Simple protocol. Here, a set communication operation can be implemented using different set communication algorithms.

[0107] Here, the set communication algorithm refers to the specific data transmission method used to implement set communication operations, such as the Ring algorithm and the Tree algorithm.

[0108] Here, a collection of communication protocols refers to a set of rules that define how data is packaged, sent, and received. Common protocols include LL (Low Latency), LL128, and Simple.

[0109] Here, the user makes the choice of a combination of collective communication algorithms and collective communication protocols.

[0110] This application proposes a new scheduling algorithm that allows users to customize the combination of set communication algorithms and set communication protocols in a multi-GPU cluster environment, thereby optimizing communication performance.

[0111] Here, users can define the same data size range for each of the aforementioned set communication algorithms. For example, one first combination includes the Ring algorithm, and another first combination includes the Tree algorithm. The two set communication algorithms each correspond to the same custom data size range, assuming the user-specified range is 1B-4GB. Here, a custom set communication algorithm can be bound to a set communication protocol. For example, a set communication algorithm such as the Ring algorithm can be bound to a set communication protocol such as the LL128 protocol to obtain a first combination, or a set communication algorithm such as the Ring algorithm can be bound to a set communication protocol such as the Simple protocol to obtain a first combination.

[0112] In this application, the scheduling algorithm used by the scheduler supports users in customizing the combination of algorithms and protocols in a multi-GPU cluster environment, thereby optimizing communication performance. This feature allows users to design the most suitable collective communication algorithm based on specific communication tasks, hardware architecture, and cluster topology. In practical applications, a flexible programming interface allows users to customize the collective communication algorithm and specify the data size range corresponding to the collective communication algorithm based on the data transmission size of the current collective communication task. This enables users to not only specify different data partitioning strategies but also precisely control the order, path, and operation mode of data transmission.

[0113] Here, the flexibility of this application is reflected in supporting users to select different aggregate communication algorithms and protocols for different communication stages. Users can define a custom aggregate communication algorithm and bind it to different aggregate communication protocols (such as LL, LL128, and Simple) to obtain different first combinations, generating a feasibility XML file containing the execution steps of the aggregate communication operation, i.e., a custom configuration file. This custom configuration file can be used to test communication performance parameters, such as bandwidth and latency parameters, based on the first combination under a target data transmission size and target system configuration; the target data transmission size is selected from a custom data size range. Users can also customize optimized combination schemes for different data scales and communication modes. Users can define a custom AllReduce aggregate communication operation and implement it using a custom aggregate communication algorithm (Ring algorithm) and a aggregate communication protocol (Simple protocol). Through such flexible combinations, users can design algorithms to reduce inter-node conflicts for fully connected networks or design more efficient communication paths for sparse networks. Furthermore, the above design allows users to test different combinations with very low overhead, thereby enabling rapid iterative optimization.

[0114] Compared to traditional algorithms like Ring and Tree, this application allows users to customize set communication algorithms and protocols based on GPU topology, generating corresponding XML configuration files (custom configuration files). Users can place the generated XML files in a specified directory (algorithms) for the scheduling algorithm to select from. Compared to traditional set communication libraries, this approach effectively overcomes the limitations of static optimization for specific hardware and communication modes in existing libraries. Users can customize set communication algorithms and protocols to meet their specific needs for heterogeneous systems or specific topologies (such as GPU topologies).

[0115] Step 202: When the scheduler determines that the directory is empty, it schedules the preset native algorithm for set communication, and executes the set communication task using the preset native algorithm for set communication and the preset native protocol for set communication.

[0116] Here, the preset native set communication algorithm can refer to a default set communication algorithm preset by the system, such as the Ring algorithm.

[0117] Here, the preset native protocol for aggregate communication can refer to a default aggregate communication protocol that is pre-set by the system. The default aggregate communication protocol is bound to the default aggregate communication algorithm, such as the LL128 protocol.

[0118] Step 203: The scheduler traverses the XML files, i.e., custom configuration files, in the directory. Each custom configuration file corresponds to a custom first combination, which represents a combination of a set communication algorithm and a set communication protocol, wherein the set communication protocol is a protocol bound to the set communication algorithm.

[0119] Here, the optimization of the static scheduling process is completed.

[0120] Here, when there is no XML file (custom configuration file) in the directory (algorithms), it means that the user does not specify a particular set communication algorithm. In this case, a native algorithm from a set communication library such as Tree or Ring is selected as the default scheduling algorithm. When there is an XML file (custom configuration file) in the directory (algorithms), the scheduling algorithm needs to traverse all XML files (custom configuration files) in the directory.

[0121] Here, the first step in optimizing static scheduling is to traverse one or more XML files, i.e., custom configuration files, stored in the directory (algorithms).

[0122] Step 204: Execute the XML file, i.e., the custom configuration file, to test the communication performance parameters of the set communication operation based on the first combination under the target data transfer size (BufferSize) and the target system configuration (such as multi-GPU architecture configuration, etc.), including bandwidth (bw) and latency (lat), where the target data transfer size is selected from the data size range.

[0123] Here, the target data transfer size is arbitrarily selected from a user-defined data size range. For example, if the user-defined data size range is 1GB-4GB, then the target data transfer size can be 1GB.

[0124] Here, the target system configuration can be specified by the user in a custom configuration file, such as GPU architecture configuration, multi-GPU architecture configuration, etc.

[0125] Here, the first combination performing set communication operation can be understood as the set communication algorithm in the first combination using the bound set communication protocol to perform set communication operation.

[0126] Here, the XML file, i.e., the custom configuration file, details all the information needed to test the communication performance parameters of the first combination performing aggregate communication operations under the target data size (BufferSize) and target system configuration (such as network topology, GPU architecture, multi-GPU architecture, heterogeneous, etc.). The first combination includes a combination of aggregate communication algorithms and protocols, and the communication performance parameters may include bandwidth (bw) parameters and latency (lat) parameters. The scheduler will traverse multiple XML configuration files (i.e., custom configuration files) in the directory according to the needs of different communication tasks. Based on these multiple XML configuration files, it will test the communication performance parameters of each first combination performing aggregate communication operations under the target data transmission size and target system configuration (such as network topology, GPU architecture, etc.). The communication performance parameters include bandwidth and latency parameters. The bandwidth parameter represents the data transmission rate, which is used to determine the data transmission time during the aggregate communication operation based on the first combination, in conjunction with the target data transmission size. The latency parameter represents the data processing time during the aggregate communication operation based on the first combination. Data processing may include calculations on the data, providing basic data support for subsequent optimization choices.

[0127] Here, the scheduler parses the XML configuration file and executes the collective communication process contained within it. It then tests and obtains the corresponding communication performance metrics, which are the communication performance parameters of the specified combination of collective communication algorithm and protocol under the current target data transmission size (BufferSize) and target system configuration. These performance parameters include bandwidth (bw) and latency (lat). The bandwidth and latency performance metrics of the specified combination of collective communication algorithm and protocol under the current target data transmission size (BufferSize) and target system configuration directly affect the actual communication time and are key factors in selecting the optimal algorithm and protocol combination.

[0128] Step 205: The scheduler determines whether the collective communication algorithm in the first combination is the first preset algorithm, namely the Tree algorithm; if it is determined that the collective communication algorithm in the first combination is the first preset algorithm, then step 206 is executed, that is, the bandwidth parameter is updated or corrected; otherwise, the bandwidth parameter is not updated or corrected, and step 207 is executed.

[0129] Step 206: Update or correct the bandwidth parameter according to the first loss factor, that is, let bw represent the bandwidth parameter, bw=bw×first loss factor, and execute step 207.

[0130] Step 207: Determine whether the set communication algorithm in the first combination is the second preset algorithm, i.e., the Ring set communication algorithm, and determine whether the protocol in the first combination is the preset protocol, i.e., Simple; if it is determined that the set communication algorithm in the first combination is the second preset algorithm, and the protocol in the first combination is the preset protocol, then execute step 208, i.e., update or correct the delay parameter; otherwise, do not update or correct the delay parameter, and execute step 209.

[0131] Step 208: Update or correct the delay parameter according to the second loss factor, that is, use lat to represent the delay parameter, lat=lat×second loss factor, and execute step 209.

[0132] Here, the task is to adjust the performance parameters for the selected algorithms, specifically the Tree and Ring algorithms.

[0133] Here, during the static scheduling process, the scheduler first determines whether to select the first preset algorithm, namely the Tree algorithm, as the custom set communication algorithm. If the algorithm is the first preset algorithm, namely the Tree set communication algorithm, a bandwidth loss factor needs to be considered. This is because the Tree algorithm improves communication efficiency by decomposing the data transmission task into multiple sub-tasks and using a hierarchical parallel processing structure. However, actual network latency, bandwidth limitations, and data transmission overhead may lead to a gap between theoretical and actual performance. That is, considering that some performance loss may occur during actual communication due to network card latency or network receive queue congestion, the introduction of the first loss factor helps the scheduler consider these influencing factors, making the calculated actual communication time closer to reality. To accurately reflect the gap between theoretical and actual performance, the scheduler will correct the theoretical bandwidth according to the set first loss factor, namely the bandwidth loss factor, to ensure high communication efficiency in a variable environment.

[0134] Here, regarding the selection of the Ring collective communication algorithm, the scheduler determines whether to choose the second preset algorithm, i.e., the Ring algorithm, and further determines whether the current protocol is the preset protocol, i.e., Simple. If both conditions are met, the latency parameter is corrected by multiplying the latency parameter corresponding to the Ring algorithm by a fixed latency reduction factor. This is because the Ring algorithm transmits data sequentially, with each node receiving and forwarding data in turn. This method is particularly sensitive to latency. Considering that various factors such as poor network conditions can cause errors between the actual communication process and the ideal situation, setting a second reduction factor, i.e., the latency reduction factor, can more accurately reflect the actual communication latency and ensure that the optimization algorithm adapts to network conditions. Although the Simple protocol has a more straightforward design, it is still affected by network latency and bandwidth. Introducing a reduction factor helps to adjust performance. In contrast, the LL and LL128 protocols are designed to achieve low latency and high bandwidth utilization among multiple nodes. The LL and LL128 protocols employ more complex communication modes, such as parallel transmission and multipath strategies. These communication modes can adaptively balance the impact of latency and bandwidth when network conditions change. The built-in adaptive capabilities of the communication mode reduce the need for additional latency loss factors, thus enabling the LL and LL128 protocols to maintain high performance in practical applications.

[0135] Here, we set the first depreciation factor (i.e., bandwidth (bw) depreciation factor) and the second depreciation factor (i.e., latency (lat) depreciation factor).

[0136] Here, in order to select appropriate collective communication algorithms and protocols more flexibly and accurately, the scheduler can divide the data range, and the scheduler sets the bandwidth (bw) depreciation factor and the latency (lat) depreciation factor according to the divided data range.

[0137] For example, assuming the data intervals are [1B, 128KB], [128KB, 32MB], and [32MB, 4GB], the scheduler can set the second depreciation factor, i.e., the latency (lat) depreciation factor, for these three data intervals as 1.3, 1.5, and 1.7, respectively. The first depreciation factor, i.e., the bandwidth (bw) depreciation factor, for these three data intervals can be 0.3, 0.5, and 0.7, respectively. That is, different depreciation factors belong to different data size intervals, and the scheduler uses different depreciation factors when adjusting the bandwidth (bw) and latency (lat) in the future.

[0138] Here, a first data interval containing the custom data size interval is determined from the pre-divided data interval. According to the preset correspondence between the pre-divided data interval and the bandwidth loss factor, the bandwidth loss factor corresponding to the first data interval is determined, and the determined bandwidth loss factor is used as the first loss factor.

[0139] Here, a first data interval containing the custom data size interval is determined from the pre-divided data interval. Based on the preset correspondence between the pre-divided data interval and the delay loss factor, a delay loss factor corresponding to the first data interval is determined, and the determined delay loss factor is used as the second loss factor.

[0140] For example, assuming the custom data size range is [1B, 50KB], the first data range containing the custom data size range is determined from the pre-divided data range as [1B, 128KB]. The first data range [1B, 128KB] has a first loss factor of 0.3 and a second loss factor of 1.3.

[0141] Step 209: Calculate the first value by dividing the target data transmission size by the bandwidth parameter; multiply the delay parameter by the preset delay weight to obtain the second value; sum the first value and the second value to obtain the third value; use the third value as the set communication time corresponding to the first combination.

[0142] It should be noted that if the bandwidth parameter is updated or corrected, the target data transmission size is divided by the updated or corrected bandwidth parameter to obtain a first value; otherwise, the target data transmission size is divided by the bandwidth parameter obtained from the test to obtain a first value. Similarly, if the latency parameter is updated or corrected, the updated or corrected latency parameter is multiplied by a preset latency weight to obtain a second value; otherwise, the latency parameter obtained from the test is multiplied by a preset latency weight to obtain a second value.

[0143] Here, the task is to calculate the communication time for each of the first combinations.

[0144] Here, for each first combination, namely the combination of the collective communication algorithm and the collective communication protocol, the scheduler will calculate the total communication time under each first combination, namely the combination of the collective communication algorithm and the collective communication protocol, based on the bandwidth parameters and latency parameters obtained from the test.

[0145] Here, the scheduler can use the following formula (1) to calculate the set communication time corresponding to each first combination, as follows: (1) Wherein, time represents the collective communication time corresponding to the first combination, BufferSize represents the target data transmission size, bw is the bandwidth parameter obtained from the test for the first combination, i.e., the combination of collective communication algorithm and collective communication protocol, to perform collective communication operations under the target data transmission size and target system configuration (GPU architecture, heterogeneous, etc.), or the updated or corrected bandwidth parameter, lat is the latency parameter obtained from the test for the first combination, i.e., the combination of collective communication algorithm and collective communication protocol, to perform collective communication operations under the target data transmission size and target system configuration (GPU architecture, etc.), or the updated or corrected latency parameter, and latcount represents the preset latency weight, i.e., the weight of latency in the current task, which can be set according to the actual situation, and the default value can be 1.

[0146] Here, formula (1) above indicates that the communication time of the first combination is divided into two parts: data transmission time (determined by the target data transmission size and bandwidth) and additional latency overhead, i.e., data processing time. Using the above formula, the scheduler can quantify the actual communication time for each algorithm and protocol combination.

[0147] Step 210: Determine whether the set communication time (time) corresponding to the first combination is less than or equal to the optimal value (minTime). minTime represents the optimal value, i.e. the shortest time. If it is determined that the set communication time (time) corresponding to the first combination is less than or equal to the optimal value (minTime), then execute step 211; otherwise, execute step 212.

[0148] Here, for the first first combination, the optimal value (minTime) can refer to the set communication time corresponding to the preset set communication native algorithm. For each subsequent first combination, the optimal value (minTime) refers to the latest optimal value set in step 211.

[0149] Step 211: Set the set communication time (time) corresponding to the first combination to the latest optimal value, that is, minTime=time. Step 212: Continue traversing the next XML file in the directory, that is, repeat steps 203 to 210 until the last first combination.

[0150] Here, the task is to update the optimal time and the optimal combination of algorithm and protocol.

[0151] Here, until the last first combination, if any of the N first combinations contains at least one first combination whose set communication time is less than the first time, the first combination corresponding to the last determined optimal value (i.e., the latest optimal value) is taken as the optimal combination, where N is an integer greater than 1; otherwise, the preset set communication native algorithm and the preset set communication protocol are combined to form the optimal combination. The preset set communication protocol is bound to the preset set communication native algorithm.

[0152] Here, the set communication time corresponding to the preset set communication native algorithm can be pre-set.

[0153] Here, the following operations can also be performed to determine the co-communication time corresponding to the preset co-communication native algorithm: test the performance parameters of performing co-communication operations based on the preset co-communication native algorithm and using the bound co-communication native protocol under the target data transmission size and the target system configuration; the performance parameters include bandwidth and latency; quotient the target data transmission size with the bandwidth to obtain a sixth value; multiply the latency with a preset latency weight to obtain a seventh value; sum the sixth value and the seventh value to obtain the co-communication time corresponding to the preset co-communication native algorithm. Here, the preset native algorithm for set communication and the preset native protocol for set communication are combined to form an optimal combination, which can be used to perform set communication tasks by using the native algorithm for set communication and the native protocol for set communication bound to the native algorithm for set communication.

[0154] Here, the preset native set communication algorithm can refer to a default set communication algorithm preset by the system, such as the Ring algorithm.

[0155] Here, the preset native protocol for aggregate communication can refer to a default aggregate communication protocol that is pre-set by the system. The default aggregate communication protocol is bound to the default aggregate communication algorithm, such as the LL128 protocol.

[0156] Here, after iterating through each first combination—the combination of a set communication algorithm and a set communication protocol—the scheduler compares the calculated communication time (time) with the currently known optimal time (represented by minTime). If the time of the current first combination is less than the optimal time, the scheduler updates the optimal time (represented by minTime) and selects the corresponding set communication algorithm and protocol combination as the optimal combination. This ensures that the scheduler can find the optimal combination for a given communication task, thereby maximizing communication performance. The optimal time (minTime) is set by default to the communication time of the native scheduling algorithm. This ensures that when the communication time of the first combination of set communication algorithms and protocols contained in the XML configuration file in the `algorithms` directory is worse than the communication time corresponding to the native scheduling algorithm, the scheduler selects the native set communication algorithm as the optimal scheduling algorithm.

[0157] Here, after traversing all possible first combinations—that is, combinations of set communication algorithms and protocols—the scheduler returns the final optimal solution, i.e., the first combination with the shortest execution time. This first combination includes both the set communication algorithm and the set communication protocol. During the actual execution of the communication task, the set communication operation will be performed according to the optimal solution. By using the optimal solution, it can be ensured that the set communication operation can be completed in the optimal way under different system configurations and task requirements.

[0158] To implement the data processing method of this application embodiment, this application embodiment also provides a data processing device, which is set in the scheduler. Figure 3 This is a schematic diagram of the composition structure of the data processing device according to an embodiment of this application, as shown below. Figure 3 As shown, the device includes: The acquisition module 31 is used to acquire at least one custom configuration file; wherein each custom configuration file corresponds to a custom first combination, the first combination represents a combination of a set communication algorithm and a set communication protocol, the set communication protocol is a protocol bound to the set communication algorithm, and each set communication algorithm corresponds to the same custom data size range; Processing module 32 is configured to test, based on the at least one custom configuration file, under a target data transmission size and a target system configuration, communication performance parameters for performing a collection communication operation based on each of the first combinations; the target data transmission size is selected from the data size range; and determine an optimal combination based on the target data transmission size and the communication performance parameters corresponding to each of the first combinations; wherein the optimal combination is used to perform the collection communication task.

[0159] In some embodiments, the acquisition module 31 is specifically used for: acquiring a directory; determining whether the directory is empty; and if the directory is determined to be not empty, traversing the directory to acquire the at least one custom configuration file in the directory.

[0160] In some embodiments, the processing module 32 is further configured to: schedule a preset native algorithm for set communication when the directory is determined to be empty; and execute a set communication task using the preset native algorithm for set communication and a preset native protocol for set communication.

[0161] In some embodiments, the processing module 32 is specifically configured to: determine the set communication time corresponding to each of the first combinations based on the target data transmission size and the communication performance parameters corresponding to each of the first combinations; and determine the optimal combination based on the set communication time corresponding to each of the first combinations.

[0162] In some embodiments, the communication performance parameters include bandwidth parameters and latency parameters; the processing module 32 is specifically used for: For each of the first combinations, perform the following operations: The first value is obtained by quoting the target data transmission size with the bandwidth parameter, which is included in the communication performance parameters corresponding to the first combination. The second value is obtained by multiplying the delay parameter, which is included in the communication performance parameters corresponding to the first combination, with the preset delay weight. Summing the first value and the second value yields the third value; The third value is used as the set communication time corresponding to the first combination.

[0163] In some embodiments, the processing module 32 is specifically used to: determine whether the collective communication algorithm included in the first combination is a first preset algorithm; if it is determined that the collective communication algorithm included in the first combination is the first preset algorithm, then multiply the bandwidth parameter included in the communication performance parameters corresponding to the first combination with the first loss factor to obtain a fourth value; and divide the target data transmission size with the fourth value to obtain the first value.

[0164] In some embodiments, the processing module 32 is specifically configured to: determine whether the collective communication algorithm included in the first combination is a second preset algorithm, and determine whether the collective communication protocol included in the first combination is a preset protocol; if it is determined that the collective communication algorithm included in the first combination is the second preset algorithm, and the collective communication protocol included in the first combination is the preset protocol, then the delay parameter included in the communication performance parameters corresponding to the first combination is multiplied by the second loss factor to obtain a fifth value; the fifth value is multiplied by the preset delay weight to obtain the second value.

[0165] In some embodiments, the number of the first combinations is N, where N is an integer greater than 1. The processing module 32 is specifically used to: determine the minimum value from the set communication times corresponding to the N first combinations; if the minimum value is less than or equal to the set communication time corresponding to the preset set communication native algorithm, then the first combination corresponding to the minimum value is taken as the optimal combination.

[0166] In practical applications, the acquisition module 31 can be implemented by the communication interface in the data processing device; the processing module 32 can be implemented by the processor in the data processing device.

[0167] It should be noted that the data processing device provided in the above embodiments is only illustrated by the division of the above program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the data processing device and the scheduling method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0168] This application also provides a scheduler, such as... Figure 4 As shown, it includes: Communication interface 41 enables information exchange with other devices; The processor 42, connected to the communication interface 41, is used to execute the methods provided by one or more technical solutions on the scheduler side when running a computer program. The computer program is stored in the memory 43.

[0169] It should be noted that the specific processing procedures of the processor 42 and the communication interface 41 are detailed in the method embodiment and will not be repeated here.

[0170] Of course, in practical applications, the various components in scheduler 40 are coupled together via bus system 44. It can be understood that bus system 44 is used to implement communication between these components. In addition to a data bus, bus system 44 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 4 The general labeled all buses as Bus System 44.

[0171] In this embodiment, memory 43 is used to store various types of data to support the operation of scheduler 40. Examples of such data include any computer program used to operate on scheduler 40.

[0172] The methods disclosed in the embodiments of this application can be applied to the processor 42, or implemented by the processor 42. The processor 42 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 42 or by instructions in the form of software. The processor 42 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 42 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the memory 43. The processor 42 reads the information in the memory 43 and combines its hardware to complete the steps of the aforementioned method.

[0173] In an exemplary embodiment, the scheduler 40 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0174] It is understood that the memory (memory 43) in the embodiments of this application can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0175] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory that stores a computer program, which can be executed by the processor 42 of the scheduler 40 to complete the steps described in the aforementioned scheduler-side method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0176] For example, embodiments of this application also provide a computer program product, including a computer program that can be executed by the processor 42 of the scheduler 40 to perform the steps described in any of the foregoing methods.

[0177] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0178] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0179] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A data processing method, characterized by, The method comprises: obtaining at least one custom configuration file; wherein each of the custom configuration files corresponds to a custom first combination, the first combination representing a combination of a collective communication algorithm and a collective communication protocol, the collective communication protocol being a protocol bound to the collective communication algorithm, and each of the collective communication algorithms corresponding to a same custom data size interval; testing, according to the at least one custom configuration file, a communication performance parameter of performing a collective communication operation based on each of the first combinations under a target data transmission size and a target system configuration; the target data transmission size being selected from the data size interval; determining an optimal combination according to the target data transmission size and the communication performance parameter corresponding to each of the first combinations; wherein the optimal combination is used to perform a collective communication task.

2. The method of claim 1, wherein, The obtaining at least one custom configuration file comprises: obtaining a directory; determining whether the directory is empty; in a case where it is determined that the directory is not empty, traversing the directory to obtain the at least one custom configuration file in the directory.

3. The method of claim 2, wherein, The method further comprises: in a case where it is determined that the directory is empty, scheduling a preset collective communication native algorithm; performing a collective communication task by using the preset collective communication native protocol through the preset collective communication native algorithm.

4. The method according to any one of claims 1 to 3, characterized in that, The determining an optimal combination according to the target data transmission size and the communication performance parameter corresponding to each of the first combinations comprises: determining a collective communication time corresponding to each of the first combinations according to the target data transmission size and the communication performance parameter corresponding to each of the first combinations; determining an optimal combination based on the collective communication time corresponding to each of the first combinations.

5. The method of claim 4, wherein, The communication performance parameter comprises a bandwidth parameter and a delay parameter; and the determining a collective communication time corresponding to each of the first combinations according to the target data transmission size and the communication performance parameter corresponding to each of the first combinations comprises: for each of the first combinations, performing the following operations: dividing the target data transmission size by a bandwidth parameter included in the communication performance parameter corresponding to the first combination to obtain a first value; multiplying a delay parameter included in the communication performance parameter corresponding to the first combination by a preset delay weight to obtain a second value; summing the first value and the second value to obtain a third value; taking the third value as the collective communication time corresponding to the first combination.

6. The method of claim 5, wherein, The dividing the target data transmission size by a bandwidth parameter included in the communication performance parameter corresponding to the first combination to obtain a first value comprises: determining whether a collective communication algorithm included in the first combination is a first preset algorithm; if it is determined that the collective communication algorithm included in the first combination is the first preset algorithm, multiplying a bandwidth parameter included in the communication performance parameter corresponding to the first combination by a first loss factor to obtain a fourth value; dividing the target data transmission size by the fourth value to obtain the first value.

7. The method of claim 5, wherein, The delay parameter included in the communication performance parameter corresponding to the first combination is multiplied by a preset delay weight to obtain a second value, including: determining whether the set communication algorithm included in the first combination is a second preset algorithm and whether the set communication protocol included in the first combination is a preset protocol; if it is determined that the set communication algorithm included in the first combination is the second preset algorithm and the set communication protocol included in the first combination is the preset protocol, the delay parameter included in the communication performance parameter corresponding to the first combination is multiplied by a second loss factor to obtain a fifth value; the fifth value is multiplied by a preset delay weight to obtain the second value.

8. The method of claim 4, wherein, The number of the first combinations is N, N is an integer greater than 1, and the optimal combination is determined based on the set communication time corresponding to each of the first combinations, including: determining a minimum value from the set communication times corresponding to the N first combinations; if the minimum value is less than or equal to the set communication time corresponding to the preset set communication native algorithm, the first combination corresponding to the minimum value is taken as the optimal combination.

9. A data processing apparatus, characterized by, including: an acquisition module configured to acquire at least one custom configuration file; each custom configuration file corresponds to a custom first combination, the first combination represents a combination of a set communication algorithm and a set communication protocol, the set communication protocol is a protocol bound to the set communication algorithm, and each set communication algorithm corresponds to the same custom data size interval; a processing module configured to test, according to the at least one custom configuration file, a communication performance parameter of performing a set communication operation based on each first combination under a target data transmission size and a target system configuration; the target data transmission size is selected from the data size interval; and determine an optimal combination according to the target data transmission size and the communication performance parameter corresponding to each first combination; wherein the optimal combination is used to perform a set communication task.

10. A scheduler characterized by, including a processor and a memory for storing a computer program capable of running on the processor, wherein the processor is configured to run the computer program to perform the steps of the method of any one of claims 1 to 8.

11. A computer readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 8.

12. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 8.

Images