Communication and computation overlap optimization method, system, device and storage medium

Abstract

This invention discloses a method, system, device, and storage medium for optimizing communication and computation overlap. These are corresponding solutions that can improve the performance of computation-communication overlap by optimizing communication parameters without introducing excessive overhead. Furthermore, they can quickly find the optimal configuration within a large search space through effective optimization methods. Based on these solutions, communication parameters can be dynamically optimized to balance resource usage and improve the performance of computation-communication overlap, while ensuring the correct execution of computational tasks and NCCL library-based communication tasks. This approach is suitable for complex computation-communication overlap scenarios and can achieve rapid convergence during distributed training of deep neural network models. Evaluations on multiple clusters and models show that this invention can achieve rapid convergence in deep neural network model training while ensuring higher performance gains compared to NCCL.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of distributed computing and deep learning, and particularly relates to a communication and computation overlap optimization method, system, device and storage medium. BACKGROUND

[0002] With the continuous expansion of the scale of deep neural networks (DNN), distributed training has become an inevitable choice, and has given rise to various parallel strategies such as tensor parallelism, fully sharded data parallel (FSDP) and expert parallelism. These strategies, while solving the memory and computing bottlenecks, also introduce significant communication overhead. In order to maximize training efficiency, communication and computation overlap has become a core optimization technology for implementing these parallel strategies.

[0003] The basic idea of communication and computation overlap is to use the time of the device to perform computing tasks to complete the necessary communication operations in parallel, thereby hiding the communication overhead in the computing process, so that the advantages of the parallel strategy can be fully utilized. There are more than 1500 types of computing operators in overlap, most of which are implemented using closed-source libraries such as cuBLAS (GPU acceleration library for efficient execution of linear algebra operations) and cuDNN (GPU acceleration library for deep neural networks) developed by NVIDIA. The communication operators in overlap are mostly provided by the NVIDIA Collective Communication Library (NCCL). The NCCL library is an open-source, efficient communication operator library developed by NVIDIA, which provides collective communication primitives including AllReduce, AllGather, ReduceScatter, All2All, etc. These implementations usually require communication to perform data copying and reduction computation on the GPU (Graphics Processing Unit).

[0004] When communication and computation operations are concurrently executed on a GPU, they compete for limited hardware resources (such as streaming multiprocessors, memory bandwidth, etc.), interfere with each other, and cause the effect of overlap to be poor. When a collective communication primitive is called, parameters such as NC (number of Channel), NT (number of thread), and C (chunksize) determine the GPU resources used by the collective communication task. Experiments show that changes in these parameters change the use of GPU resources by communication, and thus affect the interference of communication tasks on computation tasks when they are concurrent, so optimization of these parameters is necessary; in addition, in distributed training, the overlap mode of communication and computation is complex and diverse, so a general automatic optimization method that can cope with different situations is necessary.

[0005] In view of this, the present application is proposed. SUMMARY

[0006] The purpose of the present application is to provide a communication and computation overlap optimization method, system, device and storage medium, which can balance the use of communication and computation resources, improve the performance of computation and communication overlap, be suitable for complex computation and communication overlap scenarios, and quickly converge in the process of distributed training of deep neural network models.

[0007] The purpose of the present application is achieved by the following technical solutions:

[0008] A communication and computation overlap optimization method, comprising:

[0009] obtaining a communication and computation overlap group in all work nodes in distributed training;

[0010] optimizing each communication and computation overlap group, comprising: in each round, selecting a communication task to be optimized from the communication and computation overlap group, executing the communication and computation overlap group under the current parameter configuration, and obtaining a performance result; determining whether the performance of the communication task to be optimized and the overall performance of the communication and computation overlap group are optimal according to the performance result, if not, optimizing the parameters related to the allocation of computation resources in the communication task to be optimized, generating a new parameter configuration, and entering the next round, until the performance of all communication tasks in the communication and computation overlap group or the overall performance of the communication and computation overlap group reaches optimal.

[0011] A communication and computation overlap optimization system for implementing the foregoing method, comprising:

[0012] an information acquisition unit for acquiring a communication and computation overlap group in all work nodes in distributed training;

[0013] An optimization unit, used to optimize each communication and computation overlap group, includes an execution module, a coordination module, and a tuning module. In each round, the tuning module selects a communication task to be optimized from the communication and computation overlap group. The execution module executes the communication and computation overlap group under the current parameter configuration, obtains performance results, and transmits them to the tuning module through the coordination module. The tuning module determines whether the performance of the communication task to be optimized and the overall performance of the communication and computation overlap group have not reached the optimal level based on the performance results. If neither has reached the optimal level, the module optimizes the parameters related to computational resource allocation in the communication task to be optimized, generates a new parameter configuration, and proceeds to the next round until the performance of all communication tasks in the communication and computation overlap group or the overall performance of the communication and computation overlap group reaches the optimal level.

[0014] A processing device includes: one or more processors; and a memory for storing one or more programs;

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0016] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0017] As can be seen from the technical solution provided by the present invention, the performance of computational communication overlap can be improved by optimizing the communication parameters, while ensuring that the computational task and the NCCL library's collective communication task can be executed correctly and without introducing too much overhead. Evaluations on multiple clusters and models show that the present invention can achieve fast convergence in deep neural network model training, while ensuring higher performance gains compared to NCCL. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart of a communication and computation overlap optimization method provided in an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of the optimization framework for the communication and computing overlap group provided in an embodiment of the present invention.

[0021] Figure 3 A flowchart for real-time optimization and updating of communication parameters provided in an embodiment of the present invention.

[0022] Figure 4 A flowchart of a priority-based optimization method provided in an embodiment of the present invention.

[0023] Figure 5 This is a schematic diagram of a communication and computing overlap optimization system provided in an embodiment of the present invention.

[0024] Figure 6 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0026] First, the following explanations are provided for the terms that may be used in this article:

[0027] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0028] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0029] The following provides a detailed description of a communication and computing overlap optimization method, system, device, and storage medium provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they are performed according to conventional conditions in the art or conditions recommended by the manufacturer. Where the manufacturers of reagents or instruments used in the embodiments of this invention are not specified, they are all conventional products that can be purchased commercially.

[0030] Example 1

[0031] This invention provides a method for optimizing communication and computation overlap, such as... Figure 1 As shown, it mainly includes the following steps:

[0032] Step 1: Obtain the communication and computation overlap groups among all worker nodes in the distributed training.

[0033] In this embodiment of the invention, the working node mainly refers to the GPU node, and each GPU node contains one or more communication and computing overlap groups.

[0034] Step 2: Optimize each communication and computation overlap group separately.

[0035] In this embodiment of the invention, the optimization process includes multiple rounds. In each round, a communication task to be optimized is selected from the communication and computing overlap group, and the communication and computing overlap group is executed under the current parameter configuration to obtain performance results. Based on the performance results, it is determined whether the performance of the communication task to be optimized and the overall performance of the communication and computing overlap group have not reached the optimal. If neither has reached the optimal, the parameters related to computing resource allocation in the communication task to be optimized are optimized to generate a new parameter configuration and enter the next round until the performance of all communication tasks in the communication and computing overlap group or the overall performance of the communication and computing overlap group reaches the optimal.

[0036] In this embodiment of the invention, a history table is maintained, which records the optimization completion status, priority, and parameter configuration of all communication tasks in all communication and computing overlap groups. In each round, the relevant information in the history table is updated according to the optimization status.

[0037] In this embodiment of the invention, the communication and computation overlap group refers to a fixed and reusable parallel execution mode formed by computation tasks and communication tasks in a variety of parallel strategies, including a group of computation operators and communication operators that are executed concurrently in time and share GPU resources.

[0038] In this embodiment of the invention, selecting a communication task to be optimized from the communication and computing overlap group includes: selecting a communication task from the communication and computing overlap group whose optimization completion status is not completed and has the highest priority.

[0039] In this embodiment of the invention, executing the communication and computation overlap group under the current parameter configuration includes: determining whether the communication and computation overlap group has been initialized; if so, executing the communication and computation overlap group under the current parameter configuration; if not, executing initialization, using the initialized parameter configuration as the current parameter configuration, and executing the communication and computation overlap group; wherein, executing initialization means setting the parameters related to computation resource allocation in each communication task to a predetermined minimum value.

[0040] In this embodiment of the invention, the step of determining whether the performance of the communication task to be optimized and the overall performance of the communication and computing overlap group have not reached the optimal level based on the performance results, and if neither has reached the optimal level, then optimizing the parameters related to computing resource allocation in the communication task to be optimized and generating a new parameter configuration includes: determining whether the performance of the communication task to be optimized has reached the optimal level based on the performance results; if not, then updating the priority of the communication task to be optimized by combining the performance results under the current parameter configuration and the performance results under the previous configuration, wherein if this is the first time the communication and computing overlap group is executed, then the previous configuration is the preset default configuration; then, determining whether the overall performance of the communication and computing overlap group has reached the optimal level, and if not, then dynamically optimizing the parameters related to computing resource allocation in the communication task to be optimized based on the performance improvement percentage.

[0041] In this embodiment of the invention, updating the priority of the communication task to be optimized by combining the performance results under the current parameter configuration and the performance results under the previous configuration includes: determining the cost-effectiveness of the communication task to be optimized based on the performance results under the current parameter configuration and the performance results under the previous configuration, and updating the priority of the communication task to be optimized based on the cost-effectiveness of the communication task to be optimized; wherein, the cost-effectiveness of the communication task to be optimized refers to the ratio of the increase in the execution time of the computing task brought about by increasing the computing resources of the communication task to the reduction in the execution time of the communication task. The execution time of the computing task and the execution time of the communication task are both information in the performance results. The smaller the ratio, the greater the performance improvement brought by increasing the communication resources of the communication task to be optimized to the overlapping group of communication and computing, and the smaller the loss to the computing task.

[0042] In this embodiment of the invention, if the performance of the communication task to be optimized has reached its optimal level based on the performance results, the parameter configuration of the communication task to be optimized is not adjusted, its optimization completion status is marked as completed, and the process proceeds to the next round.

[0043] The above-described solution provided by the embodiments of the present invention can improve the performance of computational communication overlap by optimizing communication parameters without introducing excessive overhead; and can quickly find a configuration with better performance in a huge search space through effective optimization methods; it is suitable for complex computational communication overlap scenarios and can converge quickly during the distributed training of deep neural network models.

[0044] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0045] I. Overall Overview of the Plan

[0046] The core of the solution provided in this invention lies in optimizing each overlapping communication and computation group. This process can be accomplished through a configured execution module, tuning module, and coordination module. Overall, the tuning module is responsible for running the navigator and configuration generator (see later description) to generate new configurations for communication parameters; the execution module is deployed on each GPU node to perform computation and communication tasks and records performance results, which are then fed back to the tuning module by the coordination module; the coordination module synchronizes the updated configuration to the execution modules of all GPU nodes via a broadcast mechanism to ensure consistency. During operation, a designated GPU node runs the tuning module, leading the tuning process; simultaneously, a coordination module is configured on the corresponding GPU node.

[0047] In this embodiment of the invention, the performance result mainly refers to the execution time.

[0048] Figure 2 This is the optimization framework for the communication and computation overlap group, demonstrating the interaction process of the three modules. The tuning module, as the core decision-making unit, is responsible for deciding whether to tune communication parameters based on the returned performance results of the communication and computation overlap group. When tuning is decided, the tuning module initiates the tuning process, selects communication tasks within the communication and computation overlap group, and updates their parameter configurations. The execution module runs on each GPU node participating in training, responsible for executing specific computation-communication overlap tasks, monitoring the performance of the entire communication-computation overlap group, returning the results to the coordination module, and updating and executing the parameter configurations upon receiving new ones. The coordination module is the intermediary component connecting the tuning module and the execution module. Upon receiving notification from the tuning module that a new configuration has been generated, the coordination module broadcasts the new parameter configuration to the execution modules on all GPU nodes in training. After collecting the performance results returned by the execution modules on each GPU node, the coordination module summarizes and reports the results to the tuning module.

[0049] Figure 3 This demonstrates the specific structure of the tuning module and how it updates communication configuration parameters. First, the tuning module maintains a history table to record the optimization status of overlapping communication and computation groups. Figure 3 Provides a single communication and computing overlap group ( Figure 3This example (abbreviated as "overlapping group") contains three communication tasks, comm1 to comm3, where communication tasks 1 and 2 are not yet optimized (value 0), and communication task 3 is optimized (value 1). Each communication task has a priority, with smaller values ​​indicating higher priority. Each communication also maintains a parameter configuration list, recording the parameter configurations generated during optimization. Furthermore, the tuning module includes a navigator and a configuration generator. The navigator has two functions: one is to select communication tasks that are not yet optimized from the communication and computation overlap group based on information in the history table and hand them over to the configuration generator; the other is to calculate the new priority and optimization completion status of the communication tasks based on the received performance results, maintaining the history table for subsequent optimization. The configuration generator's role is to generate the next parameter configuration for this communication task after receiving the performance results from the navigator and update the configuration list of that communication task in the history table.

[0050] Figure 3 Steps a to f in the diagram detail the process by which the tuning module updates the parameters for a single communication task. First, the navigator reads the information of the communication and computation overlap group in the history table, selects the communication task with the highest priority whose optimization completion status is not yet complete from the communication and computation overlap group (step a), and passes the configuration information of the entire communication and computation overlap group to the coordination module (step b). Figure 3 There are three communication tasks in the overlapping group 1. Communication tasks 1 and 2 are not yet optimized, and communication task 1 has a lower priority than communication task 2. Therefore, communication task 2 is selected for optimization. In this example, it is assumed that communication task 2 has completed parameter initialization. After the execution module completes the execution under the current configuration and obtains the performance results of the communication and computation overlapping group, it is passed to the navigator through the coordination module (step c). The navigator calculates the new priority based on the performance results, and at the same time determines whether communication task 2 has been optimized and updates the history table (step d). In this example, it is assumed that communication task 2 is still not completed, so its completion status is not changed to 1, only the priority is updated. In addition, the performance results and the corresponding communication task 2 are also transmitted to the configuration generator (step e). The configuration generator calculates the learning rate of communication task 2 based on the performance results (determined based on the percentage improvement in performance), adjusts the NC, NT, and C values ​​in the configuration of communication task 2, generates a new configuration s2, and updates the history table (step f). The above optimization process will continue until all communication tasks in the communication and computation overlapping group are marked as optimized and completed, so that the execution effect of the entire communication and computation overlapping group reaches the best.

[0051] It should be noted that in the above example, comm2, group1, H2, and s2 are used to identify the collective communication task, the communication and computation overlap group, the priority, and the communication task parameter configuration information, respectively. In actual applications, the composition of the communication and computation overlap group, the content and priority of the collective communication task, the specific configuration parameters, and their identification can all be adjusted according to the actual situation, and this invention does not impose any restrictions.

[0052] The above-mentioned solution provided by the present invention can adaptively handle dynamic workloads, adjust the communication parameter configuration through real-time execution results, and thus adjust the overall performance of the communication and computation overlap group. It is particularly suitable for distributed training scenarios with complex communication and computation overlap strategies, and can give full play to the acceleration effect brought by the communication and computation overlap strategy.

[0053] II. Detailed introduction of the plan.

[0054] The solution provided in this invention can optimize communication and computation overlap groups across all GPU nodes in distributed training. A communication and computation overlap group refers to a fixed, reusable parallel execution pattern formed by computation and communication tasks in various parallel strategies. A communication and computation overlap group can contain multiple communication tasks. For example, in the reverse execution of fully sliced ​​data parallelism, the reduplication operation of the current layer's gradient and the allgather operation of the next layer's weights overlap with some computations of the current layer, resulting in a complex situation of multiple communication and computation overlaps. In the case of multiple communications, experiments show that optimizing one communication task not only affects its own execution but also indirectly affects other communications by changing the timeline and resource contention. For instance, adjusting a previous communication task may change the overlap between subsequent communication tasks and computations, altering the resource contention relationship and thus changing performance. Therefore, the combination of communication task parameter configurations within a communication and computation overlap group determines the overall performance of the group. The parameter combinations of multiple communication tasks constitute a vast search space.

[0055] In this invention, each parameter configuration of the communication task is a six-tuple.<algorithm, protocol,transport, NC, NT, C> Here, algorithm, protocol, and transport represent the algorithm, protocol, and transport respectively. Together, they determine the topology of the communication algorithm's execution logic, the method of data transmission, and the software calls required to complete the primitive semantics. Figure 3In the example shown, Ring represents a ring topology, Tree represents a tree topology, LL represents a low-latency protocol, Simple represents a standard protocol, SHM represents shared memory (i.e., data is transferred between multiple GPUs on the same machine using CPU memory as an intermediary), and P2P represents peer-to-peer communication. NC, NT, and C represent the number of communication blocks, the number of threads, and the size of the communication blocks, respectively. These are parameters related to the allocation of computing resources and together determine how to parallelize the use of network bandwidth and GPU computing power. The search space for these parameters is relatively large; for example, NC can have up to 128 possible values, NT up to 20, and C up to 8192. Therefore, this invention updates these three parameters according to the learning rate, which is determined by the percentage performance improvement.

[0056] In this invention, the main parameters adjusted are those related to computing resource allocation: NC, NT, and C. This is because these parameters directly control the intensity of resource competition between communication and computing tasks. The parameter NC affects the competition between communication and computing tasks for streaming multiprocessors (SMs). The larger the NC value, the more SM resources communication uses, and the more pronounced the competition between communication and computing. Simultaneously, NC and C jointly affect the competition between communication and computing tasks for global resources (global bandwidth, cache, etc.); the larger the value, the more pronounced the competition. When communication uses the fewest resources, the competition between communication and computing tasks is minimal, computing task performance is optimal, and execution time is shortest, but communication task execution time is longest. If, at this point, the overall execution time of communication tasks is still less than the overall execution time of computing tasks, then the performance of the communication-computing overlap group is equivalent to the computing task performance, which should be optimal. If the resources for communication tasks are increased, the execution time of communication tasks will shorten, but the corresponding competition with computing tasks will intensify, leading to a longer execution time for computing tasks. When the overall performance of the communication-computing overlap group is optimal, the execution time of communication tasks should be comparable to that of computing tasks. This invention heuristically starts from the case where the execution time of the computation task is shortest in the overlapping group of communication and computation. Each optimization step is based on priority to minimize the increase in the execution time of the computation task in order to gradually achieve a state of communication and computation balance.

[0057] Based on the above description, in the solution provided by this embodiment of the invention, a priority is calculated for each communication task in the communication and computing overlap group. Each optimization step selects a communication task for optimization based on its priority, where the priority reflects the impact of selecting that communication for optimization on the overlap group. During optimization, the performance of the overlap group is adjusted by gradually increasing communication resources, starting with the minimum resources required. The overall process is as follows: Figure 4 As shown.

[0058] Step S0 (i.e., the start time): Determine whether all communication tasks in the communication and computation overlap group have been optimized. If so, proceed to step S7; otherwise, proceed to step S1.

[0059] Step S1: Select the highest priority communication task (hereinafter referred to as the current communication task) from all communication tasks that have not been optimized in the communication and computation overlap group. If it has not been initialized, proceed to step S2 to initialize it; otherwise, proceed to step S3.

[0060] Step S2: Initialize the values ​​of the communication task parameters NC, NT, and C to their minimum values, and then proceed to step S3.

[0061] Step S3: Execute the communication and computation overlap group under the current configuration, obtain the performance results, and determine whether the performance of the current communication task has reached its optimal level. If so, proceed to step S6; otherwise, proceed to step S4. Specifically, the current execution time of the communication task can be compared with the previous execution time. If the current execution time no longer decreases or even increases, it indicates that the optimal level has been reached, and the optimal parameter configuration is the parameter configuration with the shortest execution time. Otherwise, it indicates that the performance has not reached its optimal level.

[0062] Step S4: Calculate the new priority based on the performance results obtained in Step S3 under the current configuration and the performance results under the previous configuration. Typically, the communication and computation overlap group is pre-executed under the default configuration to obtain the corresponding performance results. Therefore, if this is the first execution, the performance result under the previous configuration is the same as the performance result under the default configuration.

[0063] Priority is determined by the cost-effectiveness of optimizing the communication task. Cost-effectiveness here refers to the ratio of the increase in execution time of the computation task resulting from increasing the computational resources for that communication task to the reduction in execution time of the communication task. The smaller the value, the greater the improvement in the overlapping group and the smaller the loss to the computational task. Therefore, optimizing high-priority communications helps to speed up the optimization process while ensuring the execution performance of the overlapping group. Then, it is determined whether the performance of the communication and computational overlapping group has reached its optimal state. If not, proceed to step S5; if yes, proceed to step S6. Whether the performance of the communication and computational overlapping group has reached its optimal state can be determined based on the relative relationship between the execution times of the communication and computational tasks in the performance results. That is, whether the execution time of all communication tasks is less than or equal to the execution time of the computational tasks. If yes, it is optimal; otherwise, the performance has not reached its optimal state.

[0064] For example, for the current communication task, if the performance result under the previous configuration is: communication 6 seconds, calculation 3 seconds; and the performance result under the current configuration is: communication 5 seconds, calculation 4 seconds; then the priority of the current communication task is updated as: priority = cost-effectiveness = (4 – 3) / (6 – 5) = 1.

[0065] Step S5: Since the current communication task performance has not yet reached its optimal level, and the overall performance of the communication and computation overlap group has also not reached its optimal level, and the current communication task has the highest priority, it is necessary to continue adjusting the resource parameters of the current communication task to further improve the performance of the overlap group. At this time, the values ​​of NC, NT, and C are dynamically increased according to the performance improvement percentage to generate a new communication parameter configuration. After the adjustment is completed, the process jumps to the start of the next round of judgment, i.e., proceeding to step S0.

[0066] Step S6: At this point, there is no need to adjust the current communication parameters, so mark the completion status as completed and jump to the next round of judgment, i.e., proceed to step S0.

[0067] Step S7: At this point, all communication tasks have been optimized. All tasks will be executed using their respective optimal parameter configurations to achieve overall performance optimization of the communication and computation overlap group. Execution ends here.

[0068] The priority-based optimization method provided in this invention comprehensively considers the resource competition between communication and computation tasks. By introducing cost-effectiveness as a priority, it guides the optimization direction and reduces search complexity. It can handle complex overlapping computation and communication scenarios and converges quickly, providing a significant performance improvement.

[0069] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0070] Example 2

[0071] This invention also provides a communication and computing overlap optimization system, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 5 As shown, the system mainly includes:

[0072] The information acquisition unit acquires the communication and computation overlap groups among all worker nodes in the distributed training.

[0073] An optimization unit, used to optimize each communication and computation overlap group, includes an execution module, a coordination module, and a tuning module. In each round, the tuning module selects a communication task to be optimized from the communication and computation overlap group. The execution module executes the communication and computation overlap group under the current parameter configuration, obtains performance results, and transmits them to the tuning module through the coordination module. The tuning module determines whether the performance of the communication task to be optimized and the overall performance of the communication and computation overlap group have not reached the optimal level based on the performance results. If neither has reached the optimal level, the module optimizes the parameters related to computational resource allocation in the communication task to be optimized, generates a new parameter configuration, and proceeds to the next round until the performance of all communication tasks in the communication and computation overlap group or the overall performance of the communication and computation overlap group reaches the optimal level.

[0074] Since the main technical details of this system have been described in detail in previous embodiments, they will not be repeated here.

[0075] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0076] Example 3

[0077] The present invention also provides a processing device, such as Figure 6 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0078] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0079] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0080] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0081] The output device can be a display terminal;

[0082] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0083] Example 4

[0084] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0085] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0086] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A method for optimizing communication and computation overlap, characterized in that, include: Obtain the communication and computation overlap groups among all worker nodes in distributed training; Each communication and computation overlap group is optimized separately, including: in each round, a communication task to be optimized is selected from the communication and computation overlap group, the communication and computation overlap group is executed under the current parameter configuration, and the performance results are obtained; based on the performance results, it is determined whether the performance of the communication task to be optimized and the overall performance of the communication and computation overlap group have not reached the optimal level. If neither has reached the optimal level, the parameters related to the allocation of computing resources in the communication task to be optimized are optimized, a new parameter configuration is generated, and the next round is entered until the performance of all communication tasks in the communication and computation overlap group or the overall performance of the communication and computation overlap group reaches the optimal level. The step of selecting a communication task to be optimized from the communication and computing overlap group includes: selecting a communication task from the communication and computing overlap group whose optimization completion status is not completed and has the highest priority. The process of determining whether the performance of the communication task to be optimized and the overall performance of the communication and computing overlap group have not reached the optimal level based on the performance results, and if neither has reached the optimal level, then optimizing the parameters related to computing resource allocation in the communication task to be optimized and generating a new parameter configuration includes: determining whether the performance of the communication task to be optimized has reached the optimal level based on the performance results; if the execution time of the communication task to be optimized no longer shortens or lengthens compared to the previous execution time, then the performance has reached the optimal level; otherwise, the performance has not reached the optimal level, and updating the priority of the communication task to be optimized by combining the performance results under the current parameter configuration and the performance results under the previous configuration; wherein, if this is the first execution of the communication and computing overlap group, then the previous configuration is the preset default configuration; then, determining whether the overall performance of the communication and computing overlap group has reached the optimal level, that is, determining whether the execution time of all communication tasks in the communication and computing overlap group is less than or equal to the execution time of the computing task; if so, then the performance has reached the optimal level; if not, then the performance has not reached the optimal level, and dynamically optimizing the parameters related to computing resource allocation in the communication task to be optimized based on the percentage improvement in performance; wherein, the execution time of the computing task and the execution time of the communication task are both information in the performance results; The step of updating the priority of the communication task to be optimized by combining the performance results under the current parameter configuration with the performance results under the previous configuration includes: determining the cost-effectiveness of the communication task to be optimized based on the performance results under the current parameter configuration and the performance results under the previous configuration, and using this as the updated priority of the communication task to be optimized; wherein, the cost-effectiveness of the communication task to be optimized refers to the ratio of the increase in the execution time of the computing task brought about by increasing the computing resources of the communication task to the reduction in the execution time of the communication task. The execution time of the computing task and the execution time of the communication task are both information in the performance results. The smaller the ratio, the greater the performance improvement brought by increasing the communication resources of the communication task to be optimized to the overlapping group of communication and computing, and the smaller the loss to the computing task.

2. The communication and computation overlap optimization method according to claim 1, characterized in that, The method also includes maintaining a history table that records the optimization completion status, priority, and parameter configuration of all communication tasks in all communication and computation overlap groups, and updating the relevant information in the history table in each round based on the optimization status.

3. The communication and computation overlap optimization method according to claim 1, characterized in that, The execution of the communication and computation overlap group under the current parameter configuration includes: Determine whether the communication and computation overlap group has been initialized. If so, execute the communication and computation overlap group under the current parameter configuration; otherwise, execute initialization, use the initialized parameter configuration as the current parameter configuration, and execute the communication and computation overlap group. Here, executing initialization means setting the parameters related to computation resource allocation in each communication task to the predetermined minimum value.

4. The communication and computation overlap optimization method according to claim 1 or 2, characterized in that, The method further includes: if the performance of the communication task to be optimized has reached its optimal level based on the performance results, then the parameter configuration of the communication task to be optimized is not adjusted, its optimization completion status is marked as completed, and the process proceeds to the next round.

5. A communication and computing overlap optimization system, characterized in that, To implement the method according to any one of claims 1 to 4, comprising: The information acquisition unit acquires the communication and computation overlap groups among all worker nodes in the distributed training. An optimization unit, used to optimize each communication and computation overlap group, includes an execution module, a coordination module, and a tuning module. In each round, the tuning module selects a communication task to be optimized from the communication and computation overlap group. The execution module executes the communication and computation overlap group under the current parameter configuration, obtains performance results, and transmits them to the tuning module through the coordination module. The tuning module determines whether the performance of the communication task to be optimized and the overall performance of the communication and computation overlap group have not reached the optimal level based on the performance results. If neither has reached the optimal level, the module optimizes the parameters related to computational resource allocation in the communication task to be optimized, generates a new parameter configuration, and proceeds to the next round until the performance of all communication tasks in the communication and computation overlap group or the overall performance of the communication and computation overlap group reaches the optimal level.

6. A processing apparatus, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 4.

7. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.

Images