Task execution method and apparatus in distributed computing process of ai model

Abstract

The present application relates to the field of distributed computing, and provides a task execution method and apparatus in a distributed computing process of an AI model. The method is applied to a distributed computing system. In the distributed computing process of the AI model, for a computing task and a communication task to be executed by an acceleration device, if there is a dependency relationship between the computing task and the communication task, the computing task and the communication task may be split on the basis of the dependency relationship into multiple pairs of subtasks having the same dependency relationship. In this way, by properly scheduling said subtasks, a host can mask time overheads of some subtasks, thereby improving task execution efficiency.

Description

Task execution method and device in AI model distributed computing process

[0001] The present application claims priority to the Chinese patent application No. 202410889945.7, filed on July 3, 2024, and entitled "Task execution method and device in AI model distributed computing process", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of distributed computing, in particular to a task execution method and device in AI model distributed computing process. BACKGROUND

[0003] With the increasing size of computing tasks in artificial intelligence (AI) scenarios, distributed computing emerges as the times require. Generally, a distributed computing system includes a host and an acceleration device, and the host can control the acceleration device to execute a distributed computing task.

[0004] Taking the training scenario of an AI large model as an example, a parallel strategy (such as model parallel, sequence parallel, and hybrid expert parallel, etc.) is adopted to split a to-be-executed computing task into multiple subtasks, and these subtasks are distributed to different acceleration devices for execution in a pipelining manner, which can reduce the memory occupation of the acceleration device.

[0005] However, the above-mentioned manner needs to insert a communication task having a dependency relationship with the computing task in the pipeline to realize data exchange between different acceleration devices, which will make the computing task and the communication task have to be executed in series when being scheduled, so that the computing resource is idle when the communication task is executed, resulting in unignorable communication overhead and affecting the task execution efficiency. SUMMARY

[0006] Embodiments of the present application provide a task execution method and device in AI model distributed computing process, which can reduce communication overhead and improve task execution efficiency.

[0007] In a first aspect, the present application provides a task execution method in AI model distributed computing process, applied to a distributed computing system, the system including a host and an acceleration device, and the method including:

[0008] The host determines a computing task and a communication task to be executed by the acceleration device based on a distributed parallel computing strategy of an AI model.

[0009] The host determines a plurality of sub-computing tasks of the computing task and a plurality of sub-communication tasks of the communication task based on a dependency relationship between the computing task and the communication task, the dependency relationship indicating that the computing task depends on an execution result of the communication task, or the communication task depends on an execution result of the computing task.

[0010] The host sends, to the acceleration device, a first sub-computing task in the plurality of sub-computing tasks and a first sub-communication task in the plurality of sub-communication tasks, and the first sub-computing task and the first sub-communication task have the dependency relationship therebetween.

[0011] The acceleration device executes the first sub-computing task and the first sub-communication task according to the dependency relationship.

[0012] The distributed computing task of the AI model can be a distributed training task of the AI model or a distributed inference task of the AI model, which is not limited in the application. Through the above method, in the distributed computing process of the AI model, if there is a dependency relationship between the computing task and the communication task to be executed by the acceleration device, the computing task and the communication task can be split into a plurality of pairs of sub-tasks having the same dependency relationship according to the dependency relationship. In this way, the host can mask the time overhead of part of the sub-tasks by reasonably scheduling the sub-tasks, thereby improving the task execution efficiency.

[0013] In some embodiments, the dependency relationship is a first dependency relationship, and the first dependency relationship indicates that the communication task depends on the execution result of the computing task. For example, in the scenario of enabling model parallelism in large model training and inference, Matmul+Allreduce is a computing-then-communication task segment having the first dependency relationship. For another example, in the scenario of enabling model parallelism and sequence parallelism in large model training, Mamul+ReduceScatter is a computing-then-communication task segment having the first dependency relationship. For another example, in the scenario of enabling MoE parallelism and long sequence parallelism in large model training, Matmul+All2All is a computing-then-communication task segment having the first dependency relationship. Of course, the application is not limited to the above scenarios, for example, other computing tasks such as BiasAdd can be introduced on the basis of the above task segments. Based on this, the method further includes:

[0014] The host sends, to the acceleration device, a second sub-computing task in the plurality of sub-computing tasks and a second sub-communication task in the plurality of sub-communication tasks, and the second sub-computing task and the second sub-communication task have the dependency relationship therebetween.

[0015] The acceleration device executes the first sub-computing task and the first sub-communication task according to the dependency relationship, including: the acceleration device executes the first sub-communication task and the second sub-computing task in parallel after executing the first sub-computing task, and executes the second sub-communication task after executing the second sub-computing task.

[0016] In some embodiments, the dependency relationship is a second dependency relationship, and the second dependency relationship refers to that the computing task depends on the execution result of the communication task. For example, in the scenario of large model training enabling model parallelism and sequence parallelism, AllGather+Matmul is a pre-communication post-computing task segment with the second dependency relationship; for another example, in the scenario of large model training enabling MoE parallelism and long sequence parallelism, All2All+Matmul is a pre-communication post-computing task segment with the second dependency relationship; of course, the present application is not limited to the above scenarios, for example, other computing tasks such as BiasAdd can also be introduced on the basis of the above task segments. Based on this, the method further includes:

[0017] The host sends, to the acceleration device, a second sub-computing task in the plurality of sub-computing tasks and a second sub-communication task in the plurality of sub-communication tasks, and the second sub-computing task and the second sub-communication task have a dependency relationship therebetween;

[0018] The acceleration device executes the first sub-computing task and the first sub-communication task according to the dependency relationship, including: the acceleration device executes the first sub-communication task and the second sub-computing task in parallel after executing the first sub-computing task, and executes the second sub-communication task after executing the second sub-computing task.

[0019] Since the host performs fine-grained splitting on the computing task and the communication task, part of the communication time can be masked through reasonable task scheduling and parallel execution.

[0020] In some embodiments, the AI model includes a plurality of network layers, and the acceleration device executes at least one network layer; wherein,

[0021] If the dependency relationship is a first dependency relationship, the first dependency relationship refers to that the communication task depends on the execution result of the computing task, the computing task indicates to perform computation on the at least one network layer, and the communication task indicates to exchange data with an acceleration device other than the acceleration device in the system based on output data of the at least one network layer.

[0022] If the dependency relationship is a second dependency relationship, the second dependency relationship refers to that the computing task depends on the execution result of the communication task, the communication task indicates to exchange data with an acceleration device other than the acceleration device in the system to obtain input data of the at least one network layer, and the computing task indicates to perform computation on the at least one network layer based on the input data of the at least one network layer.

[0023] In some embodiments, before the host determines the computing task and the communication task to be executed by the acceleration device based on the distributed parallel computing strategy of the AI model, the method further comprises:

[0024] The host evaluates performance overhead of the acceleration device executing the computing task and the communication task respectively according to the first task execution strategy and the second task execution strategy, to obtain a first performance analysis result corresponding to the first task execution strategy and a second performance analysis result corresponding to the second task execution strategy, the first task execution strategy indicating that the split computing task and the split communication task are executed after the computing task and the communication task are split respectively, and the second task execution strategy indicating that the computing task and the communication task are executed without splitting the computing task and the communication task;

[0025] If the performance overhead indicated by the first performance analysis result is less than the performance overhead indicated by the second performance analysis result, the step of determining the computing task and the communication task based on the distributed parallel computing strategy is executed according to the first task execution strategy; if the performance overhead indicated by the first performance analysis result is greater than or equal to the performance overhead indicated by the second performance analysis result, the first performance analysis result and the second performance analysis result are fed back to the user, and the acceleration device is controlled to execute the computing task and the communication task based on the task execution strategy indicated by the user.

[0026] In some embodiments, the number of the plurality of sub-computing tasks and the number of the plurality of sub-communication tasks are both the first number, and the distributed parallel computing strategy comprises the first number.

[0027] In some embodiments, the number of the plurality of sub-computing tasks and the number of the plurality of sub-communication tasks are both the second number, and the second number is determined based on a reference performance analysis result corresponding to the computing task and the communication task.

[0028] In some embodiments, before the host determines the plurality of sub-computing tasks of the computing task and the plurality of sub-communication tasks of the communication task, the method further comprises:

[0029] The host determines a reference execution strategy corresponding to each reference number based on the dependency relationship and the at least one reference number, the reference execution strategy indicating that the acceleration device executes the plurality of sub-computing tasks of the computing task and the plurality of sub-communication tasks of the communication task determined based on the corresponding reference number according to the dependency relationship;

[0030] The host determines a second number meeting a condition from the at least one reference number based on a reference performance analysis result corresponding to each reference execution strategy, the reference performance analysis result indicating performance overhead of the acceleration device executing the computing task and the communication task according to the corresponding reference execution strategy.

[0031] This process is also, the number of sub-tasks is an adjustable hyperparameter, which can be determined according to the performance analysis results in different scenarios, so as to improve the overall task execution efficiency.

[0032] In some embodiments, before the acceleration device executes the first sub-computing task according to the dependency relationship, the method further comprises any one of the following:

[0033] The host pre-processes the input data of the computing task, and sends the pre-processed input data to the acceleration device; or

[0034] The host sends a pre-processing task to the acceleration device, and the acceleration device pre-processes the input data based on the pre-processing task.

[0035] With the above method, before the acceleration device executes the sub-computing task, the pre-processing is executed first, and then each sub-task can be further executed on the pre-processed result, so that redundant calculation of pre-processing is avoided, and computing resources are saved.

[0036] In some embodiments, the method further comprises any one of the following:

[0037] If the communication task belongs to the first task type, after the acceleration device executes the plurality of sub-communication tasks, the host controls the acceleration device to execute a first memory rearrangement task, the first task type indicates that the communication task is a collection type task, the first memory rearrangement task indicates that the execution results of the plurality of sub-communication tasks are rearranged in memory, and after the first memory rearrangement task is executed, the execution results of the plurality of sub-communication tasks are equivalent to the execution results of the communication task; or

[0038] If the communication task belongs to the second task type, before the acceleration device executes the plurality of sub-communication tasks, the host controls the acceleration device to execute a second memory rearrangement task, the second task type indicates that the communication task is a dispersion type task, the second memory rearrangement task indicates that the input data of the plurality of sub-communication tasks is rearranged in memory, and after the second memory rearrangement task is executed, the execution results of the plurality of sub-communication tasks are equivalent to the execution results of the communication task.

[0039] In the above manner, it can be ensured that the execution results before and after the communication task is split are equivalent, or in other words, to ensure the accuracy consistency of different operators in the computing process.

[0040] In some embodiments, the first memory rearrangement task and the second memory rearrangement task are both implemented based on a transpose operator.

[0041] In some embodiments, the method further comprises:

[0042] The host controls the acceleration device to call the communication interface, and stores an execution result of each sub-communication task to a memory space corresponding to each sub-communication task based on an offset corresponding to each sub-communication task, wherein the offset is determined based on an input data amount of the corresponding sub-communication task and a quantity of the plurality of sub-communication tasks, and the execution result stored in the memory space corresponding to each sub-communication task is equivalent to an execution result of the communication task.

[0043] In a case where the distributed computing system supports calling the communication interface, the host controls the acceleration device to call the communication interface, and writes the execution result of each sub-communication task to a correct memory space, thereby ensuring that the execution result is equivalent before and after the communication task is split, and thus the additional overhead caused by memory rearrangement can be avoided.

[0044] In a second aspect, an embodiment of the present application provides a task execution method in an AI model distributed computing process, applied to a host in a distributed computing system, the system further comprising an acceleration device, and the method comprising:

[0045] determining a computing task and a communication task to be executed by the acceleration device based on a distributed parallel computing strategy of the AI model;

[0046] determining a plurality of sub-computing tasks of the computing task and a plurality of sub-communication tasks of the communication task based on a dependency relationship between the computing task and the communication task, the dependency relationship indicating that the computing task depends on an execution result of the communication task, or the communication task depends on an execution result of the computing task;

[0047] sending a first sub-computing task in the plurality of sub-computing tasks and a first sub-communication task in the plurality of sub-communication tasks to the acceleration device, the first sub-computing task and the first sub-communication task having the dependency relationship therebetween.

[0048] In some embodiments, the dependency relationship is a first dependency relationship, the first dependency relationship indicating that the communication task depends on the execution result of the computing task; and the method further comprises:

[0049] sending a second sub-computing task in the plurality of sub-computing tasks and a second sub-communication task in the plurality of sub-communication tasks to the acceleration device, the second sub-computing task and the second sub-communication task having the dependency relationship therebetween;

[0050] wherein the acceleration device is configured to execute the first sub-communication task and the second sub-computing task in parallel after executing the first sub-computing task, and execute the second sub-communication task after executing the second sub-computing task.

[0051] In some embodiments, the dependency relationship is a second dependency relationship, the second dependency relationship indicating that the computing task depends on an execution result of the communication task; the method further includes:

[0052] sending, to the acceleration device, a second sub-computing task in the plurality of sub-computing tasks and a second sub-communication task in the plurality of sub-communication tasks, the second sub-computing task and the second sub-communication task having the dependency relationship therebetween;

[0053] wherein the acceleration device is configured to execute the first sub-communication task first, and then execute the first sub-computing task and the second sub-communication task in parallel, and after executing the second sub-communication task, execute the second sub-computing task.

[0054] In some embodiments, the AI model includes a plurality of network layers, and at least one network layer is executed on the acceleration device; wherein,

[0055] If the dependency relationship is a first dependency relationship, the first dependency relationship indicating that the communication task depends on an execution result of the computing task, the computing task indicating computing on the at least one network layer, and the communication task indicating data exchange with an acceleration device other than the acceleration device in the system based on output data of the at least one network layer;

[0056] If the dependency relationship is a second dependency relationship, the second dependency relationship indicating that the computing task depends on an execution result of the communication task, the communication task indicating data exchange with an acceleration device other than the acceleration device in the system to obtain input data of the at least one network layer, and the computing task indicating computing on the at least one network layer based on the input data of the at least one network layer.

[0057] In some embodiments, before determining the computing task and the communication task to be executed by the acceleration device based on the distributed parallel computing strategy of the AI model, the method further includes:

[0058] evaluating performance overheads of the acceleration device in executing the computing task and the communication task according to a first task execution strategy and a second task execution strategy respectively, to obtain a first performance analysis result corresponding to the first task execution strategy and a second performance analysis result corresponding to the second task execution strategy, the first task execution strategy indicating that the computing task and the communication task are executed after being split, and the second task execution strategy indicating that the computing task and the communication task are executed without being split;

[0059] If the performance overhead indicated by the first performance analysis result is less than the performance overhead indicated by the second performance analysis result, the step of determining the plurality of sub-computing tasks of the computing task and the plurality of sub-communication tasks of the communication task based on the distributed parallel computing strategy is performed according to the first task execution strategy; if the performance overhead indicated by the first performance analysis result is greater than or equal to the performance overhead indicated by the second performance analysis result, the first performance analysis result and the second performance analysis result are fed back to a user, and the computing task and the communication task are executed by the acceleration device based on a task execution strategy indicated by the user.

[0060] In some embodiments, the number of the plurality of sub-computing tasks and the number of the plurality of sub-communication tasks are a first number, and the distributed parallel computing strategy comprises the first number.

[0061] In some embodiments, the number of the plurality of sub-computing tasks and the number of the plurality of sub-communication tasks are a second number, and the second number is determined based on a reference performance analysis result corresponding to the computing task and the communication task.

[0062] In some embodiments, before the step of determining the plurality of sub-computing tasks of the computing task and the plurality of sub-communication tasks of the communication task, the method further comprises:

[0063] Based on the dependency relationship and at least one reference number, a reference execution strategy corresponding to each reference number is determined, the reference execution strategy indicating that the acceleration device executes the plurality of sub-computing tasks of the computing task and the plurality of sub-communication tasks of the communication task determined based on the corresponding reference number according to the dependency relationship;

[0064] Based on a reference performance analysis result corresponding to each reference execution strategy, a second number meeting a condition is determined from the at least one reference number, the reference performance analysis result indicating a performance overhead of the acceleration device when executing the computing task and the communication task according to the corresponding reference execution strategy.

[0065] In some embodiments, before the step of sending the first sub-computing task to the acceleration device, the method further comprises any one of the following:

[0066] Preprocessing the input data of the computing task, and sending the preprocessed input data to the acceleration device; or,

[0067] Sending a preprocessing task to the acceleration device, and the acceleration device pre-processing the input data based on the preprocessing task.

[0068] In some embodiments, the method further comprises any one of the following:

[0069] If the communication task belongs to a first task type, the acceleration device is controlled to perform a first memory rearrangement task after the plurality of sub-communication tasks are executed, the first task type indicates that the communication task is a collection type task, the first memory rearrangement task indicates that the execution results of the plurality of sub-communication tasks are rearranged in memory, and the execution results of the plurality of sub-communication tasks are equivalent to the execution result of the communication task after the first memory rearrangement task is executed; or

[0070] If the communication task belongs to a second task type, the acceleration device is controlled to perform a second memory rearrangement task before the plurality of sub-communication tasks are executed, the second task type indicates that the communication task is a dispersion type task, the second memory rearrangement task indicates that the input data of the plurality of sub-communication tasks are rearranged in memory, and the execution results of the plurality of sub-communication tasks are equivalent to the execution result of the communication task after the second memory rearrangement task is executed.

[0071] In some embodiments, the first memory rearrangement task and the second memory rearrangement task are both implemented based on a transpose operator.

[0072] In some embodiments, the method further comprises:

[0073] The acceleration device is controlled to call a communication interface, and the execution result of each sub-communication task is stored in the memory space corresponding to each sub-communication task based on the offset corresponding to each sub-communication task, wherein the offset is determined based on the input data amount of the corresponding sub-communication task and the number of the plurality of sub-communication tasks, and the execution result stored in the memory space corresponding to each sub-communication task is equivalent to the execution result of the communication task.

[0074] In a third aspect, the present application provides a task execution device in an AI model distributed computing process, configured in a host of a distributed computing system, the system further comprising an acceleration device, the device comprising at least one functional unit, the at least one functional unit being used to implement the AI model distributed computing process task execution method provided in the foregoing second aspect or any one of the possible implementation manners of the second aspect.

[0075] In a fourth aspect, the present application provides a computing device, comprising a processor and a memory, the processor being used to execute at least one program code stored in the memory, so that the computing device implements the AI model distributed computing process task execution method provided in the foregoing second aspect or any one of the possible implementation manners of the second aspect.

[0076] In a fifth aspect, the present application provides a distributed computing system, the system comprising a host and an acceleration device, the host being configured to control the acceleration device, and the system being configured to implement the task execution method in the AI model distributed computing process as provided in the foregoing first aspect or any possible implementation manner of the first aspect.

[0077] In a sixth aspect, the present application provides a computer readable storage medium configured to store at least one program code, the at least one program code, when executed by a computing device, causing the computing device to implement the task execution method in the AI model distributed computing process as provided in the foregoing second aspect or any possible implementation manner of the second aspect. The storage medium includes, but is not limited to, a volatile memory such as a random access memory, a non-volatile memory such as a flash memory, a hard disk drive (HDD), and a solid state drive (SSD).

[0078] In a seventh aspect, the present application provides a computer program product, when the computer program product is executed on a computing device, causing the computing device to implement the task execution method in the AI model distributed computing process as provided in the foregoing second aspect or any possible implementation manner of the second aspect. The computer program product can be a software installation package, which can be downloaded and executed on the computing device when the foregoing method needs to be implemented. BRIEF DESCRIPTION OF DRAWINGS

[0079] FIG. 1 is a schematic diagram of column partitioning of a parameter matrix in a model parallel scenario;

[0080] FIG. 2 is a schematic diagram of row partitioning of a parameter matrix in a model parallel scenario;

[0081] FIG. 3 is a schematic diagram of a Transformer architecture;

[0082] FIG. 4 is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0083] FIG. 5 is a schematic diagram of a hardware structure of a computing device provided by an embodiment of the present application;

[0084] FIG. 6 is a schematic diagram of a principle of a task execution method provided by an embodiment of the present application;

[0085] FIG. 7 is a schematic diagram of a principle of another task execution method provided by an embodiment of the present application;

[0086] FIG. 8 is a schematic diagram of an architecture of a distributed computing system provided by an embodiment of the present application;

[0087] FIG. 9 is a flowchart of a task execution method in an AI model distributed computing process according to an embodiment of the present application;

[0088] FIG. 10 is a flowchart of another task execution method in an AI model distributed computing process according to an embodiment of the present application;

[0089] FIG. 11 is a flowchart of another task execution method in an AI model distributed computing process according to an embodiment of the present application;

[0090] FIG. 12 is a schematic diagram of a communication task according to an embodiment of the present application;

[0091] FIG. 13 is a schematic diagram of another communication task according to an embodiment of the present application;

[0092] FIG. 14 is a schematic diagram of calling a communication interface according to an embodiment of the present application;

[0093] FIG. 15 is a schematic diagram of another calling a communication interface according to an embodiment of the present application;

[0094] FIG. 16 is a schematic diagram of a framework of a task execution method in an AI model distributed computing process according to an embodiment of the present application;

[0095] FIG. 17 is a schematic diagram of concurrently executing subtasks according to an embodiment of the present application;

[0096] FIG. 18 is a schematic diagram of concurrently executing subtasks in a preprocessing scenario according to an embodiment of the present application;

[0097] FIG. 19 is a schematic diagram of another concurrently executing subtasks according to an embodiment of the present application;

[0098] FIG. 20 is a schematic diagram of another concurrently executing subtasks in a preprocessing scenario according to an embodiment of the present application;

[0099] FIG. 21 is a schematic diagram of yet another concurrently executing subtasks according to an embodiment of the present application;

[0100] FIG. 22 is a schematic diagram of yet another concurrently executing subtasks in a preprocessing scenario according to an embodiment of the present application;

[0101] FIG. 23 is a structural schematic diagram of a task execution apparatus in an AI model distributed computing process according to an embodiment of the present application. DETAILED DESCRIPTION

[0102] For the purposes of the present application, the technical solutions and advantages will be further described in detail below with reference to the drawings. It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.), and signals involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data need to comply with relevant national and regional laws, regulations, and standards. For example, the relevant data of the distributed computing task and the task execution instructions involved in the present application are obtained under full authorization.

[0103] For the convenience of understanding, the key terms and key concepts involved in the present application will be explained first.

[0104] An artificial intelligence (AI) model is a mathematical algorithm model that uses machine learning ideas to solve practical problems. Generally, an AI model includes a large number of parameters and calculation formulas (or calculation rules).

[0105] An acceleration device, also known as an accelerator, acceleration card, or acceleration chip, is a special hardware device or computer system designed to accelerate the calculation process in AI scenarios. In the embodiments of the present application, the acceleration device is, for example, a graphics processing unit (GPU), a neural network processing unit (XPU), an intelligent processing unit (IPU), a tensor processing unit (TPU), a domain specific architecture (DSA) chip, etc., without being limited thereto.

[0106] An operator (OP) is a computing unit or computing function running on a computing device. In the field of deep learning, neural network layers and even entire models are composed of operators, which correspond to the computing logic in the neural network layers. For example, a convolution layer is an operator, and the weight summation process in a fully-connected layer (FC layer) is an operator.

[0107] Collective communication, or aggregation communication, refers to a global data migration and global control operation between a group of processors in a multi-computer system, which is widely used in distributed computing. Using collective communication, all-to-all data transmission can be achieved between different processes in distributed computing. Common collective communication tasks include broadcast, barrier synchronization, reduce, scatter, gather, AllGather, AllScatter, AllReduce, etc.

[0108] Parallel strategy in distributed AI model training refers to splitting a computing task into multiple sub-tasks and assigning them to multiple nodes (such as acceleration devices) for execution, which makes full use of parallel computing resources, greatly shortens the training time, and speeds up the model convergence strategy. In related technologies, parallel strategies include model parallel (or tensor parallel), sequence parallel (SP), and mixture of experts (MoE) parallel, etc. The core idea of these parallel strategies is to split different parts of model weights or activation values among multiple acceleration devices, allowing each acceleration device to be responsible for a part of the model computation. This method not only breaks through the memory capacity limit of a single acceleration device, but also supports the training of larger models.

[0109] Profiling results refer to detailed information and data about the running state, resource usage, time consumption, etc. of a system, program or process after performance analysis or profiling. These results can help people understand the performance bottlenecks, optimization directions, etc. of the system. Generally, performance analysis results can include the following information: execution time, i.e. the time spent by each operation, function, module or stage; CPU usage, i.e. the CPU resource occupation of different parts; memory usage, i.e. memory occupation changes, peak values, etc.; data transmission volume, i.e. data transmission volume between networks or different components; calculation amount, such as floating point operation frequency, etc.; detailed statistics of specific operations or modules, such as the specific performance of a certain layer, etc. The present application does not limit this.

[0110] Address alignment refers to aligning data storage addresses according to certain boundaries, so that each memory access of the acceleration device can efficiently read or write complete data blocks. This alignment method can reduce memory access times and delays, as it enables the acceleration device to complete more data transmission in a single or minimum access period, thereby improving memory access efficiency and data processing speed. For example, when using an acceleration device for memory-intensive processing, memory bandwidth refers to the rate at which data is transmitted between memory and processing units, and is a key factor affecting computing efficiency. In order to maximize memory bandwidth, address alignment is often used to optimize memory access. In large models, tasks such as matrix multiplication (Matmul) have a high time proportion, and correct memory address alignment is crucial to achieving the highest memory access efficiency. Taking the calculation of the output matrix Y as an example, this process involves multiple readings of the left matrix X and the right matrix W. Considering the memory access pattern, if the right matrix W is stored in row-major order and its dimensions are K x N (K is the number of rows and N is the number of columns), the address offset required to read the i-th row of W is (i-1) x N. In this layout, if the value of N does not meet the P alignment (i.e., N is not a multiple of the alignment base P), the offset memory address is likely to also not meet the P alignment requirement, resulting in a significant performance decrease when accessing the i-th row of W. To solve this problem, the following two preprocessing methods are usually used to achieve address alignment:

[0111] Preprocessing method one: padding. Adjust the number of columns to meet the alignment requirement. Illustratively, adjust the number of columns N of W to N', where N' is calculated by the formula N' = ceil(N / P) x P to ensure that the starting address of each row of data meets P alignment. This method expands the matrix W to K x N' dimensions by adding redundant columns at the end of each row of the matrix W, thereby ensuring that each row of the expanded matrix W' can be stored in memory in a P-aligned manner. This not only optimizes the memory access pattern, but also improves the efficiency of data reading.

[0112] Preprocessing method two, transpose. Meet the alignment requirement by matrix transposition. Illustratively, when the dimension K itself meets the P alignment, transpose the matrix W, i.e., change from row-major storage to column-major storage. By changing the memory arrangement in this way and following the column-major access rule in subsequent operations, the non-aligned access problem caused by row-major arrangement can be effectively avoided. This method can be applied to scenarios where the dimension K already meets the P alignment condition. Through matrix transposition, the continuity of data access and overall computing efficiency can be further improved.

[0113] Through the above two preprocessing methods, the problem of low memory access efficiency in the non-aligned case can be improved, thereby achieving better memory utilization and acceleration effect when performing high-performance computing tasks such as matrix multiplication.

[0114] The application scenario and implementation environment of the present application are introduced below.

[0115] The present application can be applied to the distributed computing process of an AI model, for example, the distributed training task of an AI model, and for example, the distributed inference task of an AI model, which is not limited by the present application. Illustratively, a distributed computing system includes multiple acceleration devices, and since each of these acceleration devices is responsible for a part of the computing task, these acceleration devices need to additionally perform a communication task to realize data exchange between different acceleration devices. This process is to insert a communication task that has a dependency relationship with the computing task in the pipeline of the computing task to realize data exchange between different acceleration devices.

[0116] Taking the model parallel strategy in the distributed training process of an AI model as an example, the core idea of model parallel is to distribute the input, model parameters and operation of each layer of the AI model to different acceleration devices for execution. This strategy needs to solve two problems: (1) splitting method, that is, how to split the model parameters to different acceleration devices; (2) mathematical equivalence, that is, to ensure mathematical consistency after splitting. Generally, model parallel is aimed at the matrix multiplication operator (Matmul) of the Transformer architecture, which fully utilizes the principle of matrix block multiplication. For example, for matrix multiplication Y = X × W, where X is a parameter matrix with dimensions M × K, W is a parameter matrix with dimensions K × N, and Y is a result matrix with dimensions M × N. If the size of the parameter matrix W is large, even exceeding the memory capacity of a single acceleration device, the parameter matrix W can be split into multiple acceleration devices, and the results are collected through collective communication to ensure that the final result is mathematically equivalent to the result obtained by computing on a single acceleration device. Illustratively, there are two splitting methods for the parameter matrix W:

[0117] The first way is column splitting. Referring to FIG. 1, FIG. 1 is a schematic diagram of column splitting of a parameter matrix in a model parallel scenario. As shown in FIG. 1, the parameter matrix W is split into W = [W1|W2] by column, and W1 and W2 are placed on two acceleration devices respectively. The two acceleration devices calculate Y1 = X x W1 and Y2 = X x W2 respectively. After the calculation is completed, for any one of the acceleration devices, the acceleration device performs an AllGather task to obtain the calculation result on the other acceleration device, splices the calculation result of itself and the obtained calculation result together, and obtains the final result matrix Y. As can be seen, by splitting the parameter matrix W that cannot be accommodated by the memory of a single acceleration device, placing the parameter matrix W on two acceleration devices, and performing an AllGather communication task, the final result can be obtained, which is mathematically equivalent to the result obtained by calculating on a single acceleration device.

[0118] The second way is row splitting. Referring to FIG. 2, FIG. 2 is a schematic diagram of row splitting of a parameter matrix in a model parallel scenario. As shown in FIG. 2, in order to meet the matrix multiplication rule, the parameter matrix X is split into X = [X1|X2] by column, and the parameter matrix W is split into W = [W1|W2] by row. The split matrices are placed on two acceleration devices respectively, and each acceleration device calculates Y1 = X1 x W1 and Y2 = X2 x W2 respectively. After the calculation is completed, for any one of the acceleration devices, the acceleration device performs an AllReduce task to obtain the calculation result on the other acceleration device, adds the calculation result of itself and the obtained calculation result, and obtains the final result matrix Y. As can be seen, by splitting the parameter matrices W and X that cannot be accommodated by the memory of a single acceleration device, placing the parameter matrices W and X on two acceleration devices, and performing an AllReduce communication task, the final result can be obtained, which is mathematically equivalent to the result obtained by calculating on a single acceleration device.

[0119] Further, the attention architecture and the feed-forward network (FFN) architecture in the Transformer architecture together include four matrix multiplication operators, and through continuous optimization and exploration, the four matrix multiplication operators can respectively adopt the two splitting manners described above to generate as few communication requirements as possible. Referring to FIG. 3, FIG. 3 is a schematic diagram of a Transformer architecture. As shown in FIG. 3, when the Transformer architecture is trained by using the distributed AI model training framework, the splitting manner usually adopted is as follows: two AllReduce communication tasks (namely, AllReduce operators) are inserted at each layer of the model in the forward direction, which are located after Linear calculation tasks (namely, Linear operators, used for linear transformation). As can be seen, the AllReduce communication task and the Linear calculation task are communication tasks and calculation tasks with a dependency relationship, in addition, the reverse process also needs to insert the same AllReduce communication task, which will not be described herein again.

[0120] In other scenarios, there is also a need to insert dependent communication tasks in the pipeline of computing tasks, for example, in the scenario of model parallel combined with sequence parallel, by splitting the AllReduce operator into two communication operators ReduceScatter and AllGather, the redundant calculation can be reduced and the memory can be saved, in this scenario, the paradigm of interdependent computing tasks and communication tasks is added two kinds: (1) first execute the Matmul computing task, and then execute the ReduceScatter communication task, that is, the execution result of the matrix multiplication computing task is used as the input of the ReduceScatter communication task; (2) first execute the AllGather communication task, and then execute the Matmul computing task, that is, the execution result of the AllGather communication task is used as the input of the Matmul computing task. For another example, in the MoE parallel scenario, by distributing the input of the model to a set composed of multiple "expert" networks, and dynamically selecting the most suitable one or several among these experts to process a particular input, efficient calculation distribution and model scalability are achieved. In this parallel mode, all-to-all communication tasks become a key component, all-to-all communication refers to the process in which each acceleration device needs to exchange data with all other acceleration devices. For example, each input is first sent to one or more experts that are most suitable for processing it, which requires data exchange between all participating acceleration devices to ensure that each expert receives the input part it needs to process, and returns the execution result to the correct location for the next calculation or aggregation. In this process, the content of all-to-all communication is dependent on the execution result of the computing task, and the subsequent computing task of the local acceleration device is dependent on the data distributed from other acceleration devices (i.e., the execution result of the communication task), in other words, MoE parallel can be understood as a mode of serial execution of calculation, communication and calculation, accordingly, communication delay will also affect the overall calculation efficiency.

[0121] The above describes the communication tasks inserted in the pipeline of computing tasks by taking model parallel and MoE parallel as examples, it should be understood that in actual application, it is not limited to the above several cases, usually, in the distributed training process of AI model, communication tasks can be divided into the following two categories:

[0122] (1) Weak dependence on the communication task of the computing task. For example, the AllReduce communication task required by the backward gradient aggregation in the data parallel mode, which allows efficient aggregation of the gradients calculated in the back propagation process among multiple acceleration devices. Under this mechanism, each acceleration device participating in the calculation first completes the forward and reverse calculations on the data batch allocated to itself, and then aggregates the gradients on all acceleration devices by performing the AllReduce communication task. It should be understood that this aggregation is a global operation that can ensure that each acceleration device has complete and consistent gradient results before updating the model parameters. The key feature of AllReduce communication is that it can be executed in parallel with the subsequent reverse calculation task on the acceleration device. During the backward propagation of the gradient, for any acceleration device, after the gradient calculation of the current reverse calculation task is completed, the acceleration device can start to perform the communication task of gradient aggregation, while the next reverse calculation task can be performed at the same time, so the overhead of the communication task can be masked by overlapping with other non-dependent calculation tasks, thereby minimizing the impact on the overall calculation time, that is, it can be understood as a weak dependence on the communication task of the computing task.

[0123] (2) Strong dependence on the communication task of the computing task. For example, the communication task in the model parallel mode, since model parallelism involves distributing different parts of the model to different acceleration devices, the communication task between acceleration devices becomes a necessary part of the model forward and reverse calculation process. This communication task is usually manifested as the execution result of the computing task needs to be used as the input of another acceleration device executing the computing task, or the execution result of the communication task needs to be used for the computing task executed by the current acceleration device. There is a logical dependency between these tasks, so they need to be executed in series. This dependency results in the overhead of the communication task cannot be masked, thereby affecting the overall calculation time.

[0124] As can be seen, due to the insertion of communication tasks with dependencies on computing tasks in the pipeline of computing tasks, the computing tasks and the communication tasks need to be executed in series when scheduling, so that the computing resources are in an idle state when the communication tasks are executed, resulting in unmaskable communication overhead, also known as "unmaskable communication". Therefore, researching parallel masking methods for dependent computing and communication tasks and reducing the proportion of end-to-end "unmaskable communication" on the model have become key to improving model training efficiency.

[0125] Based on this, the present application provides a task execution method in the distributed computing process of an AI model, which can be applied to various dependent computing tasks and communication tasks in distributed computing tasks, and improves the task execution efficiency. First, with reference to FIG. 4, the implementation environment of the embodiment of the present application is introduced.

[0126] FIG. 4 is a schematic diagram of an implementation environment according to an embodiment of the present application. As shown in FIG. 4, the implementation environment includes a distributed computing system 400, which includes a host 401 and an acceleration device 402, and the host 401 and the acceleration device 402 are communicatively connected. In some embodiments, the acceleration device 402 is also referred to as an acceleration card, an accelerator, etc., and the present application is not limited thereto.

[0127] In the embodiments of the present application, the distributed computing system 400 is a system with cross-device communication capability, which can perform various distributed computing tasks, such as distributed training tasks of AI models, or distributed inference tasks of AI models, and the present application is not limited thereto. Illustratively, the host 401 refers to a device for running a computing application, which can access a wired network or a wireless network, and is used to control the acceleration device 402 to perform various distributed computing tasks initiated by the computing application. In addition, the number of hosts 401 can be one or more, and the present application is not limited thereto.

[0128] The number of acceleration devices 402 is multiple, which are used to cooperatively perform distributed computing tasks. Among them, for any acceleration device 402, the acceleration device 402 has storage capability and computing capability, which are used to provide storage space and computing power for the distributed computing tasks of the distributed computing system 400. For example, the acceleration device 402 is a GPU, an XPU, an IPU, a TPU, a DSA chip, etc., and the present application is not limited thereto. In addition, the number of acceleration devices 402 shown in FIG. 4 is only illustrative, and the number of acceleration devices 402 can be more or less, and the present application is not limited thereto.

[0129] In the distributed computing system 400, each acceleration device 402 can be communicatively connected through a high-speed interconnection link (or a chip bus), so that different acceleration devices 402 can quickly access each other's memory through the high-speed interconnection link, realizing efficient data transmission between different acceleration devices 402. For example, the high-speed interconnection link is a Nvidia Link (NvLink), a compute express link (CXL), a universal chiplet interconnect express (UCIe), a Huawei cache coherent system (HCCS), a cache coherent interconnect for accelerators (CCIX), etc., and the present application is not limited thereto.

[0130] Illustratively, the host 401 and the plurality of acceleration devices 402 are integrated in one server, and the host 401 and the plurality of acceleration devices 402 are connected through a peripheral component interconnect express (PCIe) link. The host 401 communicates with the plurality of acceleration devices 402 through the PCIe link, controls the plurality of acceleration devices 402 to perform corresponding tasks in parallel. For example, the server is a standalone physical server, or a server cluster or a distributed file system composed of a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, etc. basic cloud computing services. Taking the cloud server as an example, the cloud server is also called a cloud platform (i.e., a short name for a cloud computing platform), which is a service based on hardware resources and software resources, providing computing, network and storage capabilities. Through the network “cloud”, huge data computing is processed and analyzed in the remote end and returned to the user, which has the characteristics of large scale, distribution, virtualization, high availability, scalability, on-demand service and security, etc. The cloud platform can realize the rapid delivery and release of configurable computing resources with small management cost or low interaction complexity between the user and the service provider.

[0131] In some embodiments, the wireless network or the wired network described above uses standard communication technologies and / or protocols. The network is usually a transmission control protocol / internet protocol (TCP / IP) network and an RDMA network in a data center network, such as an RDMA over converged Ethernet (RoCE) network, an InfiniBand (IB) network, etc., without limitation. In some other embodiments, custom and / or dedicated data communication technologies can be used instead of or in addition to the above data communication technologies.

[0132] The hardware structure of the host 401 in the distributed computing system 400 described above is introduced as follows.

[0133] The application provides a computing device which can be configured as the host 401 in the distributed computing system 400, and reference is made to FIG. 5 which is a schematic diagram of a hardware structure of a computing device provided in an embodiment of the application. As shown in FIG. 5, the computing device 500 comprises a memory 501, a processor 502, a communication interface 503 and a bus 504. The memory 501, the processor 502 and the communication interface 503 are communicatively connected to each other through the bus 504.

[0134] The memory 501 refers to a device for storing data, which can be a read-only memory (ROM) or other types of static storage device capable of storing static information and instructions, a random access memory (RAM) or other types of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disc storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program codes in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto. Illustratively, the memory 501 is used to store at least one program code, and when the program code stored in the memory 501 is executed by the processor 502, the processor 502 is used to execute the steps performed by the host in the method embodiments described below.

[0135] The processor 502 can be a network processor (NP), a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC) or an integrated circuit for controlling the execution of programs of the application. The processor 502 can be a single-CPU processor or a multi-CPU processor. The number of processors 502 can be one or more. The memory 501 and the processor 502 can be separately arranged or integrated together.

[0136] The communication interface 503 uses a transceiving module such as a transceiver to enable communication between the computing device 500 and other devices or communication networks. For example, data can be acquired through the communication interface 503.

[0137] The bus 504 can include a path for communicating information among the various components of the computing device 500 (e.g., the memory 501, the processor 502, the communication interface 503).

[0138] Embodiments of the present application also provide a computing device cluster, which includes at least one computing device, and each computing device in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc.

[0139] Based on the foregoing introduction to the application scenarios and implementation environments of the present application, the task execution method provided by the present application will be introduced below.

[0140] Based on the foregoing introduction, in some scenarios, the distributed computing system inserts a communication task that has a dependency relationship with a computing task in the pipeline of the computing task to implement data exchange between different acceleration devices during the execution of the distributed computing task of the AI model. Based on this, the present application provides a task execution method, which further finely divides the computing task and the communication task assigned to a certain acceleration device, and then reasonably schedules the divided tasks according to the dependency relationship between the computing task and the communication task, so as to mask part of the communication time and improve the task execution efficiency. The computing task and the communication task that have a dependency relationship can be understood as a dependent task segment.

[0141] In addition, in the embodiments of the present application, the dependency relationship between the computing task and the communication task includes two kinds, one is that the communication task depends on the execution result of the computing task, and the other is that the computing task depends on the execution result of the communication task. For example, in the scenario of large model training and inference with open model parallelism, Matmul+Allreduce is a first dependency relationship existing computing first and then communication task segment; for another example, in the scenario of large model training with open model parallelism and sequence parallelism, Mamul+ReduceScatter is a first dependency relationship existing computing first and then communication task segment, AllGather+Matmul is a second dependency relationship existing communication first and then computing task segment; for another example, in the scenario of large model training with open MoE parallelism and long sequence parallelism, Matmul+All2All is a first dependency relationship existing computing first and then communication task segment, All2All+Matmul is a second dependency relationship existing communication first and then computing task segment; of course, the present application is not limited to the above scenarios, for example, it can also be in the scenario of introducing other computing tasks such as BiasAdd on the basis of the above task segments.

[0142] The principle of the task execution method provided by the present application will be introduced first with reference to FIG. 6 and FIG. 7. The distributed computing system includes a host and multiple acceleration devices, the host creates a distributed computing task based on the distributed parallel computing strategy of the AI model, and respectively issues the computing task and the communication task to the multiple acceleration devices, it should be understood that the computing task and the communication task executed on each acceleration device are also the subtasks of the distributed computing task.

[0143] As shown in FIG. 6, FIG. 6 is a schematic diagram of a task execution method provided in an embodiment of the present application. There is a dependency relationship between the computing task and the communication task sent by the host to the acceleration device, which is that the communication task depends on the execution result of the computing task, referred to as the first dependency relationship hereinafter. In the related art, the host sends the computing task and the communication task to the acceleration device in sequence according to the computation graph, and the acceleration device sequentially executes the computing task and the communication task. In the present application, the host further divides the computing task and the communication task according to the dependency relationship therebetween, to obtain fine-grained subtasks, that is, the computing task is divided into sub-computing task 0 (simplified as computing 0 in the figure, the same hereinafter, and details are not repeated), sub-computing task 1, sub-computing task 2 and sub-computing task 3, and the communication task is divided into sub-communication task 0, sub-communication task 1, sub-communication task 2 and sub-communication task 3 (denoted as the number of split parts = 4). Among them, sub-communication task 0 depends on the execution result of sub-computing task 0, sub-communication task 1 depends on the execution result of sub-computing task 1, sub-communication task 2 depends on the execution result of sub-computing task 2, and sub-communication task 3 depends on the execution result of sub-computing task 3. Based on this, since sub-communication task 0 depends on the execution result of sub-computing task 0, it is not necessary to wait for the execution results of sub-computing task 1, sub-computing task 2 and sub-computing task 3. Therefore, after the host sends sub-computing task 0 to the acceleration device, it sends sub-communication task 0 and sub-computing task 1 to the acceleration device, so that the acceleration device can execute sub-communication task 0 and sub-computing task 1 in parallel. Similarly, sub-communication task 1 and sub-computing task 2 can be executed in parallel, and sub-communication task 2 and sub-computing task 3 can also be executed in parallel. In this way, compared with the related art, part of the communication time can be masked, and the task execution efficiency is improved. Moreover, theoretically, when the execution time of the computing task is greater than the execution time of the communication task and the number of split parts is 4, the communication time that can be masked can reach 3 / 4 of the original communication time; when the number of split parts is 8, the communication time that can be theoretically masked will reach 7 / 8 of the original communication time. It can be seen that through reasonable task scheduling and parallel execution, most of the communication time can be masked by the computing task, so that efficient computing and communication parallel processing is realized.

[0144] As shown in FIG. 7, FIG. 7 is a principle schematic diagram of another task execution method provided by the embodiments of the present application. There is a dependency relationship between the computing task and the communication task sent by the host to the acceleration device, which is that the computing task depends on the execution result of the communication task, referred to as a second dependency relationship hereinafter. In the related art, the host sends the communication task and the computing task to the acceleration device in sequence according to the computation graph, and the acceleration device sequentially executes the communication task and the computing task. In the present application, based on the same process as the foregoing FIG. 6, the host further divides the computing task and the communication task according to the dependency relationship therebetween, to obtain fine-grained subtasks, wherein each sub-computing task depends on the execution result of the corresponding sub-communication task. Based on this, since the sub-computing task 0 depends on the execution result of the sub-communication task 0, it is not necessary to wait for the execution results of the sub-communication task 1, the sub-communication task 2 and the sub-communication task 3, therefore, after the host sends the sub-communication task 0 to the acceleration device, it sends the sub-computing task 0 and the sub-communication task 1 to the acceleration device, so that the acceleration device can execute the sub-computing task 0 and the sub-communication task 1 in parallel. Similarly, the sub-computing task 1 and the sub-communication task 2 can be executed in parallel, and the sub-computing task 2 and the sub-communication task 3 can also be executed in parallel, which will not be described herein again.

[0145] Based on the principles of the task execution methods shown in the foregoing FIG. 6 and FIG. 7, the architecture of the distributed computing system using the method will be introduced below with reference to FIG. 8.

[0146] FIG. 8 is a schematic diagram of the architecture of a distributed computing system provided by the embodiments of the present application. It should be understood that FIG. 8 only exemplarily shows a structured schematic diagram of the distributed computing system, and the present application does not limit the division of various functional modules in the system. As shown in FIG. 8, the distributed computing system includes a host and multiple acceleration devices. Exemplarily, the host includes a management module 801 and an execution module 802, and for any one of the acceleration devices, the acceleration device includes a computing module 803 and a communication module 804.

[0147] The management module 801 of the host is configured to provide a management function for the AI model distributed computing process, for example, generate a distributed parallel computing strategy of the AI model according to the model architecture of the AI model, and create a distributed computing task according to the distributed parallel computing strategy. In some embodiments, the management module 801 is further configured to evaluate the performance overhead of the acceleration device executing the task according to different execution strategies, to obtain corresponding performance analysis results, and determine a suitable execution strategy according to the performance analysis results, and then control the acceleration device to execute the corresponding task. This process will be described in detail in subsequent method embodiments, which will not be described herein again. For another example, the management module 801 is further configured to provide a management function for each acceleration device in the system, and the like.

[0148] The execution module 802 of the host is configured to provide a task execution function for the AI model distributed computing process. Illustratively, the execution module 802 is configured to provide a task determination function 8021, a subtask determination function 8022, and a task sending function 8023. The task determination function 8021 refers to determining the computing task and the communication task to be executed by each acceleration device in the system based on the distributed parallel computing strategy of the AI model. The subtask determination function 8022 refers to dividing the computing task and the communication task respectively based on the dependency relationship between the computing task and the communication task to be executed by any acceleration device, to obtain a plurality of sub-computing tasks and a plurality of sub-communication tasks. The task sending function 8023 refers to sending each subtask to the acceleration device to control the acceleration device to execute the corresponding subtask. In some embodiments, the execution module 802 is further configured to provide a preprocessing function 8024 and a memory rearrangement function 8025, etc. The preprocessing function 8024 refers to preprocessing the input data of the computing task, and the memory rearrangement function 8025 refers to memory rearrangement of the execution result of the communication task, or memory rearrangement of the input data of the communication task. This process will be described in detail in subsequent method embodiments, and will not be described here.

[0149] For any acceleration device, the computing module 803 of the acceleration device is configured to provide a computing function for part of the network layers in the AI model, that is, to execute the computing task sent by the host. The communication module 804 of the acceleration device is configured to provide a data exchange function with other acceleration devices in the system, that is, to execute the communication task sent by the host. For example, the AI model includes a plurality of network layers, and at least one network layer is executed on the acceleration device. In the training scenario of the AI model, the computing module 803 of the acceleration device is configured to calculate the gradient of the at least one network layer in the back propagation process, and the communication module 804 is configured to send the calculated gradient parameter to other acceleration devices.

[0150] In addition, the module division and function division on the host and the acceleration device are not limited to the content shown in FIG. 8. In actual applications, more functions can be set according to user needs, for example, a storage module is configured on the host and the acceleration device to store related data in the AI model distributed computing process.

[0151] The following describes a task execution method provided by the present application through several method embodiments.

[0152] FIG. 9 is a flowchart of a task execution method in an AI model distributed computing process according to an embodiment of the present application. As shown in FIG. 9, the task execution method is applied to a distributed computing system including a host and an acceleration device. Taking the interaction between the host and the acceleration device as an example, the task execution method includes the following steps 901 to 904.

[0153] 901. The host determines, based on the distributed parallel computing strategy of the AI model, a computing task and a communication task to be executed by the acceleration device, and the computing task and the communication task have a dependency relationship.

[0154] In the embodiments of the present application, a computing application runs on the host, can access a wired network or a wireless network, and provides AI services for users. The acceleration device refers to any acceleration device in a distributed computing system. Illustratively, in response to an execution request of an AI model, the host generates a distributed parallel computing strategy of the AI model based on a model architecture of the AI model, creates a distributed computing task of the AI model based on the distributed parallel computing strategy, determines a plurality of acceleration devices participating in the distributed computing task, and determines a computing task and a communication task to be executed on each acceleration device. The execution request of the AI model can be a training request for the AI model or an inference request for the AI model, and accordingly, the distributed computing task of the AI model can be a distributed training task of the AI model or a distributed inference task of the AI model, which is not limited in the present application. Taking any acceleration device as an example, the computing task and the communication task to be executed on the acceleration device are subtasks of the distributed computing task, and the communication task refers to a task of exchanging data with other acceleration devices.

[0155] In addition, the dependency relationship can be a first dependency relationship or a second dependency relationship, wherein the first dependency relationship refers to that the communication task depends on the execution result of the computing task, and accordingly, the execution order of the computing task and the communication task is to execute the computing task first and then execute the communication task; the second dependency relationship refers to that the computing task depends on the execution result of the communication task, and accordingly, the execution order of the computing task and the communication task is to execute the communication task first and then execute the computing task.

[0156] Illustratively, the AI model includes a plurality of network layers, at least one network layer is executed on the acceleration device, if the dependency relationship is the first dependency relationship, the computing task indicates to perform computation on the at least one network layer, and the communication task indicates to exchange data with an acceleration device other than the acceleration device in the system based on output data of the at least one network layer. If the dependency relationship is the second dependency relationship, the communication task indicates to exchange data with an acceleration device other than the acceleration device in the system to obtain input data of the at least one network layer, and the computing task indicates to perform computation on the at least one network layer based on the input data of the at least one network layer. For example, taking the training scenario of the AI model as an example, executing at least one network layer on the acceleration device means performing forward computation on the at least one network layer to obtain activation values, and passing the activation values to other acceleration devices, and performing backward computation on the at least one network layer to obtain gradient parameters, and passing the gradient parameters to other acceleration devices. For another example, taking the inference scenario of the AI model as an example, executing at least one network layer on the acceleration device means performing inference computation on the at least one network layer to obtain output data of the at least one network layer, and passing the output data to other acceleration devices.

[0157] The following refers to steps 902 to 904 to introduce the process in which the host controls the acceleration device to perform corresponding tasks, taking any one of the acceleration devices as an example.

[0158] 902. The host determines a plurality of sub-computing tasks of the computing task and a plurality of sub-communication tasks of the communication task based on a dependency relationship between the computing task and the communication task to be performed by the acceleration device.

[0159] In the embodiments of the present application, the host determines a plurality of sub-computing tasks of the computing task and a plurality of sub-communication tasks of the communication task based on a dependency relationship between the computing task and the communication task, wherein the number of the plurality of sub-computing tasks and the number of the plurality of sub-communication tasks are the same, one sub-computing task corresponds to one sub-communication task, that is, for any one sub-computing task, there is one sub-communication task corresponding thereto and having a dependency relationship with the sub-computing task. It should be understood that, if the dependency relationship is the first dependency relationship, the sub-communication task depends on the execution result of the corresponding sub-computing task, and if the dependency relationship is the second dependency relationship, the sub-computing task depends on the execution result of the sub-communication task.

[0160] In some embodiments, the number of the plurality of sub-computing tasks and the number of the plurality of sub-communication tasks are both a first number, illustratively, the distributed parallel computation strategy of the AI model includes the first number, that is, the first number is a pre-set fixed parameter (or referred to as a split number, a split quantity, etc.). In some embodiments, for the distributed computation task of the AI model, the first number corresponding to each dependent computing task and communication task segment can be the same or different, which is not limited in the present application.

[0161] In some embodiments, the number of the plurality of sub-computing tasks and the number of the plurality of sub-communication tasks are both a second number, the second number is determined based on reference performance analysis results corresponding to the computing task and the communication task, wherein the reference performance analysis results are also referred to as Profiling results, and the application does not limit the manner of obtaining the reference performance analysis results. Illustratively, before determining the plurality of sub-computing tasks of the computing task and the plurality of sub-communication tasks of the communication task, the host determines, based on the dependency relationship and at least one reference number, a reference execution strategy corresponding to each reference number, the reference execution strategy indicating that the acceleration device executes the plurality of sub-computing tasks of the computing task and the plurality of sub-communication tasks of the communication task determined based on the corresponding reference number according to the dependency relationship; then, the host determines the second number meeting the condition from the at least one reference number based on the reference performance analysis result corresponding to each reference execution strategy, and for any one reference performance analysis result, the reference performance analysis result indicates the performance overhead of the acceleration device executing the computing task and the communication task according to the corresponding reference execution strategy.

[0162] For example, after performing the foregoing step 901, the host determines a reference execution strategy corresponding to each of the plurality of reference numbers 2, 4, and 8. For example, the reference execution strategy corresponding to the reference number 2 is to split the computing task into two sub-computing tasks and split the communication task into two sub-communication tasks, and then send them to the acceleration device for execution. The reference execution strategies corresponding to the remaining reference numbers are similar and will not be described again. Then, the host evaluates the performance overhead of the acceleration device executing the tasks according to each reference execution strategy, and obtains a reference performance analysis result corresponding to each reference execution strategy. For example, the reference performance analysis results indicate that the performance scores corresponding to the reference number 2, the reference number 4, and the reference number 8 are 50 points, 90 points, and 70 points, respectively. Then, the host determines the reference number 4 as the second number meeting the condition, and splits the computing task and the communication task according to the number 4 in this step. It should be understood that the above is only an example, and the specific content of the condition for determining the second number can be set according to actual needs. Moreover, the host can obtain the reference performance analysis results uniformly after the acceleration device executes according to the plurality of reference execution strategies, or can obtain the reference performance analysis result after the acceleration device executes according to each reference execution strategy, and then adjust the next reference number based on the reference performance analysis result, and finally select the second number meeting the condition. The application does not limit this.

[0163] In addition, the process of determining the second quantity according to the reference performance analysis result can be understood as a process of adaptively adjusting the number of sub-tasks. In the scenario where the distributed computing task of the AI model is a distributed training task of the AI model, this process can be performed in the warm-up stage of model training. For example, the host controls the acceleration device to perform corresponding computing tasks and communication tasks based on multiple sample batches and different reference execution strategies. One sample batch corresponds to one reference execution strategy, and the present application does not limit this. In other words, the number of sub-tasks is an adjustable hyperparameter, which can be determined according to the performance analysis result in different scenarios, so as to improve the overall task execution efficiency.

[0164] 903. The host sends a first sub-computing task in the plurality of sub-computing tasks and a first sub-communication task in the plurality of sub-communication tasks to the acceleration device, and the first sub-computing task and the first sub-communication task have a dependency relationship.

[0165] In the embodiments of the present application, the first sub-computing task is the sub-computing task with the first execution order in the plurality of sub-computing tasks, and the first sub-communication task is the sub-communication task with the first execution order in the plurality of sub-communication tasks. The dependency relationship between the first sub-computing task and the first sub-communication task is the same as the dependency relationship between the computing task and the communication task. The host can send the first sub-computing task and the first sub-communication task to the acceleration device synchronously, or send the first sub-computing task and the first sub-communication task to the acceleration device according to the execution order indicated by the dependency relationship, and the present application does not limit this.

[0166] 904. The acceleration device executes the first sub-computing task and the first sub-communication task according to the dependency relationship.

[0167] In the embodiments of the present application, if the dependency relationship is the first dependency relationship, the acceleration device executes the first sub-computing task first, and executes the first sub-communication task based on the execution result of the first sub-computing task; if the dependency relationship is the second dependency relationship, the acceleration device executes the first sub-communication task first, and executes the first sub-computing task based on the execution result of the first sub-communication task.

[0168] In some embodiments, if the shape of the input data of the computing task meets the pre-processing condition, the input data can also be pre-processed before the acceleration device performs this step, so that the acceleration device performs the first sub-computing task based on the pre-processed input data. Illustratively, the pre-processing condition is, for example, that the shape of the input data does not meet the dimension size, and accordingly, the input data meeting the pre-processing condition can also be understood as input data of a non-affine shape. By pre-processing the input data, it can be ensured that the acceleration device continuously carries out aligned addresses, wherein the pre-processing can be at least one of padding and transpose, which is not limited in the present application, and the appropriate pre-processing method can be determined according to the business requirements. For the implementation of padding and transpose, reference can be made to the foregoing description, which will not be repeated here.

[0169] In some embodiments, the host pre-processes the input data and sends the pre-processed input data to the acceleration device, or the host sends a pre-processing task to the acceleration device, and the acceleration device pre-processes the input data based on the pre-processing task, that is, both the host and the acceleration device can perform pre-processing, which is not limited in the present application. In addition, since the input data is pre-processed before the acceleration device performs the first sub-computing task in the present application, the acceleration device can perform subsequent sub-computing tasks based on the pre-processed input data, avoiding redundant computation. For example, taking a Matmul computing task as an example, after dividing a Matmul computing task into multiple fine-grained Matmul sub-tasks, theoretically, each fine-grained Matmul sub-task needs to repeat the pre-processing. However, by using the above method, the pre-processing is performed before the acceleration device performs the sub-computing task, and then each sub-task can be further performed on the pre-processed result. In this way, redundant computation of pre-processing is avoided, and computing resources are saved.

[0170] The above steps 902 to 904 introduce the process of controlling the acceleration device by the host to perform the first sub-computing task and the first sub-communication task. It should be understood that the host divides the computing task into multiple sub-computing tasks and divides the communication task into multiple sub-communication tasks. As can be known from the foregoing description of FIGS. 6 and 7, since the host performs fine-grained division on the computing task and the communication task, part of the communication time can be masked by reasonable task scheduling and parallel execution.

[0171] Illustratively, based on the above steps 902 to 904, the host further performs the following steps:

[0172] The second sub-computing task and the second sub-communication task are sent to the acceleration device in a synchronous manner or in an order indicated by the dependency relationship. Correspondingly, if the dependency relationship is the first dependency relationship, the acceleration device executes the first sub-computing task and the first sub-communication task according to the dependency relationship, including: the acceleration device executes the first sub-communication task and the second sub-computing task in parallel after executing the first sub-computing task, and executes the second sub-communication task after executing the second sub-computing task; if the dependency relationship is the second dependency relationship, the acceleration device executes the first sub-computing task and the first sub-communication task according to the dependency relationship, including: the acceleration device executes the first sub-computing task and the second sub-communication task in parallel after executing the first sub-communication task, and executes the second sub-computing task after executing the second sub-communication task. It should be understood that, for the sub-computing tasks after the second sub-computing task and the sub-communication tasks after the second sub-communication task, the execution of the computing tasks and the communication tasks is completed by executing the tasks in the manner as described above, and the overall task execution efficiency is improved.

[0173] Referring to FIGS. 10 and 11, the task execution method shown in FIG. 9 is schematically described by taking the first dependency relationship and the second dependency relationship as examples, respectively.

[0174] FIG. 10 is a flowchart of another task execution method in an AI model distributed computing process according to an embodiment of the present application. As shown in FIG. 10, the task execution method is applied to a distributed computing system including a host and an acceleration device. Taking the interaction between the host and the acceleration device as an example, the task execution method includes the following steps 1001 to 1008.

[0175] 1001. The host determines the computing task and the communication task to be executed by the acceleration device based on the distributed parallel computing strategy of the AI model, and the first dependency relationship exists between the computing task and the communication task.

[0176] This step is the same as the step 901 described above, and thus is not described again. The first dependency relationship is that the communication task depends on the execution result of the computing task.

[0177] 1002. The host determines the plurality of sub-computing tasks of the computing task and the plurality of sub-communication tasks of the communication task based on the first dependency relationship.

[0178] Wherein, taking the first quantity equal to 2 as an example, the host controls the acceleration device to execute each subtask according to the following steps 1003 to 1008.

[0179] 1003. The host sends a first sub-computing task to the acceleration device.

[0180] 1004. The acceleration device executes the first sub-computing task.

[0181] 1005. The host sends a second sub-computing task and a first sub-communication task to the acceleration device.

[0182] 1006. The acceleration device executes the second sub-computing task and the first sub-communication task in parallel.

[0183] Wherein, the acceleration device executes the first sub-communication task based on the execution result of the first sub-computing task according to the first dependency relationship, and executes the second sub-computing task in parallel on this basis.

[0184] 1007. The host sends a second sub-communication task to the acceleration device.

[0185] 1008. The acceleration device executes the second sub-communication task.

[0186] Wherein, the acceleration device executes the second sub-communication task based on the execution result of the second sub-computing task according to the first dependency relationship.

[0187] It should be noted that the application does not limit the timing of the host sending each subtask to the acceleration device, and in actual application, the to-be-executed subtask can be issued according to the execution of the issued subtask.

[0188] FIG. 11 is a flowchart of another task execution method in the AI model distributed computing process according to an embodiment of the application. As shown in FIG. 11, the task execution method is applied to a distributed computing system including a host and an acceleration device, and taking the interaction between the host and the acceleration device as an example, the task execution method includes the following steps 1101 to 1108.

[0189] 1101. The host determines the computing task and the communication task to be executed by the acceleration device based on the distributed parallel computing strategy of the AI model, and the computing task and the communication task have a second dependency relationship.

[0190] This step is the same as the aforementioned step 901, and thus will not be described again. The second dependency relationship is that the computing task depends on the execution result of the communication task.

[0191] 1102. The host determines a plurality of sub-computing tasks of the computing task and a plurality of sub-communication tasks of the communication task based on the second dependency relationship.

[0192] The host controls the acceleration device to execute each subtask according to the first dependency relationship, and the first subtask and the second subtask are executed in parallel.

[0193] 1103. The host sends a first sub-communication task to the acceleration device.

[0194] 1104. The acceleration device executes the first sub-communication task.

[0195] 1105. The host sends a second sub-communication task and a first sub-computation task to the acceleration device.

[0196] 1106. The acceleration device executes the second sub-communication task and the first sub-computation task in parallel.

[0197] The acceleration device executes the first sub-computation task based on the execution result of the first sub-communication task according to the second dependency relationship, and executes the second sub-communication task in parallel on this basis.

[0198] 1107. The host sends a second sub-computation task to the acceleration device.

[0199] 1108. The acceleration device executes the second sub-computation task.

[0200] The acceleration device executes the second sub-computation task based on the execution result of the second sub-communication task according to the second dependency relationship.

[0201] It should be noted that the application does not limit the timing of the host sending each subtask to the acceleration device. In actual application, the to-be-executed subtask can be issued according to the execution of the issued subtask.

[0202] As can be seen from the above task execution methods shown in FIGS. 6 to 11, in the embodiment of the application, when the distributed computing system executes the distributed computing task, for the computation task and the communication task to be executed by the acceleration device, if there is a dependency relationship between the computation task and the communication task, the computation task and the communication task can be split into multiple pairs of subtasks with the same dependency relationship according to the dependency relationship. In this way, the host can mask the time overhead of part of the subtasks by reasonably scheduling these subtasks, thereby improving the task execution efficiency. Moreover, this method is suitable for various computation task and communication task fragments under different dependency relationships, has universality and scalability, and can support different network topology conditions.

[0203] In addition, in the task execution method shown in FIGS. 9-11, the computing task and the communication task to be executed by the acceleration device are divided. In some embodiments, taking the distributed training task of the AI model as an example of the distributed computing task, for a part of the distributed training task, if the data processing amount of the computing task or the communication task itself is small, dividing the task into fine-grained tasks for parallel processing may result in excessive overhead at the beginning and end or insufficient communication amount, leading to insufficient bandwidth utilization and ultimately resulting in a decrease in overall performance. Therefore, in this case, the host can not perform fine-grained division (i.e., execute the original code), or reduce the number of fine-grained tasks to ensure optimal overall performance. Based on this, before executing the foregoing task execution method, the host can evaluate the performance overhead of the computing task and the communication task to be executed by the acceleration device under the two task execution strategies of division and no division, and then determine whether to adopt the foregoing task execution method.

[0204] Illustratively, before determining the computing task and the communication task to be executed by the acceleration device based on the distributed parallel computing strategy of the AI model, the host evaluates the performance overhead of the acceleration device executing the computing task and the communication task according to the first task execution strategy and the second task execution strategy based on the distributed parallel computing strategy, to obtain a first performance analysis result corresponding to the first task execution strategy and a second performance analysis result corresponding to the second task execution strategy. The first task execution strategy indicates executing the divided computing task and communication task after dividing the computing task and the communication task, and the second task execution strategy indicates executing the computing task and the communication task without dividing the computing task and the communication task. It should be understood that the number of the first task execution strategy can be one or multiple, and the same applies to the reference execution strategies corresponding to different reference numbers in the foregoing step 902, which will not be described again.

[0205] In some embodiments, if the performance overhead indicated by the first performance analysis result is less than the performance overhead indicated by the second performance analysis result, the step of determining the computing task and the communication task based on the distributed parallel computing strategy is executed according to the first task execution strategy. That is, the task execution method shown in the foregoing steps 901-904 is executed. In other words, by evaluating the performance overhead corresponding to different task execution strategies, the performance analysis result shows that dividing the computing task and the communication task can significantly reduce the performance overhead and improve the task execution efficiency compared to not dividing the computing task and the communication task, based on which the host executes the foregoing task execution method.

[0206] In some embodiments, if the performance overhead indicated by the first performance analysis result is greater than or equal to the performance overhead indicated by the second performance analysis result, the host feeds back the first performance analysis result and the second performance analysis result to the user, and controls the acceleration device to perform the computing task and the communication task based on the task execution strategy indicated by the user. In other words, by evaluating the performance overhead corresponding to different task execution strategies, the performance analysis result indicates that the performance overhead of splitting the computing task and the communication task is similar to or not much different from that of not splitting the computing task and the communication task, and based on this, the performance analysis result is fed back to the user, and the user decides which task execution strategy to use, so as to meet the individual needs of the user.

[0207] Of course, in some embodiments, if the performance overhead indicated by the first performance analysis result is greater than or equal to the performance overhead indicated by the second performance analysis result, the host can also control the acceleration device to perform the task according to the task execution strategy set by the user in advance, or the host automatically selects the second task execution strategy, etc., which is not limited in the present application.

[0208] In addition, the above method splits the computing task and the communication task to be performed by the acceleration device, that is, the functions of the original computing task and the communication task are implemented by the plurality of sub-tasks after splitting. However, in some scenarios, the execution result of performing one communication task is not completely equivalent to the execution result of performing a plurality of sub-communication tasks.

[0209] For example, taking the Allgather communication task as an example, referring to FIG. 12, FIG. 12 is a schematic diagram of a communication task provided in an embodiment of the present application. As shown in FIG. 12, taking a distributed computing system including a host and two acceleration devices, referred to as Device0 and Device1, as an example, after the host controls Device0 to execute an Allgather communication task once, the execution result of the communication task is arranged in the memory of Device0 as [Device0 part-0, Device0 part-1, Device0 part-2, Device0 part-3, Device1 part-0, Device1 part-1, Device1 part-2, Device1 part-3, …]. If the Allgather communication task is split into four sub-communication tasks, referred to as Allgather0, Allgather1, Allgather2 and Allgather3, then Device0 needs to merge the execution results of the four sub-communication tasks, for example, by calling a concat operator, as shown in the figure. At this time, the memory arrangement of the execution results of the four sub-communication tasks is [Device0 part-0, Device1 part-0, Device0 part-1, Device1 part-1, Device0 part-2, Device1 part-2, Device0 part-3, Device1 part-3, …], and it can be seen that this memory arrangement is inconsistent with the memory arrangement corresponding to the execution of the communication task once. Therefore, the merged execution result needs to be rearranged in memory to obtain a memory arrangement equivalent to the Allgather communication task. It should be noted that if the View Tensor method is used to allocate the memory required to store the complete communication task execution result in advance, the execution result of the sub-communication task can be directly written to the correct position, and at this time, the merging step does not need to be performed.

[0210] For example, taking ReduceScatter communication task as an example, referring to FIG. 13, FIG. 13 is a schematic diagram of another communication task provided by an embodiment of the present application. As shown in FIG. 13, taking a distributed computing system including a host and two acceleration devices, referred to as Device0 and Device1, as an example, before the host controls Device0 to perform a ReduceScatter communication task, the input data of the ReduceScatter communication task is arranged in the memory of Device0 as [Device0 part-0, Device0 part-1, Device0 part-2, Device0 part-3, Device1 part-0, Device1 part-1, Device1 part-2, Device1 part-3, …]. If the ReduceScatter communication task is split into four sub-communication tasks, referred to as ReduceScatter0, ReduceScatter1, ReduceScatter2 and ReduceScatter3, then the host needs to rearrange the input data of the sub-communication tasks in the memory before controlling Device0 to perform the four sub-communication tasks, and then the host needs to issue the four sub-communication tasks, and Device0 merges the execution results of the four sub-communication tasks, for example, by calling the concat operator, to obtain the memory arrangement equivalent to the execution result of the ReduceScatter communication task. It should be noted that if the View Tensor is used to allocate the memory required to store the execution result of the complete communication task in advance, the execution result of the sub-communication task can be directly written to the correct position, and in this case, the merging step is not required.

[0211] Based on the scenarios shown in FIGS. 12 and 13, the present application provides two processing methods for communication tasks based on the task execution method shown in FIGS. 6 to 11, which can ensure that the execution results before and after the communication task is split are equivalent, or in other words, to ensure the accuracy consistency of different operators in the calculation process. The two methods will be introduced below.

[0212] Implementation method one, memory rearrangement task.

[0213] The memory rearrangement task is issued by the host to the acceleration device, and is used to ensure that the execution results before and after the communication task is split are equivalent. Illustratively, the present application provides different memory rearrangement tasks for different types of communication tasks, including the following two cases:

[0214] Case one, the communication task belongs to the first task type, the first task type indicates that the communication task is a collection type task, such as an Allgather communication task. Illustratively, after the acceleration device executes a plurality of sub-communication tasks, the host controls the acceleration device to execute a first memory rearrangement task, the first memory rearrangement task indicates that the execution results of the plurality of sub-communication tasks are rearranged in memory, and after the first memory rearrangement task is executed, the execution results of the plurality of sub-communication tasks are equivalent to the execution results of the communication task.

[0215] In some embodiments, the first memory rearrangement task is implemented based on a transpose operator, which can improve the efficiency of memory rearrangement. For example, taking the Allgather communication task as an example, the way to implement the first memory rearrangement task based on the transpose operator is as follows:

[0216] “def shuffle_for_all_gather(input_, world_size, parallel_num): / / defines a function for rearranging the input tensor to adapt to the all_gather operation

[0217] #input_ represents the input tensor Tensor, world_size represents the number of acceleration devices, and parallel_num represents the number of partitions, i.e. the number of sub-communication tasks

[0218] per = input_.shape[0] / / parallel_num / / world_size / / determine the length of each sub-part

[0219] input_shape = list(input_.shape) / / convert the shape of the input tensor to a list form

[0220] reshape_tensor = torch.reshape(input_, [parallel_num, world_size, per] + input_shape[1:]) / / reshape the input tensor according to the calculated parameters to parallel_num rows, world_size columns, and per length per row, and then add the original dimension information except the first dimension

[0221] return torch.reshape(reshape_tensor.transpose(0, 1), tuple(input_shape)) / / transpose the reshaped tensor in the first and second dimensions, then restore it to the original input tensor shape and return”

[0222] Case two, the communication task belongs to the second task type, the second task type indicates that the communication task is a scatter type task, such as a ReduceScatter communication task. Illustratively, before the acceleration device executes a plurality of sub-communication tasks, the host controls the acceleration device to execute a second memory rearrangement task, the second memory rearrangement task indicating that the input data of the plurality of sub-communication tasks is rearranged in memory, and after the second memory rearrangement task is executed, the execution result of the plurality of sub-communication tasks is equivalent to the execution result of the communication task.

[0223] In some embodiments, the second memory rearrangement task is implemented based on a transpose operator, which can improve the efficiency of memory rearrangement. For example, taking the ReduceScatter communication task as an example, the way to implement the second memory rearrangement task based on the transpose operator is as follows:

[0224] “def shuffle_for_reduce_scatter(input_, world_size, parallel_num): / / defines a function for rearranging the input tensor to adapt to the reduce_scatter operation

[0225] #input_ represents the input tensor Tensor, world_size represents the number of acceleration devices, and parallel_num represents the number of partitions, i.e. the number of sub-communication tasks

[0226] per = input_.shape[0] / / parallel_num / / world_size / / determine the length of each sub-part

[0227] input_shape = list(input_.shape) / / convert the shape of the input tensor to a list form

[0228] reshape_tensor = torch.reshape(input_, [parallel_num, world_size, per] + input_shape[1:]) / / reshape the input tensor according to the calculated parameters to parallel_num rows, world_size columns, and per length per row, and then add the original dimension information except the first dimension

[0229] return torch.reshape(reshape_tensor.transpose(0, 1), tuple(input_shape)) / / transpose the first and second dimensions of the reshaped tensor, then restore it to the original input tensor shape and return”

[0230] It should be noted that the first memory rearrangement task and the second memory rearrangement task can also be implemented in other ways, and any memory rearrangement method for ensuring that the execution results of the communication task before and after splitting are equivalent can be applied to the present application, and the present application does not limit this.

[0231] Implementation mode two, communication interface.

[0232] The communication interface is a communication interface with stride. Stride is also called offset, which refers to the size of the memory address that needs to be skipped when writing 1 / M of the data into the output tensor during communication between M acceleration devices. M is a positive integer. In the case where the distributed computing system supports calling the communication interface, the host controls the acceleration device to call the communication interface to write the execution results of each sub-communication task to the correct memory space, ensuring that the execution results of the communication task before and after splitting are equivalent, so as to avoid the additional overhead caused by memory rearrangement.

[0233] Illustratively, the host controls the acceleration device to call the communication interface, and stores the execution results of each sub-communication task to the memory space corresponding to each sub-communication task based on the offset corresponding to each sub-communication task, wherein the offset is determined based on the input data amount of the corresponding sub-communication task and the number of sub-communication tasks, and the execution results stored in the memory space corresponding to each sub-communication task are equivalent to the execution results of the communication task. For example, for any sub-communication task (denoted as the jth sub-communication task), the offset of the ith portion of data corresponding to the ith acceleration device is determined based on the formula stride=j / L+iQL, wherein 0≤i≤M, L represents the input data amount of the sub-communication task, and Q represents the number of sub-communication tasks. L is determined based on the input data amount (denoted as S) of the communication task and the number Q of sub-communication tasks, i.e. L=S / Q.

[0234] For example, taking the Allgather communication task as an example, referring to FIG. 14, FIG. 14 is a schematic diagram of calling a communication interface provided in an embodiment of the present application. As shown in FIG. 14, taking a distributed computing system including a host and two acceleration devices, referred to as Device0 and Device1, as an example, when the i th acceleration device executes the j th sub-communication task, the communication interface is called, and the execution result of the j th sub-communication task is stored in the corresponding memory space based on the offset stride = j / L+iQL. For example, as shown in (1) of FIG. 14, taking j = 0, Q = 4, and M = 2 as an example, the offset of the 0 th data from Device0 is 0, and the offset of the 0 th data from Device1 is 4L. As shown in (2) of FIG. 14, taking j = 2, Q = 4, and M = 2 as an example, the offset of the 0 th data from Device0 is 2L, and the offset of the 0 th data from Device1 is 6L. It should be noted that FIG. 14 takes part of the data as an example for introduction, and the blank cells represent omissions.

[0235] For example, taking the ReduceScatter communication task as an example, referring to FIG. 15, FIG. 15 is another schematic diagram of calling a communication interface provided in an embodiment of the present application. As shown in FIG. 15, taking a distributed computing system including a host and two acceleration devices, referred to as Device0 and Device1, as an example, when the i th acceleration device executes the j th sub-communication task, the communication interface is called, and the offset stride = j / L+iQL of the i th data used for reduce is determined, and the execution result of the j th sub-communication task is stored in the corresponding memory space. For example, as shown in (1) of FIG. 15, taking j = 0, Q = 4, and M = 2 as an example, the offset of the 0 th data of Device0 used for reduce is 0, and the offset of the 0 th data of Device1 used for reduce is 4L. As shown in (2) of FIG. 15, taking j = 2, Q = 4, and M = 2 as an example, the offset of the 2 nd data of Device0 used for reduce is 2L, and the offset of the 2 nd data of Device1 used for reduce is 6L. It should be noted that FIG. 15 takes part of the data as an example for introduction, and the blank cells represent omissions.

[0236] The task execution method provided in the present application is described in detail through the foregoing contents shown in FIGS. 6-15. In some embodiments, the host can adaptively determine the split number of a task segment with dependency in a distributed computing task according to performance analysis results, determine whether preprocessing is needed according to the shape of the input data of the computing task, determine whether memory rearrangement is needed according to whether the execution results before and after communication task splitting are equivalent, and the like. Referring to FIG. 16, another task execution method provided in the present application is introduced.

[0237] FIG. 16 is a schematic diagram of a framework of a task execution method in an AI model distributed computing process according to an embodiment of the present application. As shown in FIG. 16, the task execution method is applied to a distributed computing system, which includes the following stages:

[0238] Stage 1, in response to an execution request of an AI model initiated by a computing application, the host creates a distributed computing task, determines a plurality of acceleration devices participating in the distributed computing task, and determines the computing task and the communication task to be executed on each acceleration device. During the execution of the task pipeline of the distributed computing task, for a group of computing tasks and communication tasks with a dependency relationship, i.e., a task segment with a dependency, the host can determine whether to perform fine-grained segmentation on the current task segment according to a pre-set rule or performance analysis result. It should be understood that, taking the distributed training task of the AI model as an example, for part of the task segment of the distributed training task, if the data processing amount of the computing or communication task itself is small, the parallel processing of the fine-grained task may result in excessive overhead at the beginning and end or insufficient communication amount, which may lead to insufficient bandwidth utilization and ultimately result in overall performance degradation. Therefore, in this case, the host can not perform fine-grained segmentation (i.e., execute the original code), or reduce the number of fine-grained tasks to ensure optimal overall performance. In addition, based on the foregoing 902, the number of fine-grained tasks is a tunable hyperparameter, which can be determined according to the performance analysis result in different scenarios to determine the optimal number of segmentation, which will not be described here.

[0239] Stage 2, for the task segment to be executed by fine-grained segmentation, the host executes the corresponding scheduling logic according to the dependency relationship of the task segment. As known from the foregoing, the dependency relationship is divided into a first dependency relationship and a second dependency relationship, wherein the first dependency relationship refers to that the communication task depends on the execution result of the computing task, and correspondingly, the first dependency relationship corresponds to the “first computing and then communication script logic”, and the scheduling code will ensure that the communication task is executed after the completion of the computing task, such as the Matmul+Allreduce task segment, the Matmul+ReduceScatter task segment, etc. The second dependency relationship refers to that the computing task depends on the execution result of the communication task, and correspondingly, the second dependency relationship corresponds to the “first communication and then computing script logic”, and the scheduling code will ensure that the computing task is executed after the completion of the communication task, such as the AllGather+Matmul task segment, the All2All+Matmul segment, etc. In addition, both of the two scheduling codes use an asynchronous communication interface, and add a synchronization operation at a suitable position to ensure the correctness of the data and the efficiency of the concurrency. Corresponding to the reasonable scheduling process of the plurality of sub-computing tasks and the plurality of sub-communication tasks, which will not be described here.

[0240] Stage 3, after entering the "compute first communication script logic" or "communication first compute script logic", the host pre-processes the input data of the computing task according to whether the shape of the input data meets the pre-processing condition, so as to control the acceleration device to perform a plurality of sub-computing tasks based on the pre-processed input data in the subsequent process. This process is described in the foregoing step 904 and will not be repeated here.

[0241] Stage 4, the host determines whether memory rearrangement is needed according to the task type of the communication task in the current task segment, and if needed, selects the corresponding processing mode according to whether the distributed computing system supports calling the communication interface with stride. If the distributed computing system does not support calling the communication interface with stride, the host controls the acceleration device to perform the memory rearrangement task as described above; if the distributed computing system supports calling the communication interface with stride, the host controls the acceleration device to call the communication interface, and stores the execution result of each sub-communication task to the memory space corresponding to each sub-communication task based on the offset corresponding to each sub-communication task.

[0242] Stage 5, after the host controls the acceleration device to perform a plurality of sub-computing tasks and a plurality of sub-communication tasks, the performance of the current splitting strategy is analyzed, and the performance benefit is evaluated, and the number of splitting parts, i.e. the number of sub-computing tasks and sub-communication tasks, is adjusted according to the performance benefit. In some scenarios, the pre-processing condition can also be adaptively adjusted according to the performance analysis result, so as to maximize the performance benefit. This dynamic adjustment strategy makes the task execution method provided by the present application adaptable to various complex scenarios, has high flexibility, and can realize efficient computing and communication parallel processing.

[0243] The following takes several task segments as examples to illustrate the task execution method provided by the present application.

[0244] The first one is Matmul+Allreduce task segment.

[0245] The Matmul+Allreduce task fragment includes a Matmul computing task and an Allreduce communication task, and the dependency relationship between the computing task and the communication task is a first dependency relationship. Based on this, the host determines a plurality of sub-computing tasks of the Matmul computing task and a plurality of sub-communication tasks of the Allreduce communication task, and by reasonably scheduling these sub-tasks, the time overhead of part of the sub-tasks is masked. For example, referring to FIG. 17, FIG. 17 is a schematic diagram of concurrently executing sub-tasks provided by an embodiment of the present application. As shown in FIG. 17, the plurality of sub-computing tasks of the Matmul computing task are respectively denoted as Matmul0, Matmul1, Matmul2 and Matmul3, and the plurality of sub-communication tasks of the Allreduce communication task are respectively denoted as Allreduce0, Allreduce1, Allreduce2 and Allreduce3. FIG. 17 shows schematic diagrams of concurrently executing each sub-task in three cases of a computing time being greater than a communication time, the computing time being equal to the communication time, and the computing time being less than the communication time. It can be seen that the computing task and the communication task are split into a plurality of pairs of sub-tasks having the same dependency relationship, and thus the host can mask the time overhead of part of the sub-tasks by reasonably scheduling these sub-tasks, thereby improving the task execution efficiency.

[0246] In some embodiments, if the shape of the input data of the Matmul computing task meets the preprocessing condition, the input data is preprocessed, and then the acceleration device is controlled to execute a plurality of sub-computing tasks based on the preprocessed input data. For example, since the right matrix used by each sub-computing task is the same, the right matrix in the input data of the Matmul computing task can be preprocessed, so that the acceleration device can execute each sub-computing task based on the preprocessed input data, avoid redundant computation, and improve the overall task execution efficiency. The schematic diagram of concurrent execution in this scenario can be referred to FIG. 18, as shown in FIG. 18, the input data is preprocessed before Matmul0, Matmul1, Matmul2 and Matmul3 are executed.

[0247] In addition, for the Allreduce communication task, since the execution result of the plurality of sub-communication tasks can be obtained after the host combines the execution results of the plurality of sub-communication tasks, which is equivalent to the execution result of the Allreduce communication task, therefore, when the task fragment is executed, there is no need to perform memory rearrangement.

[0248] The second, Allgather+Matmul task fragment.

[0249] The Allgather+Matmul task fragment includes an Allgather communication task and a Matmul calculation task, and the dependency relationship between the calculation task and the communication task is a second dependency relationship. Based on this, the host determines a plurality of sub-calculations of the Matmul calculation task and a plurality of sub-communications of the Allgather communication task, and by reasonably scheduling these sub-tasks, the time overhead of part of the sub-tasks is masked. For example, referring to FIG. 19, FIG. 19 is a schematic diagram of another concurrent execution of sub-tasks provided by an embodiment of the present application. As shown in FIG. 19, the plurality of sub-calculations of the Matmul calculation task are respectively denoted as Matmul0, Matmul1, Matmul2, and Matmul3, and the plurality of sub-communications of the Allgather communication task are respectively denoted as Allgather0, Allgather1, Allgather2, and Allgather3. FIG. 19 shows a schematic diagram of concurrent execution of each sub-task in three cases: the calculation time is greater than the communication time, the calculation time is equal to the communication time, and the calculation time is less than the communication time. As can be seen, the calculation task and the communication task are split into a plurality of pairs of sub-tasks that have the same dependency relationship. In this way, the host can mask the time overhead of part of the sub-tasks by reasonably scheduling these sub-tasks, thereby improving the task execution efficiency.

[0250] In some embodiments, if the shape of the input data of the Matmul calculation task meets the preprocessing condition, the input data is preprocessed, and then the acceleration device is controlled to execute a plurality of sub-calculations based on the preprocessed input data. For example, since the right matrix used by each sub-calculation is the same, the right matrix in the input data of the Matmul calculation task can be preprocessed. In this way, the acceleration device can execute each sub-calculation based on the preprocessed input data, avoid redundant calculation, and improve the overall task execution efficiency. The schematic diagram of concurrent execution in this scenario can be referred to FIG. 20, as shown in FIG. 20, the input data is preprocessed before Matmul0, Matmul1, Matmul2, and Matmul3 are executed.

[0251] In addition, for the Allgather communication task, it can be known from the foregoing content shown in FIG. 12 that the execution result obtained by executing one Allgather communication task is not completely equivalent to the execution result obtained by executing a plurality of sub-communications, and therefore memory rearrangement needs to be performed when the task fragment is executed. For example, the memory rearrangement task is implemented by execution, or the communication interface is called, which will not be described herein again.

[0252] The third, Matmul+ReduceScatter task fragment.

[0253] The Matmul+ReduceScatter task fragment includes a Matmul computing task and a ReduceScatter communication task, and the dependency relationship between the computing task and the communication task is a first dependency relationship. Based on this, the host determines a plurality of sub-computing tasks of the Matmul computing task and a plurality of sub-communication tasks of the ReduceScatter communication task, and by reasonably scheduling these sub-tasks, the time overhead of part of the sub-tasks is masked. For example, referring to FIG. 21, FIG. 21 is a schematic diagram of concurrently executing sub-tasks according to another embodiment of the present application. As shown in FIG. 21, the plurality of sub-computing tasks of the Matmul computing task are denoted as Matmul0, Matmul1, Matmul2, and Matmul3 respectively, and the plurality of sub-communication tasks of the ReduceScatter communication task are denoted as ReduceScatter0, ReduceScatter1, ReduceScatter2, and ReduceScatter3 respectively. FIG. 21 shows schematic diagrams of concurrently executing each sub-task in three cases: the computing time is greater than the communication time, the computing time is equal to the communication time, and the computing time is less than the communication time. As can be seen, the computing task and the communication task are split into a plurality of pairs of sub-tasks that have the same dependency relationship. In this way, the host can mask the time overhead of part of the sub-tasks by reasonably scheduling these sub-tasks, thereby improving the task execution efficiency.

[0254] In some embodiments, if the shape of the input data of the Matmul computing task meets the preprocessing condition, the input data is preprocessed, and then the acceleration device is controlled to execute a plurality of sub-computing tasks based on the preprocessed input data. For example, since the right matrix used by each sub-computing task is the same, the right matrix in the input data of the Matmul computing task can be preprocessed. In this way, the acceleration device can execute each sub-computing task based on the preprocessed input data, avoid redundant computation, and improve the overall task execution efficiency. The schematic diagram of concurrent execution in this scenario can be referred to FIG. 22. As shown in FIG. 22, the input data is preprocessed before Matmul0, Matmul1, Matmul2, and Matmul3 are executed.

[0255] In addition, for the ReduceScatter communication task, it can be known from the foregoing content shown in FIG. 13 that the execution result obtained by executing one ReduceScatter communication task is not completely equivalent to the execution result obtained by executing a plurality of sub-communication tasks, and therefore memory rearrangement needs to be performed when the task fragment is executed. For example, the memory rearrangement task is executed, or a communication interface is called, which will not be described herein again.

[0256] It should be understood that the task execution method corresponding to the above several task fragments is only for illustration, and the task execution method provided by the application can be applied to any task fragment with a dependency relationship in a distributed computing task, has universality and scalability, and can support different network topology cases.

[0257] Based on the foregoing task execution method, the application further provides a task execution device in an AI model distributed computing process, which can be configured in a host in a distributed computing system. The device can realize part or all of the functions of the host through software, hardware, or a combination of both. Illustratively, referring to FIG. 23, the device includes a task determination unit 2301, a sub-task determination unit 2302, and a task sending unit 2303.

[0258] The task determination unit 2301 is configured to determine, based on a distributed parallel computing strategy of an AI model, a computing task and a communication task to be executed by an acceleration device.

[0259] The sub-task determination unit 2302 is configured to determine, based on a dependency relationship between the computing task and the communication task, a plurality of sub-computing tasks of the computing task and a plurality of sub-communication tasks of the communication task. The dependency relationship indicates that the computing task depends on an execution result of the communication task, or the communication task depends on an execution result of the computing task.

[0260] The task sending unit 2303 is configured to send, to the acceleration device, a first sub-computing task in the plurality of sub-computing tasks and a first sub-communication task in the plurality of sub-communication tasks. The first sub-computing task and the first sub-communication task have a dependency relationship.

[0261] In some embodiments, the dependency relationship is a first dependency relationship, and the first dependency relationship indicates that the communication task depends on the execution result of the computing task. The task sending unit 2303 is further configured to:

[0262] send, to the acceleration device, a second sub-computing task in the plurality of sub-computing tasks and a second sub-communication task in the plurality of sub-communication tasks. The second sub-computing task and the second sub-communication task have a dependency relationship.

[0263] The acceleration device is configured to execute the first sub-communication task and the second sub-computing task in parallel after executing the first sub-computing task, and execute the second sub-communication task after executing the second sub-computing task.

[0264] In some embodiments, the dependency relationship is a second dependency relationship, and the second dependency relationship indicates that the computing task depends on the execution result of the communication task. The task sending unit 2303 is further configured to:

[0265] send, to the acceleration device, a second sub-computing task in the plurality of sub-computing tasks and a second sub-communication task in the plurality of sub-communication tasks, the second sub-computing task and the second sub-communication task having a dependency relationship therebetween;

[0266] wherein the acceleration device is configured to execute the first sub-communication task first, and then execute the first sub-computing task and the second sub-communication task in parallel, and then execute the second sub-computing task after the execution of the second sub-communication task.

[0267] In some embodiments, the AI model comprises a plurality of network layers, and at least one network layer is executed on the acceleration device; wherein,

[0268] If the dependency relationship is a first dependency relationship, the first dependency relationship refers to that a communication task depends on an execution result of a computing task, the computing task instructs to perform computation on at least one network layer, and the communication task instructs to exchange data with an acceleration device other than the acceleration device in the system based on output data of the at least one network layer.

[0269] If the dependency relationship is a second dependency relationship, the second dependency relationship refers to that a computing task depends on an execution result of a communication task, the communication task instructs to exchange data with an acceleration device other than the acceleration device in the system to obtain input data of the at least one network layer, and the computing task instructs to perform computation on the at least one network layer based on the input data of the at least one network layer.

[0270] In some embodiments, based on the distributed parallel computing strategy of the AI model, before determining the computing task and the communication task to be executed by the acceleration device, the apparatus further comprises:

[0271] a performance analysis unit configured to evaluate performance overheads of the acceleration device executing the computing task and the communication task according to a first task execution strategy and a second task execution strategy respectively, to obtain a first performance analysis result corresponding to the first task execution strategy and a second performance analysis result corresponding to the second task execution strategy, the first task execution strategy instructing to execute the computing task and the communication task after the computing task and the communication task are split respectively, and the second task execution strategy instructing to execute the computing task and the communication task without splitting the computing task and the communication task.

[0272] wherein, if the performance overhead indicated by the first performance analysis result is less than the performance overhead indicated by the second performance analysis result, the step of determining the computing task and the communication task based on the distributed parallel computing strategy is executed according to the first task execution strategy; and if the performance overhead indicated by the first performance analysis result is greater than or equal to the performance overhead indicated by the second performance analysis result, the first performance analysis result and the second performance analysis result are fed back to a user, and the acceleration device is controlled to execute the computing task and the communication task based on a task execution strategy indicated by the user.

[0273] In some embodiments, the number of the plurality of sub-computing tasks and the number of the plurality of sub-communication tasks are both the first number, and the distributed parallel computing strategy comprises the first number.

[0274] In some embodiments, the number of the plurality of sub-computing tasks and the number of the plurality of sub-communication tasks are both the second number, and the second number is determined based on reference performance analysis results corresponding to the computing task and the communication task.

[0275] In some embodiments, before determining the plurality of sub-computing tasks of the computing task and the plurality of sub-communication tasks of the communication task, the apparatus further comprises a number determining unit configured to:

[0276] determine, based on the dependency relationship and the at least one reference number, a reference execution strategy corresponding to each reference number, the reference execution strategy indicating that the plurality of sub-computing tasks and the plurality of sub-communication tasks of the computing task and the communication task determined based on the corresponding reference number are executed by the acceleration device according to the dependency relationship;

[0277] determine, from the at least one reference number, a second number meeting a condition based on reference performance analysis results corresponding to each reference execution strategy, the reference performance analysis result indicating a performance overhead of the acceleration device when executing the computing task and the communication task according to the corresponding reference execution strategy.

[0278] In some embodiments, before sending the first sub-computing task to the acceleration device, the task sending unit 2303 is further configured to:

[0279] preprocess the input data of the computing task, and send the preprocessed input data to the acceleration device; or

[0280] send a preprocessing task to the acceleration device, and the acceleration device preprocesses the input data based on the preprocessing task.

[0281] In some embodiments, the apparatus further comprises a memory rearrangement unit configured to perform any of the following:

[0282] if the communication task belongs to a first task type, the first task type indicating that the communication task is a collection task, the memory rearrangement unit is configured to control the acceleration device to execute a first memory rearrangement task after the execution of the plurality of sub-communication tasks, the first memory rearrangement task indicating that the execution results of the plurality of sub-communication tasks are rearranged in memory, and the execution results of the plurality of sub-communication tasks are equivalent to the execution result of the communication task after the execution of the first memory rearrangement task; or

[0283] If the communication task belongs to the second task type, the control acceleration device controls the acceleration device to perform a second memory rearrangement task before performing the plurality of sub-communication tasks, the second task type indicates that the communication task is a scatter type task, the second memory rearrangement task indicates that the input data of the plurality of sub-communication tasks is rearranged in memory, and after the second memory rearrangement task is performed, the execution result of the plurality of sub-communication tasks is equivalent to the execution result of the communication task.

[0284] In some embodiments, the first memory rearrangement task and the second memory rearrangement task are both implemented based on a transpose operator.

[0285] In some embodiments, the apparatus further comprises a control unit configured to:

[0286] The control acceleration device calls the communication interface and stores the execution result of each sub-communication task in the memory space corresponding to each sub-communication task based on the offset corresponding to each sub-communication task, wherein the offset is determined based on the input data amount of the corresponding sub-communication task and the number of the plurality of sub-communication tasks, and the execution result stored in the memory space corresponding to each sub-communication task is equivalent to the execution result of the communication task.

[0287] Of course, the apparatus can also include other functional units to implement the functions of the host involved in the above method embodiments. In actual application, the above functions can be completed by different functional units according to needs, that is, the internal structure of the apparatus is divided into different functional units to complete all or part of the above described functions. In addition, the task execution apparatus in the AI model distributed computing process and the task execution method in the AI model distributed computing process provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0288] The present application also provides a computer readable storage medium for storing at least one program code, when the at least one program code is executed by a computing device, the computing device implements the aforementioned AI model distributed computing process task execution method.

[0289] The present application also provides a computer program product, when the computer program product runs on a computing device, the computing device implements the aforementioned AI model distributed computing process task execution method.

[0290] The terms "first", "second", and the like in the present application are used to distinguish between elements or items that have substantially the same function and are similar to each other, and it should be understood that there is no logical or chronological dependency between "first", "second", and "n", and the number and execution order are not limited. It should also be understood that although the following description uses the terms first, second, and the like to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various described examples, a first acceleration device can be referred to as a second acceleration device, and similarly, a second acceleration device can be referred to as a first acceleration device. The first acceleration device and the second acceleration device can both be acceleration devices, and in some cases, can be separate and distinct acceleration devices.

[0291] The term "at least one" in the present application means one or more, and the term "a plurality of" in the present application means two or more, for example, a plurality of acceleration devices means two or more acceleration devices.

[0292] The above description is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0293] In the above-described embodiments, all or part can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part can be implemented in the form of program structure information. The program structure information includes one or more program instructions. When the program instructions are loaded and executed on a computing device, all or part of the processes or functions in the embodiments of the present application are generated.

[0294] A person of ordinary skill in the art can understand that all or part of the steps of the above-described embodiments can be completed by hardware, or by a program instructing related hardware, which can be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0295] The above-described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, a person of ordinary skill in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for task execution in an AI model distributed computing process, characterized in that, Applied to a distributed computing system, the system comprising a host and an acceleration device, the method comprises: The host determines a computing task and a communication task to be executed by the acceleration device based on a distributed parallel computing strategy of the AI model; The host determines a plurality of sub-computing tasks of the computing task and a plurality of sub-communication tasks of the communication task based on a dependency relationship between the computing task and the communication task, the dependency relationship indicating that the computing task depends on the execution result of the communication task, or the communication task depends on the execution result of the computing task; The host sends a first sub-computing task in the plurality of sub-computing tasks and a first sub-communication task in the plurality of sub-communication tasks to the acceleration device, and the first sub-computing task and the first sub-communication task have the dependency relationship; The acceleration device executes the first sub-computing task and the first sub-communication task according to the dependency relationship.

2. The method of claim 1, wherein, The dependency relationship is a first dependency relationship, which indicates that the communication task depends on the execution result of the computing task; the method further comprises: The host sends a second sub-computing task in the plurality of sub-computing tasks and a second sub-communication task in the plurality of sub-communication tasks to the acceleration device, and the second sub-computing task and the second sub-communication task have the dependency relationship; The acceleration device executes the first sub-computing task and the first sub-communication task according to the dependency relationship, comprising: the acceleration device executes the first sub-communication task and the second sub-computing task in parallel after executing the first sub-computing task, and executes the second sub-communication task after executing the second sub-computing task.

3. The method of claim 1, wherein, The dependency relationship is a second dependency relationship, which indicates that the computing task depends on the execution result of the communication task; the method further comprises: The host sends a second sub-computing task in the plurality of sub-computing tasks and a second sub-communication task in the plurality of sub-communication tasks to the acceleration device, and the second sub-computing task and the second sub-communication task have the dependency relationship; The acceleration device executes the first sub-computing task and the first sub-communication task according to the dependency relationship, comprising: the acceleration device executes the first sub-communication task and the second sub-computing task in parallel after executing the first sub-computing task, and executes the second sub-communication task after executing the second sub-computing task.

4. The method according to any one of claims 1 to 3, characterized in that, The AI model comprises a plurality of network layers, and at least one network layer is executed on the acceleration device; wherein If the dependency relationship is a first dependency relationship, which indicates that the communication task depends on the execution result of the computing task, the computing task indicates computing on the at least one network layer, and the communication task indicates data exchange based on the output data of the at least one network layer and an acceleration device other than the acceleration device in the system; If the dependency relationship is a second dependency relationship, the second dependency relationship refers to that the computing task depends on an execution result of the communication task, the communication task instructs data exchange with an acceleration device other than the acceleration device in the system to obtain input data of the at least one network layer, and the computing task instructs computation on the at least one network layer based on the input data of the at least one network layer.

5. The method according to any one of claims 1 to 4, characterized in that, Before the host determines the computing task and the communication task to be executed by the acceleration device based on the distributed parallel computing strategy of the AI model, the method further comprises: The host evaluates performance overheads of the acceleration device executing the computing task and the communication task respectively according to a first task execution strategy and a second task execution strategy to obtain a first performance analysis result corresponding to the first task execution strategy and a second performance analysis result corresponding to the second task execution strategy, the first task execution strategy instructs to execute the split computing task and the split communication task after the computing task and the communication task are split, and the second task execution strategy instructs to execute the computing task and the communication task without splitting the computing task and the communication task. If the performance overhead indicated by the first performance analysis result is less than the performance overhead indicated by the second performance analysis result, the step of determining the computing task and the communication task based on the distributed parallel computing strategy is executed according to the first task execution strategy.

6. The method according to any one of claims 1 to 5, characterized in that, The number of the plurality of sub-computing tasks and the number of the plurality of sub-communication tasks are both a first number, and the distributed parallel computing strategy comprises the first number.

7. The method according to any one of claims 1 to 5, characterized in that, The number of the plurality of sub-computing tasks and the number of the plurality of sub-communication tasks are both a second number, and the second number is determined based on a reference performance analysis result corresponding to the computing task and the communication task.

8. The method of claim 7, wherein, Before the host determines the plurality of sub-computing tasks of the computing task and the plurality of sub-communication tasks of the communication task, the method further comprises: The host determines a reference execution strategy corresponding to each reference number based on the dependency relationship and at least one reference number, the reference execution strategy instructs the acceleration device to execute the plurality of sub-computing tasks of the computing task and the plurality of sub-communication tasks of the communication task determined based on the corresponding reference number according to the dependency relationship. The host determines the second number meeting the condition from the at least one reference number based on a reference performance analysis result corresponding to each reference execution strategy, the reference performance analysis result indicates performance overheads of the acceleration device executing the computing task and the communication task according to the corresponding reference execution strategy.

9. The method according to any one of claims 1 to 8, characterized in that, Before the acceleration device executes the first sub-computing task according to the dependency relationship, the method further comprises any one of the following: The host pre-processes input data of the computing task and sends the pre-processed input data to the acceleration device; Or, The host sends a pre-processing task to the acceleration device, and the acceleration device pre-processes the input data based on the pre-processing task.

10. The method according to any one of claims 1 to 9, characterized in that, The method further includes any one of the following: If the communication task belongs to a first task type, the host controls the acceleration device to perform a first memory rearrangement task after the acceleration device performs the plurality of sub-communication tasks, the first task type indicates that the communication task is a collection type task, the first memory rearrangement task indicates memory rearrangement on execution results of the plurality of sub-communication tasks, and the execution results of the plurality of sub-communication tasks are equivalent to execution results of the communication task after the first memory rearrangement task is performed. Or, If the communication task belongs to a second task type, the host controls the acceleration device to perform a second memory rearrangement task before the acceleration device performs the plurality of sub-communication tasks, the second task type indicates that the communication task is a dispersion type task, the second memory rearrangement task indicates memory rearrangement on input data of the plurality of sub-communication tasks, and the execution results of the plurality of sub-communication tasks are equivalent to the execution results of the communication task after the second memory rearrangement task is performed.

11. The method of claim 10, wherein, The first memory rearrangement task and the second memory rearrangement task are both implemented based on a transpose operator.

12. The method according to any one of claims 1 to 10, characterized in that, The method further includes: The host controls the acceleration device to call a communication interface and store execution results of each sub-communication task to a memory space corresponding to each sub-communication task based on an offset corresponding to each sub-communication task, wherein the offset is determined based on an input data amount of the corresponding sub-communication task and a number of the plurality of sub-communication tasks, and the execution results stored in the memory space corresponding to each sub-communication task are equivalent to the execution results of the communication task.

13. A task execution method in an AI model distributed computing process, characterized in that, A host applied to a distributed computing system, the system further includes an acceleration device, and the method includes: Based on a distributed parallel computing strategy of the AI model, determining a computing task and a communication task to be executed by the acceleration device; Based on a dependency relationship between the computing task and the communication task, determining a plurality of sub-computing tasks of the computing task and a plurality of sub-communication tasks of the communication task, the dependency relationship indicating that the computing task depends on execution results of the communication task, or the communication task depends on execution results of the computing task; Sending a first sub-computing task in the plurality of sub-computing tasks and a first sub-communication task in the plurality of sub-communication tasks to the acceleration device, and the first sub-computing task and the first sub-communication task have the dependency relationship.

14. The method of claim 13, wherein, The dependency relationship is a first dependency relationship, which indicates that the communication task depends on the execution results of the computing task; the method further includes: Sending a second sub-computing task in the plurality of sub-computing tasks and a second sub-communication task in the plurality of sub-communication tasks to the acceleration device, and the second sub-computing task and the second sub-communication task have the dependency relationship; The acceleration device is configured to execute the first sub-communication task and the second sub-computing task in parallel after the first sub-computing task is executed, and execute the second sub-communication task after the second sub-computing task is executed.

15. The method of claim 13, wherein, The dependency relationship is a second dependency relationship, and the second dependency relationship indicates that the computing task depends on an execution result of the communication task. The method further includes: sending, to the acceleration device, a second sub-computing task in the plurality of sub-computing tasks and a second sub-communication task in the plurality of sub-communication tasks, the second sub-computing task and the second sub-communication task having the dependency relationship therebetween; 16. The method according to any one of claims 13 to 15, characterized in that, wherein the acceleration device is configured to execute the first sub-communication task first, and then execute the first sub-computing task and the second sub-communication task in parallel, and after executing the second sub-communication task, execute the second sub-computing task. The AI model includes a plurality of network layers, and at least one network layer is executed on the acceleration device; wherein If the dependency relationship is a first dependency relationship, the first dependency relationship indicates that the communication task depends on an execution result of the computing task, the computing task indicates computing on the at least one network layer, and the communication task indicates data exchange with an acceleration device other than the acceleration device in the system based on output data of the at least one network layer.

17. The method according to any one of claims 13 to 16, characterized in that, If the dependency relationship is a second dependency relationship, the second dependency relationship indicates that the computing task depends on an execution result of the communication task, the communication task indicates data exchange with an acceleration device other than the acceleration device in the system to obtain input data of the at least one network layer, and the computing task indicates computing on the at least one network layer based on the input data of the at least one network layer. Before determining the computing task and the communication task to be executed by the acceleration device based on the distributed parallel computing strategy of the AI model, the method further includes: evaluating performance overheads of executing the computing task and the communication task on the acceleration device according to a first task execution strategy and a second task execution strategy respectively, to obtain a first performance analysis result corresponding to the first task execution strategy and a second performance analysis result corresponding to the second task execution strategy, the first task execution strategy indicating that the computing task and the communication task are executed after being split, and the second task execution strategy indicating that the computing task and the communication task are executed without being split.

18. The method according to any one of claims 13 to 17, characterized in that, If the performance overhead indicated by the first performance analysis result is less than the performance overhead indicated by the second performance analysis result, the step of determining the computing task and the communication task based on the distributed parallel computing strategy is executed according to the first task execution strategy.

19. The method according to any one of claims 13 to 17, characterized in that, The number of the plurality of sub-computing tasks and the number of the plurality of sub-communication tasks are a first number, and the distributed parallel computing strategy includes the first number. The number of the plurality of sub-computing tasks and the number of the plurality of sub-communication tasks are a second number, and the second number is determined based on a reference performance analysis result corresponding to the computing task and the communication task.

20. The method of claim 19, wherein, Before the determining the plurality of sub-computing tasks of the computing task and the plurality of sub-communication tasks of the communication task, the method further comprises: determining, based on the dependency relationship and at least one reference quantity, a reference execution strategy corresponding to each reference quantity, the reference execution strategy indicating that the accelerator executes the plurality of sub-computing tasks of the computing task and the plurality of sub-communication tasks of the communication task based on the corresponding reference quantity according to the dependency relationship; determining, based on a reference performance analysis result corresponding to each reference execution strategy, a second quantity meeting a condition from the at least one reference quantity, the reference performance analysis result indicating a performance overhead of the accelerator when executing the computing task and the communication task according to the corresponding reference execution strategy.

21. The method according to any one of claims 13 to 20, characterized in that, Before the sending the first sub-computing task to the accelerator, the method further comprises any one of the following: preprocessing input data of the computing task, and sending the preprocessed input data to the accelerator; or sending a preprocessing task to the accelerator, the accelerator being configured to preprocess the input data based on the preprocessing task.

22. The method of any one of claims 13-21, wherein, The method further comprises any one of the following: if the communication task belongs to a first task type, controlling the accelerator to execute a first memory rearrangement task after the execution of the plurality of sub-communication tasks, the first task type indicating that the communication task is a collection type task, the first memory rearrangement task indicating that a memory rearrangement is performed on an execution result of the plurality of sub-communication tasks, and the execution result of the plurality of sub-communication tasks is equivalent to an execution result of the communication task after the execution of the first memory rearrangement task; or if the communication task belongs to a second task type, controlling the accelerator to execute a second memory rearrangement task before the execution of the plurality of sub-communication tasks, the second task type indicating that the communication task is a dispersion type task, the second memory rearrangement task indicating that a memory rearrangement is performed on input data of the plurality of sub-communication tasks, and the execution result of the plurality of sub-communication tasks is equivalent to the execution result of the communication task after the execution of the second memory rearrangement task.

23. The method of claim 22, wherein, The first memory rearrangement task and the second memory rearrangement task are both implemented based on a transpose operator.

24. The method of any one of claims 13-23, wherein, The method further comprises: controlling the accelerator to call a communication interface, and storing an execution result of each sub-communication task to a memory space corresponding to the sub-communication task based on an offset corresponding to the sub-communication task, the offset being determined based on an input data amount of the corresponding sub-communication task and a quantity of the plurality of sub-communication tasks, and the execution result stored in the memory space corresponding to each sub-communication task being equivalent to the execution result of the communication task. 25.A task execution apparatus in an AI model distributed computing process, comprising: A host configured in a distributed computing system, the system further comprising an accelerator, the apparatus comprising: a task determination unit configured to determine a computing task and a communication task to be executed by the accelerator based on a distributed parallel computing strategy of the AI model; The subtask determining unit is configured to determine a plurality of sub-computing tasks of the computing task and a plurality of sub-communication tasks of the communication task based on a dependency relationship between the computing task and the communication task, the dependency relationship indicating that the computing task depends on an execution result of the communication task or the communication task depends on an execution result of the computing task. The task sending unit is configured to send a first sub-computing task in the plurality of sub-computing tasks and a first sub-communication task in the plurality of sub-communication tasks to the acceleration device, the first sub-computing task and the first sub-communication task having the dependency relationship therebetween.

26. A distributed computing system, comprising: The system includes a host and an acceleration device, the host being configured to control the acceleration device, and the system being configured to implement the method according to any one of the preceding claims 1 to 12.

27. A computing device, comprising: The computing device includes a processor and a memory, the processor being configured to execute at least one piece of program code stored in the memory to enable the computing device to implement the method according to any one of the preceding claims 13 to 24.

28. A computer-readable storage medium, characterized in that, The computer-readable storage medium is configured to store at least one piece of program code, the at least one piece of program code, when executed by a computing device, enabling the computing device to implement the method according to any one of the preceding claims 13 to 24.

29. A computer program product, characterised in that, The computer program product, when running on a computing device, enables the computing device to implement the method according to any one of the preceding claims 13 to 24.

Images