Parallel execution method and device for deep learning model

Abstract

The invention discloses a parallel execution method of a deep learning model. The method comprises the following steps: representing a calculation task related to the deep learning model as calculation graph data; wherein the computational graph data is the computational graph data represented by an intermediate representation form supported by the deep learning compiler; obtaining a parallel computing mode corresponding to the computing task and represented based on the intermediate representation form, and compiling the computing graph data into parallel task execution codes corresponding to the computing nodes in the computing cluster based on the parallel computing mode; the parallel task execution codes corresponding to all the computing nodes are dispatched to all the computing nodes in the computing cluster, and all the computing nodes execute the parallel task execution codes in parallel.

Description

Technical Field

[0001] The embodiments of this specification belong to the field of artificial intelligence, and more particularly, to a method and device for parallel execution of a deep learning model. Background Art

[0002] Large models usually refer to complex and high-dimensional models used in the field of machine learning and artificial intelligence, especially deep neural networks with huge parameters. These models have attracted much attention due to their powerful learning and reasoning capabilities, and they can demonstrate strong performance in a variety of tasks such as natural language processing, computer vision, language translation, and image recognition.

[0003] Common large models, such as the GPT (Generative Pre-trained Transformer) series, BERT (Bidirectional Encoder Representation Transformer), ResNet, etc., have achieved breakthroughs in multiple fields by learning complex representations from large amounts of data.

[0004] In practical applications, when executing training tasks related to deep learning models, in order to make the execution of the training tasks more efficient, a parallel optimization strategy can usually be adopted to convert the training tasks into execution codes suitable for execution on different computing nodes, and then send these execution codes to different computing nodes for parallel execution.

[0005] However, currently, when performing parallel optimization on training tasks related to deep learning models, manual optimization is usually used. This is not only difficult to develop and tune, with a long development cycle, but also highly dependent on expert experience. Summary of the invention

[0006] This specification proposes a parallel execution method for a deep learning model, the method comprising:

[0007] Representing a computing task related to a deep learning model as computing graph data; wherein the computing graph data is computing graph data represented by an intermediate representation supported by a deep learning compiler corresponding to the deep learning model;

[0008] Obtaining a parallel computing mode corresponding to the computing task represented by the intermediate representation, and compiling the computing graph data into parallel task execution codes corresponding to each computing node in a computing cluster based on the parallel computing mode; the computing cluster is used to execute computing tasks related to the deep learning model;

[0009] The parallel task execution codes corresponding to the respective computing nodes are respectively dispatched to the respective computing nodes in the computing cluster, and the respective computing nodes execute the parallel task execution codes in parallel.

[0010] Optionally, the computing tasks related to the deep learning model are represented as computing graph data, including:

[0011] Simulate and execute computing tasks related to the deep learning model, and record computing operations generated during the simulation of executing the computing tasks and data flows during the execution of the computing operations;

[0012] Based on an intermediate representation supported by the deep learning compiler, the recorded computing operations and the data flows are represented as computing graph data.

[0013] Optionally, obtaining a parallel computing mode corresponding to the computing task represented by the intermediate representation includes:

[0014] The optimization algorithm supported by the deep learning compiler is run to automatically perform parallel optimization calculations on the computation graph data to obtain an optimal parallel computing method corresponding to the computation graph data, and the optimal parallel computing method is represented based on the intermediate representation as a parallel computing method corresponding to the computing task.

[0015] Optionally, the computing task is executed in parallel on each computing node in the computing cluster in the form of a pipeline task; the pipeline task includes a plurality of subtasks executed in sequence; the parallel computing method includes boundary information on the computing graph data based on the plurality of subtasks represented by the intermediate representation; the boundary information is marked on the computing graph data;

[0016] Acquiring a parallel computing mode corresponding to the computing task represented by the intermediate representation, including:

[0017] The boundary information marked on the computation graph data and represented based on the intermediate representation is obtained.

[0018] Optionally, compiling the computation graph data into parallel task execution codes corresponding to each computing node in the computing cluster based on the parallel computing mode includes:

[0019] Based on the parallel computing method, the computing graph data is compiled into parallel task execution codes that are adapted to the computing power of each computing node in the computing cluster.

[0020] Optionally, compiling the computation graph data into parallel task execution codes corresponding to each computing node in the computing cluster based on the parallel computing method includes:

[0021] Based on the boundary information, the computation graph data is divided into a plurality of sub-computation graph data corresponding to the plurality of subtasks one by one;

[0022] Based on the several sub-computation graph data, pipeline tasks are generated for each computing node in the computing cluster, and the pipeline tasks are compiled into pipeline task execution codes; wherein the pipeline task is a pipeline task composed of the several subtasks in an execution order; the pipeline task execution code includes execution codes corresponding to the several subtasks.

[0023] Optionally, the parallel task execution codes corresponding to the respective computing nodes are respectively dispatched to the respective computing nodes in the computing cluster, and the respective computing nodes execute the parallel task execution codes in parallel, including:

[0024] Acquire an input data set required for executing the pipeline task, and perform data splitting on the input data set to obtain input data subsets corresponding to each computing node in the computing cluster; wherein the input data subsets corresponding to different computing nodes contain different input data;

[0025] The pipeline task execution code and the input data subsets corresponding to the respective computing nodes are respectively scheduled to the respective computing nodes in the computing cluster, and the respective computing nodes execute the pipeline task execution code in parallel based on the input data subsets.

[0026] Optionally, the method further includes:

[0027] Further performing data splitting on the input data subsets corresponding to the respective computing nodes in the computing cluster to obtain input data of the execution codes corresponding to the respective subtasks contained in the pipeline task execution code;

[0028] The step of dispatching the pipeline task execution code and the input data subsets corresponding to the respective computing nodes to the respective computing nodes in the computing cluster, and having the respective computing nodes execute the pipeline task execution code in parallel based on the input data subsets, comprises:

[0029] The pipeline task execution code and the input data of the execution code corresponding to each subtask contained in the pipeline task execution code are respectively scheduled to each computing node in the computing cluster, and each computing node executes the execution code corresponding to each subtask contained in the pipeline task execution code in parallel based on the input data.

[0030] Optionally, the deep learning compiler supports SPMD parallel strategy;

[0031] Performing data splitting on the input data set or the input data subset includes:

[0032] The input data set or the input data subset is split using a data splitting method supported by the SPMD parallel strategy.

[0033] Optionally, the pipeline task execution code and the input data of the execution code corresponding to each subtask contained in the pipeline task execution code are respectively scheduled to each computing node in the computing cluster, and each computing node executes the execution code corresponding to each subtask contained in the pipeline task execution code in parallel based on the input data, including:

[0034] Determine execution times on the respective computing nodes for execution codes corresponding to the respective subtasks contained in the pipeline task execution code; wherein at least part of the execution codes corresponding to the respective subtasks contained in the pipeline task execution code correspond to different execution times on the respective computing nodes;

[0035] The pipeline task execution code and the input data of the execution code corresponding to each subtask contained in the pipeline task execution code are respectively scheduled to each computing node in the computing cluster, and each computing node executes the execution code corresponding to each subtask contained in the pipeline task execution code in parallel based on the input data and the execution time.

[0036] Optionally, the execution code corresponding to each subtask contained in the pipeline task execution code is used to determine the execution time on each computing node, including:

[0037] Run the pipeline arrangement algorithm supported by the deep learning compiler, and perform pipeline arrangement processing on the execution code corresponding to each subtask contained in the pipeline task execution code to determine the execution time of the execution code corresponding to each subtask on each computing node.

[0038] Optionally, running a pipeline arrangement algorithm supported by the deep learning compiler to perform pipeline arrangement processing on the execution codes corresponding to the respective subtasks contained in the pipeline task execution code to determine the execution time of the execution codes corresponding to the respective subtasks on the respective computing nodes includes:

[0039] Run the pipeline arrangement algorithm supported by the deep learning compiler, at least refer to the computing power status of each computing node, and perform pipeline arrangement processing on the execution code corresponding to each subtask contained in the pipeline task execution code to determine the execution time of the execution code corresponding to each subtask on each computing node.

[0040] Optionally, at least part of the execution codes corresponding to the respective subtasks contained in the pipeline task execution code corresponds to different execution times on the respective computing nodes.

[0041] Optionally, the parallel task execution codes corresponding to the respective computing nodes are respectively dispatched to the respective computing nodes in the computing cluster, and the respective computing nodes execute the parallel task execution codes in parallel, including:

[0042] Obtaining configuration information of a runtime environment defined for each computing node;

[0043] The pipeline task execution code and the configuration information are respectively dispatched to each computing node in the computing cluster, and each computing node configures a runtime environment based on the configuration information, and executes the pipeline task execution code in parallel in the configured runtime environment.

[0044] Optionally, the parallel task execution codes corresponding to the respective computing nodes are respectively dispatched to the respective computing nodes in the computing cluster, and the respective computing nodes execute the parallel task execution codes in parallel, including:

[0045] Acquire a data communication operation defined for the pipeline task, and compile the data communication operation into a data communication execution code; wherein the data communication operation includes a data communication operation performed between the computing nodes and a data communication operation performed between the subtasks during the execution of the pipeline task execution code;

[0046] The pipeline task execution code and the data communication execution code are respectively scheduled to each computing node in the computing cluster, and the pipeline task execution code is executed in parallel by each computing node. In the process of executing the pipeline task execution code, the data communication execution code is executed to realize data communication between the computing nodes and data communication between the subtasks.

[0047] Optionally, the deep learning model is a deep learning model defined in a deep learning framework accessed by the deep learning compiler.

[0048] Optionally, the deep learning framework includes any one of the following:

[0049] Tensorflow framework, PyTorch framework, JAX framework.

[0050] Optionally, the computing task includes a training task for the deep learning model;

[0051] Correspondingly, the computational graph data includes computational graph data related to the forward propagation calculation in the training task, computational graph data related to the back propagation calculation in the training task, and computational graph data related to the model parameter update calculation in the training task.

[0052] Optionally, the deep learning compiler includes an XLA compiler.

[0053] This specification also proposes a parallel execution device for a deep learning model, the device comprising:

[0054] A representation module that represents a computing task related to a deep learning model as a computational graph data; wherein the computational graph data is computational graph data represented by an intermediate representation supported by a deep learning compiler corresponding to the deep learning model;

[0055] A compiling module, which obtains a parallel computing mode corresponding to the computing task represented by the intermediate representation, and compiles the computing graph data into a parallel task execution code corresponding to each computing node in a computing cluster based on the parallel computing mode; the computing cluster is used to execute computing tasks related to the deep learning model;

[0056] The scheduling module schedules the parallel task execution codes corresponding to the respective computing nodes to the respective computing nodes in the computing cluster, so that the respective computing nodes execute the parallel task execution codes in parallel.

[0057] In the above embodiment, the intermediate representation supported by the deep learning compiler is used to represent the parallel computing mode corresponding to the computing tasks related to the deep learning model, so that the deep learning compiler can clarify the parallel computing mode adopted when parallel optimization of the above computing tasks is performed by parsing the parallel computing mode represented by the intermediate representation, so that the deep learning compiler can utilize its own optimization capability to further compile the computing graph data of the computing task into parallel task execution code based on the parallel computing mode, and execute it in parallel on each computing node in the computing cluster connected to the deep learning compiler, thereby realizing automatic parallel optimization of the computing tasks of the deep learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0059] Figure 1is a flowchart of a parallel execution method of a deep learning model shown in an embodiment of this specification;

[0060] Figure 2 is an architecture diagram of a deep learning system built based on the XLA compiler shown in an embodiment of this specification;

[0061] Figure 3 It is a flowchart of an XLA compiler automatically performing parallel execution optimization on computing tasks related to a deep learning model, as shown in an embodiment of this specification;

[0062] Figure 4 is a schematic structural diagram of an electronic device shown in an embodiment of this specification;

[0063] Figure 5 It is a block diagram of a parallel execution device for a deep learning model shown in one embodiment of this specification. DETAILED DESCRIPTION

[0064] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0065] Since deep learning models usually contain a large number of model parameters, computing tasks related to deep learning models (such as training tasks) are usually more complex, both in terms of the computing operations involved and the data flow.

[0066] In practical applications, before deploying a deep learning model defined by a deep learning training framework to a computing device for execution, it is usually necessary to adopt some optimization measures to optimize the computing tasks related to the deep learning model.

[0067] For example, these optimization measures may generally include measures to improve computational efficiency, measures to reduce memory usage, measures to accelerate data transmission, measures to increase parallelism, and so on.

[0068] Among them, parallel optimization of computing tasks related to deep learning models is a very important optimization of computing tasks related to deep learning models.

[0069] By performing parallel optimization on computing tasks related to deep learning models, computing tasks related to deep learning models can be optimized into multiple tasks that can be executed in parallel on different computing devices. This can not only fully utilize the computing power of each computing device, but also significantly improve the computing efficiency of computing tasks related to deep learning models.

[0070] In the related art, manual optimization is usually used when parallel optimization is performed on computing tasks related to deep learning models.

[0071] However, the manual optimization approach is not only difficult to develop and tune, with a long development cycle, but also highly dependent on expert experience.

[0072] Based on this, this specification proposes a technical solution that uniformly represents a parallel computing method for computing tasks related to a deep learning model based on an intermediate representation supported by a deep learning model compiler, and utilizes the optimization capabilities of the deep learning compiler itself to automatically perform parallel optimization for the computing tasks of the deep learning model based on this parallel computing method.

[0073] In this technical solution, the deep learning compiler can represent the computing task related to the deep learning model as computing graph data based on the intermediate representation supported by the deep learning compiler, and obtain the parallel computing mode corresponding to the computing task represented by the intermediate representation;

[0074] For example, in one embodiment, the deep learning compiler can automatically perform parallel optimization calculations on the computational graph data by running a supported optimization algorithm to obtain the optimal parallel computing method corresponding to the computational graph data, and then express the optimal parallel computing method based on the intermediate representation.

[0075] After obtaining the parallel computing method corresponding to the computing task represented by the intermediate representation, the deep learning compiler can use its own optimization capabilities to compile the computing graph data into parallel task execution codes corresponding to each computing node in the computing cluster based on the parallel computing method.

[0076] For example, in one embodiment, the computing task can be executed in parallel on each computing node in the computing cluster in the form of a pipeline task, and the parallel computing method can be specifically the boundary information represented by the above intermediate representation form marked on the computing graph data; wherein the boundary information is used to represent the boundary of several subtasks executed in sequence contained in the computing task on the computing graph data. In this computing scenario, the deep learning compiler can specifically divide the computing graph data into several sub-computation graph data corresponding to the above-mentioned several subtasks based on the above-mentioned boundary information, and generate pipeline tasks for each computing node in the computing cluster based on the several sub-computation graph data. The pipeline task can specifically be a pipeline task composed of the above-mentioned several subtasks in the order of execution. Then, the pipeline tasks generated for each computing node can be compiled into pipeline task execution codes respectively.

[0077] After the deep learning compiler compiles the computational graph data into parallel task execution codes corresponding to each computing node in the computing cluster based on the parallel computing method, it can utilize the scheduling capability of the deep learning compiler itself to schedule the parallel task execution codes to each computing node in the computing cluster, and each computing node executes the parallel task execution codes in parallel to complete the parallel optimization of computing tasks for the deep learning model.

[0078] In the above scheme, the intermediate representation supported by the deep learning compiler is used to represent the parallel computing method corresponding to the computing tasks related to the deep learning model, so that the deep learning compiler can clarify the parallel computing method used for parallel optimization of the above computing tasks by parsing the parallel computing method represented by the intermediate representation, so that the deep learning compiler can utilize its own optimization capabilities to further compile the computing graph data of the computing task into parallel task execution code based on the parallel computing method, and execute it in parallel on each computing node in the computing cluster connected to the deep learning compiler, thereby realizing automatic parallel optimization of the computing tasks of the deep learning model.

[0079] See also Figure 1 , Figure 1 This is a flowchart of a parallel execution method of a deep learning model shown in this specification, and the method includes the following execution process:

[0080] Step 102, representing a computing task related to the deep learning model as computing graph data; wherein the computing graph data is computing graph data represented by an intermediate representation supported by a deep learning compiler corresponding to the deep learning model;

[0081] A deep learning compiler is a software tool specifically designed to optimize and accelerate the execution of deep learning models. Using a deep learning compiler, a deep learning model defined in a high-level programming language can be compiled into optimized executable code that can be executed on a specific computing device or hardware platform.

[0082] It should be noted that the specific type of the deep learning compiler is not particularly limited in this specification;

[0083] For example, in one example, the above-mentioned deep learning compiler can be specifically an XLA compiler. Among them, XLA (Accelerated Linear Algebra) is a high-performance deep learning compiler designed to provide efficient computing performance for various hardware (such as CPU, GPU and TPU). XLA is designed to optimize deep learning workloads.

[0084] The above-mentioned deep learning model can specifically be a deep learning model defined by a developer in a specific deep learning framework based on a high-level programming language. In one embodiment shown, the above-mentioned deep learning compiler can specifically access a deep learning framework, and the above-mentioned deep learning model can be a deep learning model defined by a developer in a deep learning framework accessed by the deep learning compiler based on a high-level programming language. Among them, the deep learning framework can include a Tensorflow framework, a PyTorch framework, a JAX framework, and the like.

[0085] For example, still taking the above-mentioned deep learning compiler as the XLA compiler as an example, the XLA compiler can usually be used as the backend compiler component of popular deep learning frameworks such as TensorFlow, PyTorch and JAX. When developers define a deep learning model on deep learning frameworks such as TensorFlow, PyTorch and JAX, the XLA compiler can continue to compile the deep learning model into optimized executable code for execution on a specific computing device or a specific hardware platform as a backend compiler component.

[0086] When a deep learning model defined in a deep learning framework is deployed to various computing nodes for execution, the deep learning compiler usually uses the intermediate representation (IntermediateRepresentation) it supports to represent the computing tasks related to the deep learning model as computational graph data (Computational Graph).

[0087] It needs to be explained that the intermediate representation supported by the deep learning compiler is an intermediate form of programming language between source code and execution code. In practical applications, the intermediate representation supported by the deep learning compiler can be used to describe the computing tasks related to the deep learning model. Among them, different deep learning compilers usually support different intermediate representations. For example, still taking the above-mentioned deep learning compiler as the XLA compiler as an example, the intermediate representation supported by the XLA compiler is called HLO (High-Level Operator).

[0088] In one embodiment shown, when the deep learning compiler represents a computing task related to a deep learning model as computing graph data, it can specifically simulate the execution of the computing task related to the deep learning model, and capture and record the computing operations generated in the process of simulating the execution of the computing task and the data flow in the process of executing the computing operation; then, the recorded computing operations and data flow can be represented as computing graph data based on the intermediate representation supported by the deep learning compiler.

[0089] For example, in practical applications, computational graph data usually includes nodes representing computational operations and edges representing the flow of code data. The deep learning compiler can represent the computational operations as nodes of the computational graph data and the data flow as edges of the computational graph data based on the supported intermediate representation.

[0090] It should be noted that computing tasks related to deep learning models may include both training tasks for deep learning models and reasoning tasks based on trained deep learning models, which are not specifically limited in this specification.

[0091] For example, taking the computing task related to the deep learning model as a training task as an example, the calculations involved in the process of training a deep learning model usually include forward propagation calculations, back propagation calculations, and calculations for updating model parameters; therefore, in this case, the deep learning compiler uses the intermediate representation it supports to represent the training task related to the deep learning model. The calculation graph data generally includes the calculation graph data related to the forward propagation calculation in the training task, the calculation graph data related to the back propagation calculation in the training task, and the calculation graph data related to the model parameter update calculation in the training task.

[0092] In some embodiments, when the deep learning compiler represents the training task related to the deep learning model as computational graph data, the computational graph data related to the forward propagation calculation of the training task can be represented only based on the intermediate representation supported by the deep learning compiler, and then the deep learning framework itself can be further utilized to derive the computational graph data related to the back propagation calculation in the training task and the computational graph data related to the model parameter update calculation in the training task based on the computational graph data related to the forward propagation calculation in the training task.

[0093] For example, still taking the above-mentioned deep learning compiler as the XLA compiler and the above-mentioned deep learning framework as the PyTorch framework as an example, in this scenario, a library named torchxla is usually provided in the PyTorch framework to support the functions of the XLA compiler.

[0094] In this case, after the XLA compiler represents the computational graph data related to the above-mentioned forward propagation calculation in the form of HLO based on the supported intermediate representation, on the one hand, the auto grad (automatic differentiation) capability of torchxla can be used in the PyTorch framework to automatically derive the computational graph data related to the above-mentioned back-propagation calculation based on the computational graph data related to the above-mentioned forward propagation calculation; on the other hand, the apply grad (gradient application) capability of torchxla can be used in the PyTorch framework to automatically derive the computational graph data related to the above-mentioned model parameter update calculation based on the computational graph data related to the above-mentioned back-propagation calculation.

[0095] In this way, the inference computing capabilities of the deep learning framework itself can be fully utilized, reducing the processing pressure of the deep learning compiler when compiling and optimizing computing tasks into execution code.

[0096] Step 104, obtaining a parallel computing mode corresponding to the computing task represented by the intermediate representation, and compiling the computing graph data into parallel task execution codes corresponding to each computing node in a computing cluster based on the parallel computing mode; the computing cluster is used to execute computing tasks related to the deep learning model;

[0097] In this specification, in order to make full use of the compilation and optimization capabilities of the deep learning compiler itself, in addition to representing the computing tasks related to the deep learning model as computing graph data based on the above intermediate representation, the parallel computing method corresponding to the computing task can also be represented based on the intermediate representation. Among them, the parallel computing method corresponding to the computing task refers to the parallel method adopted by each computing node in the computing cluster when executing the computing task in parallel.

[0098] In one embodiment shown, the deep learning compiler can also specifically support related optimization algorithms. In this case, the deep learning compiler can automatically perform parallel optimization calculations on the above-mentioned calculation graph data by running the supported optimization algorithm to solve the optimal parallel computing method corresponding to the calculation graph data. For example, in practical applications, a parallel optimization goal can be set in advance, and the optimization algorithm can be iteratively run until the optimal parallel computing method that meets the parallel optimization goal is solved.

[0099] After solving the optimal parallel computing method corresponding to the calculation graph data, the deep learning compiler can further express the optimal parallel computing method based on the supported intermediate representation as the obtained parallel computing method corresponding to the computing task.

[0100] Among them, the specific algorithm types of the above-mentioned optimization algorithms supported by the deep learning compiler are no longer specifically limited in this specification. In practical applications, the above-mentioned optimization algorithms may specifically include any form of optimization algorithms that can be used for parallel optimization of computational graph data.

[0101] For example, in one implementation shown, the above optimization algorithm may specifically be an optimization algorithm based on graph partitioning.

[0102] If the above-mentioned optimization algorithm is an optimization algorithm based on graph segmentation, then when the deep learning compiler runs the optimization algorithm and automatically performs parallel optimization calculations on the above-mentioned calculation graph data, it can specifically execute the following calculation process: First, the deep learning compiler can estimate the resource consumption of each node in the above-mentioned calculation graph data, such as estimating the FLOPS and memory consumption of the calculation operations corresponding to each node.

[0103] Secondly, the deep learning compiler can also pre-set an optimization goal for parallel optimization of the computational graph data; wherein, the optimization goal can be customized based on actual optimization requirements; for example, the optimization goal for parallel optimization of the computational graph data can be to achieve load balancing among the computing nodes in the computing cluster when executing the calculations corresponding to the computational graph data in parallel, or to minimize the communication volume among the computing nodes in the computing cluster when executing the calculations corresponding to the computational graph data in parallel, and so on.

[0104] Then, the deep learning compiler can select a suitable graph partitioning algorithm and use the graph partitioning algorithm to partition the computational graph data according to a certain partitioning strategy to obtain multiple subgraphs.

[0105] Furthermore, the deep learning compiler can evaluate the performance of each computing node in the cluster when executing the computing tasks corresponding to the computing graph data in parallel based on this partitioning strategy based on the resource consumption of the nodes contained in each sub-graph obtained by the partitioning, and determine whether the above optimization goal is met based on the evaluated performance. If the optimization goal is not met, you can try to use other partitioning strategies, re-partition the computing graph data, and then iterate the above process until the above optimization goal is met.

[0106] It should be emphasized that the above example is only a schematic illustration of the above-mentioned optimization algorithm as an optimization algorithm based on graph segmentation. In practical applications, the deep learning compiler can obviously also run other types of optimization algorithms to automatically perform parallel optimization calculations on the above-mentioned calculation graph data, such as dynamic programming algorithms, etc., and no further examples will be given one by one in this manual.

[0107] By allowing the deep learning compiler to access the optimization algorithm, the deep learning compiler can automatically optimize the computational graph data of the deep learning model in parallel by running the optimization algorithm when compiling and optimizing the deep learning models defined based on different deep learning frameworks, without considering the differences between different deep learning models. This can improve the generalization ability of the deep learning compiler and allow the deep learning compiler to flexibly adapt to different deep learning frameworks.

[0108] Of course, in addition to automatically performing parallel optimization calculations on the above-mentioned computational graph data by running the optimization algorithms supported by the deep learning compiler, the deep learning compiler can also support developers to manually optimize the above-mentioned computational graph data.

[0109] In this case, the developer can manually express the parallel computing method corresponding to the computing task based on the intermediate representation supported by the deep learning compiler. The deep learning compiler can directly obtain the parallel computing method manually expressed by the developer for the subsequent compilation process of the computing task.

[0110] It should be noted that the specific data representation of the above parallel computing method obtained by the deep learning compiler is also specifically limited in this specification. In practical applications, the specific data representation of the above parallel computing method can be clarified based on the specific parallel computing scenario.

[0111] In one embodiment shown, the computing task can be executed in parallel on each computing node in the computing cluster in the form of a pipeline task.

[0112] Specifically, the pipeline task may include several subtasks executed in sequence; the several subtasks may generally correspond to different execution stages of the above computing task. For example, in practical applications, for a multi-layer deep learning model, the several subtasks constituting the pipeline task may correspond to different computing layers of the deep learning model.

[0113] In this parallel computing scenario, the parallel computing method may specifically be based on the boundary information of the plurality of subtasks on the computation graph data represented by the intermediate representation. That is, the specific data representation of the parallel computing method may be information for representing the boundary of the plurality of subtasks on the plurality of subtasks.

[0114] In this case, the boundary information may be specifically marked on the computational graph data in the form of a label or mark.

[0115] For example, in one example, when the deep learning compiler automatically performs parallel optimization calculations on the computational graph data based on the supported optimization algorithm and obtains the optimal parallel computing method, it can represent the optimal parallel computing method based on the supported intermediate representation, and then automatically mark the optimal parallel computing method on the above computational graph data in the form of a label or mark.

[0116] In another example, the developer can manually represent the parallel computing method based on the intermediate representation supported by the deep learning compiler, and then manually mark the parallel computing method on the computational graph data in the form of a label or mark. When the deep learning compiler obtains the parallel computing method corresponding to the above computing task represented by the intermediate representation, it can obtain the above boundary information marked on the computational graph data by parsing the computational graph data.

[0117] After the deep learning compiler obtains the parallel computing method corresponding to the above-mentioned computing task represented by the above-mentioned intermediate representation, since the parallel computing method is a parallel computing method represented by the intermediate representation supported by the deep learning compiler, and the intermediate representation itself is a programming language supported by the deep learning compiler, therefore for the deep learning compiler, by parsing the parallel computing method represented by the intermediate representation, it is possible to clearly understand the parallel computing method used for parallel optimization of the above-mentioned computing task, so that the subsequent compilation process can be automatically executed based on the above-mentioned parallel computing method, and the above-mentioned computing graph data can be compiled into parallel task execution codes corresponding to each computing node in the above-mentioned computing cluster.

[0118] It should be noted that the specific compilation method adopted by the deep learning compiler when compiling the above-mentioned computational graph data into parallel task execution code usually depends on the above-mentioned parallel computing method and is not specifically limited in this specification.

[0119] In one embodiment shown, if the above-mentioned computing task is executed in parallel on each computing node in the above-mentioned computing cluster in the form of a pipeline task, in this parallel computing scenario, the above-mentioned parallel computing method can be based on the boundary information of several subtasks represented by the intermediate representation supported by the deep learning compiler on the computing graph data.

[0120] When the deep learning compiler compiles the above-mentioned computational graph data into parallel task execution codes corresponding to each computing node in the computing cluster based on the above-mentioned parallel computing method, it can first divide the computational graph data into a plurality of sub-computational graph data corresponding to the above-mentioned plurality of sub-tasks based on the above-mentioned boundary information. Then, based on the above-mentioned plurality of sub-computational graph data, it generates pipeline tasks for each computing node in the computing cluster respectively, and compiles the generated pipeline tasks into pipeline task execution codes respectively.

[0121] For example, when the deep learning compiler generates pipeline tasks for each computing node in the computing cluster based on the above-mentioned several sub-computation graph data, and then compiles the generated pipeline tasks into pipeline task execution codes, it can first generate the pipeline task and then compile the pipeline task into the pipeline task execution code, or it can first compile the above-mentioned several sub-computation graph data into task execution codes, and then create the pipeline task based on the compiled task execution codes corresponding to the above-mentioned several sub-computation graph data, and generate the pipeline task execution code. This is not specifically limited in this specification.

[0122] Among them, since the pipeline task is a pipeline task composed of the above-mentioned several subtasks in the execution order, accordingly, the pipeline task execution code can specifically include the execution codes corresponding to the above-mentioned several subtasks. The execution codes corresponding to these subtasks can still follow the execution order of the above-mentioned several subtasks themselves in the above-mentioned pipeline task execution code.

[0123] By executing the several subtasks included in the above-mentioned computing task in parallel on each computing node in the above-mentioned computing cluster in the form of pipeline tasks, each computing node can completely execute each subtask included in the above-mentioned computing task locally, without the need to split each subtask included in the above-mentioned computing task into different computing nodes for parallel execution, thereby significantly reducing the communication overhead of each computing node when executing the above-mentioned computing task.

[0124] It should be emphasized that the above example is only a schematic illustration of the above computing tasks being executed in parallel on each computing node in the above computing cluster in the form of pipeline tasks. In actual applications, the above computing tasks can obviously also be executed in parallel on each computing node in the computing cluster using other forms of parallel computing besides pipeline tasks.

[0125] For example, in one example, the above computing task can still be included in a number of subtasks that are executed in sequence and are scheduled to different computing nodes in the computing cluster for parallel execution. In this case, for different computing nodes, since they respectively execute different subtasks included in the computing task, it is usually necessary to introduce additional data communication processes between the computing nodes. For example, in the process of executing each subtask, each computing node transmits the computing results to each other.

[0126] In another example, the computation graph data of the above computation task may not be split, but the above computation task may be executed in parallel on each computing node in the computing cluster as a whole. In this case, the input data set required to execute the computation task may be split, so that each computing node can execute the computation task in parallel based on different input data subsets.

[0127] Step 106 : dispatching the parallel task execution code to each computing node in the computing cluster respectively, so that each computing node executes the parallel task execution code in parallel.

[0128] After the deep learning compiler compiles the above-mentioned computational graph data into parallel task execution codes corresponding to each computing node in the above-mentioned computing cluster based on the above-mentioned parallel computing method, it can also use its own optimization capabilities to further perform parallel optimization on the parallel task execution code to improve the execution performance of the parallel task execution code on each computing node.

[0129] In practical applications, the specific optimization measures used when performing parallel optimization on the parallel task execution code can be flexibly selected based on specific parallel optimization requirements and are not particularly limited in this specification.

[0130] In one embodiment shown, when the parallel task execution code is optimized in parallel, the input data set required to execute the computing task can be split to allow each computing node to execute the same parallel task execution code in parallel based on different input data subsets.

[0131] For example, still taking the scenario in which the above-mentioned computing task is executed in parallel on each computing node in the above-mentioned computing cluster in the form of a pipeline task, in this parallel computing scenario, when the deep learning compiler performs parallel optimization on the compiled pipeline task execution code, it can first obtain the input data set required to execute the pipeline task, and perform data splitting on the input data set to obtain input data subsets corresponding to each computing node in the computing cluster; wherein, the input data contained in the input data subsets corresponding to different computing nodes may be different.

[0132] Then, the deep learning compiler can use its own scheduling capabilities to schedule the pipeline task execution code and the input data subsets corresponding to each computing node to each computing node in the computing cluster, and each computing node will execute the pipeline task execution code in parallel based on its own input data subset.

[0133] In another embodiment shown, when the parallel task execution code is parallel optimized, in addition to splitting the input data set required to execute the computing task and allowing each computing node to execute the same parallel task execution code in parallel based on different input data subsets, the input data subsets of each computing node can be further split on this basis to obtain input data corresponding to each subtask included in the computing task.

[0134] For example, still taking the scenario in which the above-mentioned computing task is executed in parallel on each computing node in the above-mentioned computing cluster in the form of a pipeline task, in this parallel computing scenario, the deep learning compiler can also further perform data splitting on the input data subsets corresponding to each computing node in the computing cluster to obtain the input data of the execution code corresponding to each subtask contained in the pipeline task execution code.

[0135] Then, the deep learning compiler can use its own scheduling capabilities to schedule the pipeline task execution code and the input data of the execution code corresponding to each subtask contained in the pipeline task to each computing node in the computing cluster, and each computing node will execute the execution code corresponding to the above-mentioned subtasks contained in the pipeline task execution code in parallel based on the input data.

[0136] It should be noted that the data splitting method adopted by the deep learning compiler when splitting the above-mentioned input data set or the above-mentioned input data subset will not be specifically limited in this specification.

[0137] For example, in one embodiment shown, the deep learning compiler can specifically support SPMD (Single-Program Multi-Data) parallel strategy. The SPMD parallel strategy is a parallel strategy that allows the same program to run simultaneously on multiple processing units, but each processing unit operates on a different input data set.

[0138] In this parallel scenario, the above pipeline tasks and the subtasks included in the above pipeline tasks can be executed in parallel on the various computing nodes in the computing cluster using the SPMD parallel strategy.

[0139] On the one hand, the deep learning compiler can use the data splitting method supported by the SPMD parallel strategy to split the input data set required to execute the pipeline task to obtain input data subsets corresponding to each computing node in the computing cluster.

[0140] On the other hand, the deep learning compiler can also use the data splitting method supported by the SPMD parallel strategy to further split the input data subsets corresponding to each computing node in the computing cluster to obtain the input data of the execution code corresponding to each subtask contained in the pipeline task execution code.

[0141] Among them, the data splitting methods supported by the SPMD parallel strategy can usually include data splitting methods involved in parallel strategies such as TP (TensorParallelism), DP (Data Parallelism), and SP (SequentialParallelism). Among them, based on the TP strategy, the tensors involved in the data (such as weight matrices or activation values) are usually split. Based on the DP strategy, the data content itself is usually split; for example, the input data set is split into different batches. Based on the SP strategy, the sequences involved in the data are usually split.

[0142] In practical applications, when the deep learning compiler uses the data splitting method supported by the SPMD parallel strategy to further split the input data subsets corresponding to each computing node in the computing cluster, it can flexibly select a specific data splitting method based on specific needs; for example, any one or a combination of multiple of the TP, DP, and SP strategies shown above can be selected.

[0143] In one embodiment shown, when the parallel task execution code is optimized in parallel, in addition to further splitting the input data subsets of each computing node to obtain input data corresponding to each subtask included in the computing task, it is also possible to determine different execution times for each subtask included in the computing task on this basis, so that each subtask can be executed at different execution times on each computing node in a staggered manner.

[0144] For example, in one implementation shown, the scenario in which the computing task is executed in parallel on each computing node in the computing cluster in the form of a pipeline task is still taken as an example. In this parallel computing scenario, the deep learning compiler can determine the execution time on each computing node in the computing cluster for the execution code corresponding to each subtask contained in the compiled pipeline task execution code. Among them, the execution code corresponding to each subtask contained in the pipeline task execution code can correspond to different execution times on each computing node; of course, in actual applications, in order to make full use of the computing power of each computing node, only some of the subtasks contained in the pipeline task execution code can correspond to different execution times on each computing node.

[0145] Then, the deep learning compiler can use its own scheduling capabilities to schedule the pipeline task execution code and the input data of the execution code corresponding to each subtask contained in the pipeline task execution code to each computing node in the computing cluster. Each computing node will execute the execution code corresponding to each subtask contained in the pipeline task execution code in parallel based on the input data and the execution time determined for each subtask.

[0146] In this way, on the basis of using the SPMD parallel strategy to execute the subtasks contained in the above-mentioned pipeline task in parallel on each computing node in the computing cluster, each subtask can be further executed at different execution times on each computing node, thereby maximizing the use of the computing power of each computing node.

[0147] It should be noted that in practical applications, in addition to supporting the SPMD parallel strategy, the above-mentioned deep learning compiler can usually also support the MPMD (Multiple Programs Multiple Data) parallel strategy.

[0148] The MPMD parallel strategy is different from the SPMD parallel strategy in that this parallel strategy allows multiple different programs to run simultaneously on multiple processing units, and each processing unit operates on a different input data set.

[0149] If, on the basis of using the SPMD parallel strategy to execute the subtasks contained in the above pipeline task in parallel on each computing node in the computing cluster, the subtasks contained in the above pipeline task are further executed at different execution times on each computing node, then at the same time, each computing node can use different input data to execute the different subtasks contained in the above pipeline task in parallel. This parallel execution mode is obviously similar to the MPMD parallel strategy in effect.

[0150] It can be seen that in this way, on the basis of using the SPMD parallel strategy to execute the subtasks contained in the above-mentioned pipeline task in parallel on each computing node in the computing cluster, each subtask can be further executed on each computing node in a manner similar to the MPMD parallel strategy, thereby realizing a parallel execution mode that executes pipeline tasks in parallel on each computing node based on both the SPMD parallel strategy and the MPMD parallel strategy. In one embodiment shown, when the deep learning compiler determines the execution time on each computing node for the execution code corresponding to each subtask contained in the pipeline task execution code, the execution code corresponding to each subtask contained in the pipeline task execution code can be pipelined by running the pipeline arrangement algorithm supported by the deep learning compiler to perform pipeline arrangement processing to determine the execution time on each computing node for the execution code corresponding to each subtask.

[0151] For example, in one example, in order to make the execution codes corresponding to each subtask contained in the pipeline task execution code correspond to different execution times on each computing node, different time windows can be forced to be specified for different subtasks by configuring relevant computing parameters for the pipeline arrangement algorithm, so that the pipeline arrangement algorithm can only perform pipeline arrangement for the execution codes corresponding to each subtask contained in the pipeline task execution code within different time windows to determine the execution time on each computing node.

[0152] In one embodiment shown, when the deep learning compiler runs the pipeline scheduling algorithm and performs pipeline scheduling processing on the execution codes corresponding to the subtasks contained in the pipeline task execution code, it can also obtain the computing power status of each computing node in real time, and at least refer to the computing power status of each computing node to perform pipeline scheduling processing on the execution codes corresponding to each subtask contained in the pipeline task execution code, so as to determine the execution time of the execution code corresponding to each subtask on each computing node.

[0153] In this way, the computing power status of each computing node can be referenced in real time to determine the execution time of each subtask on each computing node, so as to maximize the use of the algorithms of each computing node.

[0154] It should be noted that the specific algorithm type of the pipeline arrangement algorithm supported by the deep learning compiler is not specifically limited in this specification. In practical applications, it can be flexibly selected based on specific needs.

[0155] In one embodiment shown, since different computing nodes may have different hardware configurations and computing environments, in order to ensure the execution efficiency and reliability of the parallel task execution code compiled by deep learning and scheduled to each computing node in the computing cluster when executed on each computing node, in actual applications, the runtime environment (Runtimeenvironment) for executing the parallel task execution code can also be customized for each computing node in the computing cluster.

[0156] For example, in actual applications, the specific hardware configuration of each computing node can be referenced to customize a runtime environment for each computing node that is adapted to its own hardware configuration. For computing nodes with different hardware configurations, the configuration of the customized runtime environment will usually also be different.

[0157] In this case, when the deep learning compiler schedules the pipeline task execution code, it can first obtain the configuration information of the runtime environment defined for each computing node, and then use its own scheduling capabilities to schedule the pipeline task execution code and the configuration information to each computing node in the computing cluster. After receiving the pipeline task execution code and the configuration information scheduled by the deep learning compiler, each computing node can configure the runtime environment locally based on the configuration information, and then execute the pipeline task execution code in parallel in the configured runtime environment.

[0158] In one embodiment shown, since each computing node may also involve some data communication operations that need to be performed between each computing node and data communication operations that need to be performed between each subtask in the process of executing the pipeline task execution code scheduled by the deep learning compiler, in order to ensure that these data communication operations can be normally implemented on each computing node, in actual applications, data communication operations can also be customized for the above-mentioned pipeline tasks. For example, the defined data communication operations may include a send operation for sending data (such as calculation results), a receive operation for receiving data, and so on.

[0159] In this case, after completing the parallel optimization of the pipeline task execution code, the deep learning compiler can also obtain the data communication operations defined for the pipeline task and compile these data communication operations into data communication execution codes; wherein the data communication operations can specifically include data communication operations performed between each computing node and data communication operations performed between each subtask during the execution of the parallel task execution code.

[0160] Then, the deep learning compiler can use its own scheduling capabilities to schedule the pipeline task execution code and the data communication execution code to each computing node in the computing cluster respectively, and each computing node executes the pipeline task execution code in parallel. In the process of executing the pipeline task execution code, the data communication execution code is executed to realize data communication between the computing nodes and data communication between the subtasks.

[0161] It should be noted that in practical applications, the above Figure 1 The method flow shown can be specifically applied to a deep learning compiler corresponding to a deep learning model.

[0162] The following will take the above-mentioned deep learning compiler as the XLA compiler as an example. Figure 1 The method flow shown is described in more detail.

[0163] See also Figure 2 , Figure 2 This is an architecture diagram of a deep learning system built based on the XLA compiler shown in this specification.

[0164] like Figure 2 As shown, the deep learning system can specifically be a multi-layer system architecture, which can specifically include a deep learning framework layer, a computational graph capture layer, a parallel optimization layer and a runtime environment layer.

[0165] Among them, the deep learning framework layer is specifically used to provide a deep learning framework that can be accessed by the XLA compiler;

[0166] For example, Figure 2 The deep learning frameworks that can be accessed by the XLA compiler provided in the deep learning framework layer shown include the Tensorflow framework, the PyTorch framework, the JAX framework, etc. Developers define deep learning models based on high-level programming languages ​​in these deep learning frameworks accessed by the XLA compiler.

[0167] The computational graph capture layer is specifically used to provide data structures for representing computational graph data related to various deep learning frameworks that can be accessed by the XLA compiler, and supports further representing the computational graph data in the form of HLO supported by the XLA compiler.

[0168] For example, Figure 2 The GraphDef shown is a data structure supported by the TensorFlow framework for representing computational graph data. Figure 2 TorchFX and Dynamo shown are data structures supported by the PyTorch framework for representing computational graph data. Figure 2The JaxPr shown is a data structure supported by the Jax framework for representing computational graph data. HLO is an intermediate representation supported by the XLA compiler.

[0169] The parallel optimization layer is specifically used to provide parallel optimization strategies supported by the XLA compiler for parallel optimization of computing tasks related to deep learning models. In practical applications, the XLA compiler can compile the computational graph data in the form of HLO generated by the computational graph data capture layer into parallel task execution code, and then perform parallel optimization on the parallel task execution code based on the parallel optimization strategies provided by the parallel optimization layer.

[0170] For example, Figure 2 The parallel optimization strategies supported by the XLA compiler shown may include SPMD parallel strategies and MPMD parallel strategies. Among them, the SPMD parallel strategy further supports specific parallel strategies such as TP, DP, SP, etc. In practical applications, the data splitting methods involved in parallel strategies such as TP, DP, SP, etc. can be flexibly adopted to split the input data set required for the execution of the deep learning model. The MPMD parallel strategy further supports the PP (Pipeline Parallelism) parallel strategy. In practical applications, the calculation graph data splitting method supported by the PP parallel strategy can also be adopted to split the calculation graph data into multiple sub-calculation graph data corresponding to multiple sub-tasks one by one, and then generate pipeline tasks based on the multiple sub-calculation graph data.

[0171] The runtime environment layer is specifically used to provide a customized runtime environment for the parallel task execution code after parallel optimization by the parallel optimization layer.

[0172] The following Figure 2 Based on the architecture of the deep learning system shown, a specific example is used to describe in detail the process of the XLA compiler automatically performing parallel execution optimization on computing tasks related to the deep learning model.

[0173] See also Figure 3 , Figure 3 This specification provides a flowchart of an XLA compiler that automatically optimizes the parallel execution of computing tasks related to deep learning models.

[0174] In this embodiment, a computing task related to a deep learning model is taken as an example of a training task for the deep learning model.

[0175] See also Figure 3Developers can select a suitable target deep learning framework from the deep learning frameworks accessible by the XLA compiler provided by the above deep learning framework layer according to specific development requirements, and define a deep learning model in the target deep learning framework based on a high-level programming language.

[0176] The target deep learning framework may be any one of the deep learning frameworks such as Tensorflow, PyTorch, and JAX that can be accessed by the XLA compiler provided by the deep learning framework layer.

[0177] Before the deep learning model defined in the selected target deep learning framework is deployed to the various computing nodes in the computing cluster connected to the XLA compiler for operation, the XLA compiler can simulate the execution of training tasks related to the deep learning model, and capture and record the computing operations generated in the process of simulating the execution of the training task and the data flow in the process of executing the computing operations, and then use the data structure related to the target deep learning framework provided by the above-mentioned graph capture layer to represent the recorded computing operations and data flows as computational graph data, and then further represent the computational graph data in the form of HLO supported by the XLA compiler.

[0178] For example, if the target deep learning framework is the TensorFlow framework, the recorded computing operations and data flows can be represented as computing graph data using the GraphDef data structure provided by the computing graph capture layer; if the target deep learning framework is the PyTorch framework, the recorded computing operations and data flows can be represented as computing graph data using the TorchFX or Dynamo data structure provided by the computing graph capture layer; if the target deep learning framework is the Jax framework, the recorded computing operations and data flows can be represented as computing graph data using the JaxPr data structure provided by the computing graph capture layer.

[0179] Among them, the calculations involved in the process of training the deep learning model generally include forward propagation calculations, back propagation calculations, and calculations for updating model parameters; therefore, the above-mentioned calculation graph data may specifically include forward propagation calculation graph data, back propagation calculation graph data, and model parameter update calculation graph data.

[0180] Please continue to see Figure 3In order to utilize the derivation and calculation capabilities of the deep learning framework itself, the XLA compiler can only use the data structure related to the target deep learning framework provided by the above-mentioned calculation graph capture layer to represent the forward propagation calculation graph data related to the training task, and further represent the forward propagation calculation graph data in the form of HLO supported by the XLA compiler. Then, the derivation and calculation capabilities of the target deep learning framework itself can be utilized to continue to derive the back-propagation calculation graph data and the model parameter update calculation graph based on the forward propagation calculation graph data, and the derived back-propagation calculation graph data and the model parameter update calculation graph are also represented in the form of HLO supported by the XLA compiler.

[0181] For example, taking the PyTorch framework as the target deep learning framework, a library called torchxla is usually provided in the PyTorch framework to support the functions of the XLA compiler. In this case, after the XLA compiler represents the forward propagation calculation graph data in the form of HLO based on the supported intermediate representation, on the one hand, the auto grad (automatic differentiation) capability of torchxla can be used in the PyTorch framework to automatically derive the back propagation calculation graph data based on the forward propagation calculation graph data; on the other hand, the apply grad (gradient application) capability of torchxla can be used in the PyTorch framework to automatically derive the model parameter update calculation graph data based on the back propagation calculation graph data, and then the derived back propagation calculation graph data and model parameter update calculation graph data are also represented in the form of HLO supported by the XLA compiler.

[0182] In addition to expressing the computational graph data related to the training task in the form of HLO supported by the XLA compiler, the XLA compiler can also automatically perform parallel optimization calculations on the computational graph data by running supported optimization algorithms to solve the optimal stage division method for the training task; for example, Figure 3 It is shown that the training task is divided into three execution stages, namely stage 1, stage 2 and stage 3. Then, the boundary information of stage 1, stage 2 and stage 3 on the computational graph data is represented in the form of HLO supported by the XLA compiler, and the boundary information is automatically annotated on the computational graph data in the form of labels or tags.

[0183] Of course, in practical applications, Figure 3The stage1, stage2, and stage3 shown above may also be stages manually divided by the developer. In this case, the developer may manually annotate the boundary information of stage1, stage2, and stage3 on the computational graph data in the form of labels or marks based on the HLO format supported by the XLA compiler.

[0184] Please continue to see Figure 3 After the XLA compiler represents the computation graph data of the training task in the form of HLO supported by the XLA compiler, it can obtain the boundary information in the form of HLO marked on the computation graph data, and based on the boundary information, split the forward propagation computation graph data, the back propagation computation graph data, and the model parameter update computation graph data into a number of sub-computation graph data corresponding to each stage.

[0185] Then, based on the above-mentioned several sub-computation graph data, pipeline tasks consisting of stage1, stage2 and stage3 executed in sequence are generated for each computing node in the computing cluster, and the generated pipeline tasks are compiled into pipeline task execution codes corresponding to each computing node in the above-mentioned computing cluster.

[0186] For example, the generated pipeline tasks may specifically include pipeline tasks related to forward propagation calculations, pipeline tasks related to reverse forward propagation calculations, and pipeline tasks related to model parameter update calculations.

[0187] After the XLA compiler compiles the generated pipeline tasks into pipeline task execution codes corresponding to the respective computing nodes in the computing cluster, it can also utilize the parallel optimization strategy provided by the parallel optimization layer to perform parallel optimization on the pipeline task execution codes.

[0188] On the one hand, the XLA compiler can use the SPMD parallel strategy provided by the parallel optimization layer to perform parallel optimization on the pipeline task execution code.

[0189] Specifically, based on the data splitting method supported by the SPMD parallel strategy, the input data set required to execute the pipeline task can be split to obtain input data subsets corresponding to each computing node in the computing cluster.

[0190] For example, assuming that the computing cluster connected to the XLA compiler includes computing node 1, computing node 2, and computing node 3, the input data set required to execute the pipeline task can be split into three data subsets and assigned to computing node 1, computing node 2, and computing node 3 respectively, so that these three computing nodes can use different input data subsets to execute the pipeline task in parallel using the SPMD parallel strategy.

[0191] In addition, the XLA compiler can also use the data splitting method supported by the SPMD parallel strategy to further split the input data subsets corresponding to each computing node in the computing cluster to obtain the input data of each stage contained in the pipeline task.

[0192] Furthermore, based on the data splitting method supported by the SPMD parallel strategy, data splitting can be performed inside each stage included in the pipeline task, so as to further split the input data of each stage into input data corresponding to each sub-computation included in the stage.

[0193] For example, in actual applications, each stage may contain a large number of sub-calculations, which may include sub-calculations that can be executed in parallel or sub-calculations that need to be executed sequentially. Figure 3 As shown in the figure, assuming that stage1 contained in the pipeline task can contain two sub-calculations that can be executed in parallel, then for stage1, a horizontal splitting method can be used to further split the input data subset of the computing node into part1 (i.e. Figure 3 P1) and part2 (i.e. Figure 3 P2) is used as the input data of the two sub-calculations in stage1 that can be executed in parallel. Similarly, the input data of stage2 is the output data of stage1. Assuming that stage2 can contain two sub-calculations that need to be executed sequentially, the output data of stage1 can be further split into part3 (i.e. Figure 3 P3) and part4 (i.e. Figure 3 Furthermore, the input data of stage3 is the output data of stage2. Assuming that stage3 can contain three sub-calculations that can be executed in parallel, the output data of stage2 can be further split into part5 (i.e. Figure 3 P5 shown in), part6 (i.e. Figure 3 P6) and part7 (i.e. Figure 3 P7 shown in FIG.

[0194] It should be noted that Figure 3 The specific method of dividing data within each stage shown above is only illustrative. In actual applications, the division can be flexibly performed according to specific needs, and will not be listed one by one in this specification.

[0195] On the other hand, the XLA compiler, while adopting the data splitting method supported by SPMD to split the input data set required to execute the pipeline task and the input data subset corresponding to each computing node, can also adopt the MPMD parallel strategy provided by the above-mentioned parallel optimization layer to parallel optimize the above-mentioned pipeline task execution code, thereby realizing a parallel execution mode on each computing node that executes pipeline tasks in parallel based on both the SPMD parallel strategy and the MPMD parallel strategy.

[0196] Specifically, the XLA compiler can run the supported pipeline arrangement algorithm to perform pipeline arrangement processing on the execution code corresponding to each stage contained in the pipeline task execution code, so as to determine different execution times for the execution code corresponding to each stage, so that each stage contained in the pipeline task can be executed at different execution times on each computing node in a staggered manner.

[0197] For example, Figure 3 As shown in the figure, the generated pipeline tasks can be pipeline tasks related to forward propagation calculation, pipeline tasks related to reverse forward propagation calculation, and pipeline tasks related to model parameter update calculation. Different execution times can be determined for different stages contained in these three pipeline tasks. For example, Figure 3 In the figure, the execution times T1, T2 and T3 are determined for stage1, stage2 and stage3 included in the pipeline tasks related to the forward propagation calculation, respectively; the execution times T4, T5 and T6 are determined for stage1, stage2 and stage3 included in the pipeline tasks related to the backward propagation calculation, respectively; the execution times T7, T8 and T9 are determined for stage1, stage2 and stage3 included in the pipeline tasks related to the model parameter update, respectively. This allows the various stages included in the above three pipeline tasks to be executed at different execution times on the computing nodes at different peak times.

[0198] Since the execution code corresponding to each stage is executed on each computing node at different execution times, it is equivalent to that at the same time, each computing node can use different input data to execute different stages in parallel; therefore, this parallel execution mode is obviously similar to the MPMD parallel strategy in effect.

[0199] After completing the parallel optimization for the pipeline task execution code, the XLA compiler can also obtain the data communication operations defined for the pipeline task and compile these data communication operations into data communication execution code; wherein the data communication operations can specifically include data communication operations performed between various computing nodes and data communication operations performed between various subtasks during the execution of the parallel task execution code. For example, Figure 3 As shown, the defined data communication operations may include a send operation for sending data and a receive operation for receiving data.

[0200] Then, the XLA compiler can use its own scheduling capabilities to schedule the pipeline task execution code, the data communication execution code, and the input data of the execution code corresponding to each stage contained in the pipeline task execution code to each computing node in the computing cluster.

[0201] Please continue to see Figure 3 , each computing node can execute the execution code corresponding to each stage contained in the pipeline task execution code in parallel in the customized runtime environment provided by the above-mentioned runtime environment layer based on the input data and the execution time determined for each subtask, and in the process of executing the pipeline task execution code, by executing the data communication execution code, implement data communication operations such as send operations and receive operations between each computing node and between each stage.

[0202] In the above technical scheme, the intermediate representation supported by the deep learning compiler is used to represent the parallel computing mode corresponding to the computing tasks related to the deep learning model, so that the deep learning compiler can clarify the parallel computing mode adopted for parallel optimization of the above computing tasks by parsing the parallel computing mode represented by the intermediate representation, so that the deep learning compiler can utilize its own optimization capability to further compile the computing graph data of the computing task into parallel task execution code based on the parallel computing mode, and execute it in parallel on each computing node in the computing cluster connected to the deep learning compiler, thereby realizing automatic parallel optimization of the computing tasks of the deep learning model.

[0203] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of an apparatus, an electronic device, and a storage medium.

[0204] Figure 4 is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Figure 4 At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410, and may also include other required hardware. One or more embodiments of this specification may be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 710 into the memory 408 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but may also be hardware or logic devices.

[0205] like Figure 5 As shown, Figure 5 is a block diagram of a parallel execution device for a deep learning model according to an exemplary embodiment of the present specification, and the device can be operated in Figure 4 In the electronic device shown in the figure, the technical solution of this specification is implemented. The device 500 includes:

[0206] A representation module 801 represents a computing task related to a deep learning model as a computational graph data; wherein the computational graph data is a computational graph data represented by an intermediate representation supported by a deep learning compiler corresponding to the deep learning model;

[0207] A compiling module 802 is configured to obtain a parallel computing method corresponding to the computing task represented by the intermediate representation, and compile the computing graph data into a parallel task execution code corresponding to each computing node in a computing cluster based on the parallel computing method; the computing cluster is used to execute computing tasks related to the deep learning model;

[0208] The scheduling module 803 schedules the parallel task execution codes corresponding to the respective computing nodes to the respective computing nodes in the computing cluster, so that the respective computing nodes execute the parallel task execution codes in parallel.

[0209] Accordingly, the present specification also provides an electronic device, which includes a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the steps in all the method flows described above.

[0210] Accordingly, the present specification also provides a computer-readable storage medium on which executable computer program instructions are stored; wherein, when the instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0211] Accordingly, the present specification also provides a computer program product having executable computer program instructions stored thereon; wherein, when the computer program instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0212] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0213] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0214] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, the present application does not exclude that with the development of computer technology in the future, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0215] Although one or more embodiments of the present specification provide method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any specific order.

[0216] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more of the present specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0217] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0218] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0219] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0220] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0221] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0222] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0223] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0224] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0225] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0226] The above description is only an example of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included in the scope of the claims.

Claims

1. A method for parallel execution of a deep learning model, the method comprising: Representing a computing task related to a deep learning model as computing graph data; wherein the computing graph data is computing graph data represented by an intermediate representation supported by a deep learning compiler corresponding to the deep learning model; Obtaining a parallel computing mode corresponding to the computing task represented by the intermediate representation, and compiling the computing graph data into parallel task execution codes corresponding to each computing node in a computing cluster based on the parallel computing mode; the computing cluster is used to execute computing tasks related to the deep learning model; The parallel task execution codes corresponding to the respective computing nodes are respectively dispatched to the respective computing nodes in the computing cluster, and the respective computing nodes execute the parallel task execution codes in parallel.

2. The method of claim 1, wherein the computing tasks associated with the deep learning model are represented as computing graph data, comprising: Simulate and execute computing tasks related to the deep learning model, and record computing operations generated during the simulation of executing the computing tasks and data flows during the execution of the computing operations; Based on an intermediate representation supported by the deep learning compiler, the recorded computing operations and the data flows are represented as computing graph data.

3. The method according to claim 1, wherein obtaining the parallel computing mode corresponding to the computing task represented by the intermediate representation comprises: The optimization algorithm supported by the deep learning compiler is run to automatically perform parallel optimization calculations on the computation graph data to obtain an optimal parallel computing method corresponding to the computation graph data, and the optimal parallel computing method is represented based on the intermediate representation as a parallel computing method corresponding to the computing task.

4. The method of claim 3, wherein the computing task is executed in parallel on each computing node in the computing cluster in the form of a pipeline task; the pipeline task includes a plurality of subtasks executed in sequence; the parallel computing method includes boundary information on the computing graph data based on the plurality of subtasks represented by the intermediate representation; the boundary information is marked on the computing graph data; Acquiring a parallel computing mode corresponding to the computing task represented by the intermediate representation, including: The boundary information marked on the computation graph data and represented based on the intermediate representation is obtained.

5. The method according to claim 4, Compiling the computation graph data into parallel task execution codes corresponding to each computing node in the computing cluster based on the parallel computing method includes: Based on the boundary information, the computation graph data is divided into a plurality of sub-computation graph data corresponding to the plurality of subtasks one by one; Based on the several sub-computation graph data, pipeline tasks are generated for each computing node in the computing cluster, and the pipeline tasks are compiled into pipeline task execution codes; wherein the pipeline task is a pipeline task composed of the several subtasks in an execution order; the pipeline task execution code includes execution codes corresponding to the several subtasks.

6. The method according to claim 5, Dispatching the parallel task execution codes corresponding to the respective computing nodes to the respective computing nodes in the computing cluster, and having the respective computing nodes execute the parallel task execution codes in parallel, including: Acquire an input data set required for executing the pipeline task, and perform data splitting on the input data set to obtain input data subsets corresponding to each computing node in the computing cluster; wherein the input data subsets corresponding to different computing nodes contain different input data; The pipeline task execution code and the input data subsets corresponding to the respective computing nodes are respectively scheduled to the respective computing nodes in the computing cluster, and the respective computing nodes execute the pipeline task execution code in parallel based on the input data subsets.

7. The method of claim 6, further comprising: Further performing data splitting on the input data subsets corresponding to the respective computing nodes in the computing cluster to obtain input data of the execution codes corresponding to the respective subtasks contained in the pipeline task execution code; The step of dispatching the pipeline task execution code and the input data subsets corresponding to the respective computing nodes to the respective computing nodes in the computing cluster, and having the respective computing nodes execute the pipeline task execution code in parallel based on the input data subsets, comprises: The pipeline task execution code and the input data of the execution code corresponding to each subtask contained in the pipeline task execution code are respectively scheduled to each computing node in the computing cluster, and each computing node executes the execution code corresponding to each subtask contained in the pipeline task execution code in parallel based on the input data.

8. The method according to claim 6 or 7, wherein the deep learning compiler supports the SPMD parallel strategy; Performing data splitting on the input data set or the input data subset includes: The input data set or the input data subset is split using a data splitting method supported by the SPMD parallel strategy.

9. The method according to claim 7, wherein the pipeline task execution code and the input data of the execution code corresponding to each subtask contained in the pipeline task execution code are respectively dispatched to each computing node in the computing cluster, and each computing node executes the execution code corresponding to each subtask contained in the pipeline task execution code in parallel based on the input data, comprising: Determine execution times on the respective computing nodes for execution codes corresponding to the respective subtasks contained in the pipeline task execution code; wherein at least part of the execution codes corresponding to the respective subtasks contained in the pipeline task execution code correspond to different execution times on the respective computing nodes; The pipeline task execution code and the input data of the execution code corresponding to each subtask contained in the pipeline task execution code are respectively scheduled to each computing node in the computing cluster, and each computing node executes the execution code corresponding to each subtask contained in the pipeline task execution code in parallel based on the input data and the execution time.

10. The method according to claim 9, wherein the execution code corresponding to each subtask contained in the pipeline task execution code is used to determine the execution time on each computing node, respectively, including: Run the pipeline arrangement algorithm supported by the deep learning compiler, and perform pipeline arrangement processing on the execution code corresponding to each subtask contained in the pipeline task execution code to determine the execution time of the execution code corresponding to each subtask on each computing node.

11. The method according to claim 10, running the pipeline arrangement algorithm supported by the deep learning compiler, performing pipeline arrangement processing on the execution codes corresponding to the respective subtasks contained in the pipeline task execution code, so as to determine the execution time of the execution codes corresponding to the respective subtasks on the respective computing nodes, comprising: Run the pipeline arrangement algorithm supported by the deep learning compiler, at least refer to the computing power status of each computing node, and perform pipeline arrangement processing on the execution code corresponding to each subtask contained in the pipeline task execution code to determine the execution time of the execution code corresponding to each subtask on each computing node.

12. The method according to claim 9, wherein the parallel task execution codes corresponding to the respective computing nodes are respectively dispatched to the respective computing nodes in the computing cluster, and the respective computing nodes execute the parallel task execution codes in parallel, comprising: Obtaining configuration information of a runtime environment defined for each computing node; The pipeline task execution code and the configuration information are respectively dispatched to each computing node in the computing cluster, and each computing node configures a runtime environment based on the configuration information, and executes the pipeline task execution code in parallel in the configured runtime environment.

13. The method according to claim 9, wherein the parallel task execution codes corresponding to the respective computing nodes are respectively dispatched to the respective computing nodes in the computing cluster, and the respective computing nodes execute the parallel task execution codes in parallel, comprising: Acquire a data communication operation defined for the pipeline task, and compile the data communication operation into a data communication execution code; wherein the data communication operation includes a data communication operation performed between the computing nodes and a data communication operation performed between the subtasks during the execution of the pipeline task execution code; The pipeline task execution code and the data communication execution code are respectively scheduled to each computing node in the computing cluster, and the pipeline task execution code is executed in parallel by each computing node. In the process of executing the pipeline task execution code, the data communication execution code is executed to realize data communication between the computing nodes and data communication between the subtasks.

14. The method of claim 1, wherein the deep learning model is a deep learning model defined in a deep learning framework accessed by the deep learning compiler.

15. The method of claim 14, wherein the deep learning framework comprises any one of the following: Tensorflow framework, PyTorch framework, JAX framework.

16. The method of claim 2, wherein the computing task comprises a training task for the deep learning model; Correspondingly, the computational graph data includes computational graph data related to the forward propagation calculation in the training task, computational graph data related to the back propagation calculation in the training task, and computational graph data related to the model parameter update calculation in the training task.

17. The method of claim 1, wherein the deep learning compiler comprises an XLA compiler.

18. A parallel execution device for a deep learning model, the device comprising: A representation module that represents a computing task related to a deep learning model as a computational graph data; wherein the computational graph data is a computational graph data represented by an intermediate representation supported by a deep learning compiler corresponding to the deep learning model; A compiling module, which obtains a parallel computing mode corresponding to the computing task represented by the intermediate representation, and compiles the computing graph data into a parallel task execution code corresponding to each computing node in a computing cluster based on the parallel computing mode; the computing cluster is used to execute computing tasks related to the deep learning model; The scheduling module schedules the parallel task execution codes corresponding to the respective computing nodes to the respective computing nodes in the computing cluster, so that the respective computing nodes execute the parallel task execution codes in parallel.

19. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 17 are implemented.

20. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 17.

21. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

Images