A model processing method, apparatus, device, and computer-readable storage medium

Abstract

The present disclosure relates to a model processing method, device, equipment and computer readable storage medium. The present disclosure obtains a first computation graph corresponding to a to-be-trained model and a parallelization strategy of the to-be-trained model, the parallelization strategy of the to-be-trained model including at least one of pipelining parallelism, model parallelism, data parallelism and operator splitting, adds parallelization information in the first computation graph according to the parallelization strategy of the to-be-trained model, obtains a second computation graph, determines a distributed computation graph according to the second computation graph and a computing resource, and trains the to-be-trained model according to the distributed computation graph. The present disclosure realizes the technology based on computation graph editing to support multiple parallelization strategies, so that multiple parallelization strategies can be integrated in a set of systems, thereby realizing a distributed training framework capable of supporting multiple parallelization strategies.

Description

Technical Field

[0001] This disclosure relates to the field of information technology, and in particular to a model processing method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] With the development of deep neural networks, the number of parameters in deep learning models, neural network models, and other models is constantly increasing, but the update speed of the hardware used to train the models cannot keep up with the iteration speed of the models.

[0003] Existing technologies employ distributed training methods to train models. Common distributed training methods include data parallelism, model parallelism, pipelined parallelism, operator splitting, and hybrid parallelism. Hybrid parallelism can be a combination of two or more of data parallelism, model parallelism, pipelined parallelism, and operator splitting. However, existing distributed training frameworks cannot support all parallelism strategies and their combinations. Summary of the Invention

[0004] To address the aforementioned technical problems, or at least partially address them, this disclosure provides a model processing method, apparatus, device, and computer-readable storage medium to implement a distributed training framework capable of supporting multiple parallelization strategies.

[0005] In a first aspect, embodiments of this disclosure provide a model processing method, including:

[0006] Obtain the first computation graph corresponding to the model to be trained, and the parallelization strategy of the model to be trained, wherein the parallelization strategy of the model to be trained includes at least one of pipeline parallelism, model parallelism, data parallelism and operator splitting.

[0007] Parallelization information is added to the first computation graph according to the parallelization strategy of the model to be trained, to obtain the second computation graph;

[0008] Based on the second computation graph and computational resources, a distributed computation graph is determined;

[0009] The model to be trained is trained based on the distributed computing graph.

[0010] Secondly, embodiments of this disclosure provide a model processing apparatus, including:

[0011] The acquisition module is used to acquire the first computation graph corresponding to the model to be trained and the parallelization strategy of the model to be trained. The parallelization strategy of the model to be trained includes at least one of pipeline parallelism, model parallelism, data parallelism and operator splitting.

[0012] An addition module is used to add parallelization information to the first computation graph according to the parallelization strategy of the model to be trained, so as to obtain a second computation graph.

[0013] The determination module is used to determine the distributed computing graph based on the second computing graph and computing resources;

[0014] The training module is used to train the model to be trained based on the distributed computing graph.

[0015] Thirdly, embodiments of this disclosure provide a model processing apparatus, including:

[0016] Memory;

[0017] Processor; and

[0018] Computer programs;

[0019] The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in the first aspect.

[0020] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method described in the first aspect.

[0021] The model processing method, apparatus, device, and computer-readable storage medium provided in this disclosure acquire a first computation graph corresponding to the model to be trained and a parallelization strategy of the model to be trained. The parallelization strategy of the model to be trained includes at least one of pipelining parallelism, model parallelism, data parallelism, and operator splitting. Parallelization information is added to the first computation graph according to the parallelization strategy of the model to be trained to obtain a second computation graph. A distributed computation graph is determined according to the second computation graph and computing resources. The model to be trained is trained according to the distributed computation graph. This realizes a technique based on computation graph editing to support multiple parallelization strategies, enabling multiple parallelization strategies to be integrated into a single system, thereby realizing a distributed training framework that can support multiple parallelization strategies. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0023] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A schematic diagram illustrating data parallelism provided for embodiments of this disclosure;

[0025] Figure 2 A schematic diagram illustrating model parallelism provided for embodiments of this disclosure;

[0026] Figure 3 A schematic diagram of pipelined parallelism provided in an embodiment of this disclosure;

[0027] Figure 4 Another schematic diagram of pipelined parallelism provided in this embodiment of the disclosure;

[0028] Figure 5 A schematic diagram of operator decomposition provided in an embodiment of this disclosure;

[0029] Figure 6 A flowchart of the model processing method provided in the embodiments of this disclosure;

[0030] Figure 7 A schematic diagram illustrating an application scenario provided by an embodiment of this disclosure;

[0031] Figure 8 A schematic diagram illustrating another application scenario provided by an embodiment of this disclosure;

[0032] Figure 9 A flowchart of a model processing method provided in another embodiment of this disclosure;

[0033] Figure 10 A schematic diagram illustrating another application scenario provided by an embodiment of this disclosure;

[0034] Figure 11 A schematic diagram illustrating another application scenario provided by an embodiment of this disclosure;

[0035] Figure 12 A schematic diagram of the distributed training framework provided in the embodiments of this disclosure;

[0036] Figure 13 A schematic diagram illustrating the model parameter dimensions provided in the embodiments of this disclosure;

[0037] Figure 14 A schematic diagram of a method for partitioning a first computational graph provided in an embodiment of this disclosure;

[0038] Figure 15 A schematic diagram illustrating the method for partitioning virtual devices provided in this embodiment of the disclosure;

[0039] Figure 16 A schematic diagram illustrating another application scenario provided by an embodiment of this disclosure;

[0040] Figure 17This is a schematic diagram of the structure of the model processing apparatus provided in the embodiments of this disclosure;

[0041] Figure 18 This is a schematic diagram of the structure of the model processing device provided in an embodiment of this disclosure. Detailed Implementation

[0042] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0043] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0044] Typically, distributed training methods are used to train models. Common distributed training methods include data parallelism, model parallelism, pipelined parallelism, operator splitting, and other parallelization strategies. However, existing distributed training frameworks cannot support various parallelization strategies and their combinations. To address this issue, this disclosure provides a model processing method, which will be described below with reference to specific embodiments.

[0045] In the embodiments of this disclosure, the parallelization strategy can also be called a parallel strategy, specifically a collection of distributed parallel methods. For example, parallelization strategies include: data parallelism, model parallelism, pipelined parallelism, operator splitting, and hybrid parallelism. Hybrid parallelism can be a combination of two or more of data parallelism, model parallelism, pipelined parallelism, and operator splitting. Data parallelism, model parallelism, pipelined parallelism, and operator splitting will be described in detail below.

[0046] Data parallelism manifests as follows: each of multiple devices loads an identical copy of the model; that is, each device trains the same model. However, the sample data used to train the model differs across devices. For example, the sample data used for training the model in different devices may be different subsets of data. The set of data subsets from each device constitutes the complete set used to train the model. Furthermore, each device can synchronize model parameters across copies at the end of the iteration. Figure 1The diagram illustrates data parallelism. For example, Model 10 and Model 11 are the same model. Device 0 is used to train Model 10, and Device 1 is used to train Model 11. In one iteration of training, the sample data used by Device 0 to train Model 10 is denoted as input1, and the sample data used by Device 1 to train Model 11 is denoted as input2. Input1 and input2 can be different. Device 0 can output a training result based on input1, which may be, for example, a gradient. Similarly, Device 1 can output a training result based on input2, which may also be, for example, a gradient. Since the training results output by Device 0 and Device 1 may differ, they can be aggregated to obtain an aggregated result. Furthermore, the parameters of Model 10 and Model 11 can be updated based on this aggregated result, making the parameters of Model 10 and Model 11 identical, and then the next iteration of training can begin.

[0047] Model parallelism manifests as different devices across multiple devices being used to train different layers of a model. For example, a model may include multiple layers (e.g., network layers), with different devices responsible for the computation of different layers; that is, different layers of the model are assigned to different devices. Specifically, one or more layers of the model can be assigned to the same device. Figure 2 The diagram illustrates model parallelism. For example, the model includes layers 1, 2, 3, and 4. Layers 1 and 2 can be assigned to device 0, and layers 3 and 4 can be assigned to device 1. The input to device 0 is sample data, and the output of device 0 can be used as the input to device 1.

[0048] Pipeline parallelism is an auxiliary parallel strategy that can be used alone or in combination with model parallelism.

[0049] Figure 3The diagram illustrates the standalone use of pipelined parallelism. For example, 30 represents the model, which is trained by a device, and 31 represents the dataset used to train the model. This dataset can include a large amount of sample data. Since a large amount of sample data consumes significant storage space and computational resources, when using pipelined parallelism alone, the dataset 31 can be divided into multiple smaller partitions. For example, 32 represents any one of these partitions. Furthermore, the device can sequentially input each partition into the model 30 to train it. Each partition input into the model 30 yields a training result, which may be, for example, a gradient. For instance, if the dataset 31 is divided into 10 partitions, and these 10 partitions are sequentially input into the model 30 to train it, 10 training results can be obtained. Further processing of these 10 training results yields the model parameters.

[0050] Figure 4 The diagram illustrates a hybrid approach combining pipelined parallelism and model parallelism. Figure 4 As shown, model 40 includes layers 1, 2, 3, and 4. Based on the previously described model parallelism, layers 1, 2, 3, and 4 can be assigned to different devices for computation. For example, device 1 is responsible for layer 1 computation, device 2 for layer 2 computation, device 3 for layer 3 computation, and device 4 for layer 4 computation. It is understandable that... Figure 4 The illustrated hybrid use of pipelined parallelism and model parallelism is merely illustrative and not intended to impose specific limitations. In the case of a hybrid use of pipelined parallelism and model parallelism, the input data 41 of layer 1 can be divided into multiple partitions, for example, F... 0,1 and F 1,1 Each represents a fragment. Device 1 can process multiple fragments sequentially. For example, device 1 can process fragment F first. 0,1 Equipment 1 pair of F-slices 0,1 The processing result can be denoted as F. 0,2 F 0,2 It can be used as input to layer 2, that is, device 1 can input F. 0,2 Send to device 2. Similarly, device 2 sends to F. 0,2 The processing result obtained after processing can be denoted as F. 0,3 Device 2 can F 0,3 Send to device 3, F 0,3 As input to layer 3. Similarly, device 3 to F 0,3 The processing result obtained after processing can be denoted as F. 0,4 Device 3 can F 0,4 Send to device 4, F 0,4 As input to layer 4. It is understandable that when device 2 inputs to F...0,2 While processing, device 1 can also process F. 0,1 The next slice is F 1,1 Processing, F 1,1 The flow of the processing results and subsequent processing are similar to F. 0,1 This will not be elaborated further here. In other words, by using a combination of pipelined parallelism and model parallelism, the devices corresponding to different layers in Model 40 can perform parallel computations. For example, at time t, devices 1, 2, 3, and 4 can perform parallel computations simultaneously, thereby improving computational efficiency.

[0051] Operator decomposition specifically means that each layer of the model includes one or more operators. The operators in each layer are used to train some of the model's parameters. Furthermore, the number of parameters in different layers can be different or the same. Figure 5 The diagram illustrates one method of operator splitting. For example, the storage and computation portions of the operator in each layer are split and stored and computed on device 0 and device 1, respectively. In some embodiments, any one or more layers in the model can also be split. The storage portion of the operator can be the parameters used to train the operator.

[0052] The devices mentioned above can specifically be computing devices such as graphics processing units (GPUs) or central processing units (CPUs).

[0053] It is understood that the model described above can be a neural network model, a deep learning model, or other types of models. In other words, this disclosure does not specifically limit the model. Deep learning is a branch of machine learning, an algorithm that uses artificial neural networks as its architecture to learn representations of data. Furthermore, a deep learning model can also be called a deep neural network model; specifically, a neural network model with three or more layers can be a deep learning model.

[0054] Figure 6 A flowchart of the model processing method provided in this embodiment of the disclosure. The specific steps of the method are as follows:

[0055] S601. Obtain the first computation graph corresponding to the model to be trained and the parallelization strategy of the model to be trained. The parallelization strategy of the model to be trained includes at least one of pipeline parallelism, model parallelism, data parallelism and operator splitting.

[0056] In this embodiment, the model to be trained can be, for example, a neural network model or a deep learning model. This model can be a user's single-machine, single-card model. Furthermore, the model in this embodiment can be applied in fields such as computer vision, natural language processing, and knowledge graphs.

[0057] The model processing method described in this embodiment can be executed by a distributed training framework. This distributed training framework can be deployed on one or more machines, which can specifically include computers, computer clusters, servers, etc., where servers can specifically include cloud servers, remote servers, etc. Each machine can include multiple devices, such as computing devices like GPUs or CPUs.

[0058] In one possible application scenario, this distributed training framework is deployed on a cloud server. It is understood that this embodiment does not limit the number of cloud servers; for example, it can be one or multiple. Here, one server is used as an example for illustrative purposes. Figure 7 The cloud server 70 is shown. The model to be trained is stored on the user terminal 71. This embodiment does not limit the specific product form of the terminal 71; for example, it can be a laptop, desktop computer, tablet computer, personal computer (PC), etc. Specifically, the terminal 71 can send the model to be trained to the cloud server 70. After receiving the model, the cloud server 70 generates a computation graph corresponding to the model, which is denoted as the first computation graph. The first computation graph can be the user's original single-machine, single-card computation graph. Alternatively, the terminal 71 can generate the first computation graph based on the local model to be trained and send the first computation graph to the cloud server 70. The cloud server 70 can use this distributed training framework to train the model and feed back the training results to the terminal 71.

[0059] In another possible application scenario, cloud server 70 can send the distributed training framework to terminal 71, which can then deploy the distributed training framework on itself or other devices. For example, after the distributed training framework is deployed on terminal 71, terminal 71 can generate a first computation graph based on the local model to be trained, use this first computation graph as input to the distributed training framework, train the model to be trained through the distributed training framework, and output the training results. The distributed training framework described in this embodiment can support multiple parallelization strategies. For example, it can support not only pipelined parallelism, model parallelism, data parallelism, or operator splitting, but also combinations of two or more of these strategies. In other words, the distributed training framework in this embodiment can support not only a single parallelization strategy among pipelined parallelism, model parallelism, data parallelism, and operator splitting, but also hybrid parallelism. Therefore, the distributed training framework can train the model to be trained using not only a single parallelization strategy, but also a hybrid parallelism approach.

[0060] Furthermore, when the distributed training framework obtains the first computation graph of the model to be trained, it can further obtain the parallelization strategy of the model to be trained. The parallelization strategy of the model to be trained can be at least one of pipelining parallelism, model parallelism, data parallelism, and operator splitting. That is to say, the parallelization strategy of the model to be trained can be a single parallelization strategy or a hybrid parallelization strategy.

[0061] In one possible implementation, obtaining the parallelization strategy of the model to be trained includes: determining the parallelization strategy of the model to be trained based on the first computation graph corresponding to the model to be trained.

[0062] For example, when the distributed training framework obtains the first computational graph of the model to be trained, it can determine the parallelization strategy of the model based on the first computational graph. For example, the distributed training framework can analyze the first computational graph using machine learning methods such as reinforcement learning to determine the feature information of the model to be trained, and then determine the parallelization strategy of the model to be trained based on the feature information.

[0063] In another possible implementation, obtaining the parallelization strategy of the model to be trained includes: obtaining the parallelization strategy of the model to be trained selected by the user.

[0064] For example, this distributed training framework provides a user interface through which users can select the parallelization strategy for the model to be trained. Specifically, the user interface offers pipelined parallelism, model parallelism, data parallelism, and operator splitting, allowing users to choose one or more of these as the parallelization strategy for the model to be trained. Furthermore, as a possible implementation, when the distributed training framework obtains the parallelization strategy selected by the user, it can also calculate the computational resources, cost, time, performance, and other metrics required for that selected parallelization strategy or combinations of various parallelization strategies. Further, the distributed training framework can feed this metrics back to the user so that the user can adjust or confirm the selected parallelization strategy.

[0065] S602. Parallelization information is added to the first computation graph according to the parallelization strategy of the model to be trained, to obtain the second computation graph.

[0066] For example Figure 8 The number 81 in the diagram represents the first computation graph corresponding to the model to be trained. 811, 812, 813, and 814 represent different layers included in the model to be trained, respectively. The distributed training framework can add parallelization information to the first computation graph 81 based on the parallelization strategy of the model to be trained, resulting in a second computation graph 82. This parallelization information is related to the parallelization strategy of the model to be trained. For example, the parallelization strategy of the model to be trained might be: model parallelism between the first and second parts of the model, data parallelism within the first part, and operator splitting within the second part. This parallelization information can include the parallelization strategy of the model to be trained. Additionally, this parallelization information can also include parameter information of the parallelization strategy, such as the number of devices required for data parallelism, the number of objects to be split into by operator splitting, or the number of fragments.

[0067] Optionally, adding parallelization information to the first computation graph according to the parallelization strategy of the model to be trained to obtain a second computation graph includes: dividing the first computation graph into multiple first subgraphs according to the parallelization strategy of the model to be trained; and adding parallelization information to each of the multiple first subgraphs according to the parallelization strategy of the model to be trained to obtain a second computation graph.

[0068] For example, a distributed training framework can divide the first computation graph into multiple subgraphs based on the parallelization strategy of the model to be trained. Each subgraph can include one or more layers of the model to be trained. For example, the distributed training framework divides the first computation graph into... Figure 8 The child shown Figure 1 Kazuko Figure 2 ,son Figure 1 Kazuko Figure 2 These are denoted as the first subgraph. For example, subgraph... Figure 1 Corresponding to the second part of the model to be trained, sub Figure 2 This corresponds to the first part of the model to be trained. Furthermore, based on the parallelization strategy of the model to be trained, in the sub-... Figure 1 Kazuko Figure 2 Parallelization information is added to each subgraph in the graph to obtain the second computation graph 82.

[0069] Optionally, the parallelization information includes parallelization information between different first subgraphs and parallelization information within each first subgraph.

[0070] For example, distributed training frameworks in sub- Figure 1 The added parallelization information can include sub Figure 1 The parallelization information within the content, and may also include sub-contents. Figure 1 Kazuko Figure 2 Parallelization information between them. Similarly, distributed training frameworks in sub-... Figure 2 The added parallelization information can include sub Figure 2 The parallelization information within the content, and may also include sub-contents. Figure 1 Kazuko Figure 2 Parallelization information between them.

[0071] Optionally, parallelization information between different first subgraphs includes: the parallelization strategy employed between different first subgraphs. The parallelization strategy employed between different first subgraphs includes: pipeline parallelism and / or model parallelism.

[0072] For example, child Figure 1 Kazuko Figure 2 Parallelization information between sub- Figure 1 Kazuko Figure 2 The parallelization strategy used is model parallelism.

[0073] Optionally, the parallelization information between different first subgraphs may also include: parameter information of the parallelization strategy used between different first subgraphs.

[0074] For example, in some other embodiments, the sub Figure 1 Kazuko Figure 2 The parallelization strategy involves adding pipeline parallelism to model parallelism. As mentioned above, pipeline parallelism can divide the model's sample dataset into multiple smaller partitions. When pipeline parallelism and model parallelism are used in combination, the sub-parts can be further divided. Figure 1 The input data is divided into multiple smaller pieces, and the sub-data is further divided into smaller pieces. Figure 2 The input data is divided into multiple smaller pieces. In this case, the sub- Figure 1 Kazuko Figure 2Parallelization information between them can include not only sub- Figure 1 Kazuko Figure 2 The parallelization strategy used between them may also include sub- Figure 1 Kazuko Figure 2 The parameter information of the parallelization strategy used between them, for example, this parameter information may specifically be sub-parameters. Figure 1 The number of partitions into which the input data is divided, and the sub-parts. Figure 2 The number of partitions into which the input data is divided. Among them, sub... Figure 1 The number of partitions and sub-parts into which the input data is divided. Figure 2 The number of partitions into which the input data is divided can be the same or different. Additionally, sub-... Figure 1 Kazuko Figure 2 The parameter information for the parallelization strategy used can be configured by the user or by the distributed training framework.

[0075] Optionally, the parallelization information within each first subgraph includes: the parallelization strategy within each first subgraph. The parallelization strategy within each first subgraph includes: data parallelism and / or operator splitting.

[0076] For example, Figure 8 The child shown Figure 1 The internal part is split using operators, and the sub-parts are split using operators. Figure 2 Data parallelism is employed internally. In other embodiments, the sub- Figure 1 Data parallelism can be used internally, and sub-data parallelism can be used in the internal data processing. Figure 2 The internal part uses operator splitting. Additionally, the sub-parts... Figure 1 The parallelization strategy used internally can also be combined with sub- Figure 2 The parallelization strategy used within each subgraph is the same. Furthermore, the parallelization strategy used for each subgraph can also be a hybrid parallelization approach, for example, sub-... Figure 1 Internally, it employs a combination of data parallelism and operator decomposition, and / or sub-operators. Figure 2 It employs a combination of data parallelism and operator splitting.

[0077] Optionally, the parallelization information within each first subgraph may also include: parameter information of the parallelization strategy within each first subgraph.

[0078] For example, when the child Figure 1 The internal part is split using operators, and the sub-parts are split using operators. Figure 2 When using data parallelism internally, sub Figure 1 The parallelization information within can also include parameter information for operator decomposition, for example, sub-operator... Figure 1 The number of parts split. Similarly, the number of sub-parts. Figure 2 The parallelization information within the code can also include parameter information for data parallelism, such as how many devices are needed to execute the data parallelism. Additionally, sub-... Figure 1 Internal or sub Figure 2The parameters of the parallelization strategy can be configured by the user or by the distributed training framework.

[0079] like Figure 8 As shown, sub Figure 1 Kazuko Figure 2 The dashed borders of each element, and circles 1 and 2, represent children. Figure 1 Kazuko Figure 2 Parallelization information between them, i.e., sub Figure 1 Kazuko Figure 2 They were assigned to different devices for computation. Additionally, the sub-devices... Figure 1 The parallelization information within can be represented by the dashed box in the 811, indicating the sub-parallelization information. Figure 1 The internal partitioning uses operators. Additionally, the number of dashed boxes within 811 can indicate the sub-parts. Figure 1 The number of parts split. For example, the two dashed boxes inside 811 indicate that the sub-parts are split into smaller parts. Figure 1 It was split and executed on two devices. Similarly, it can also be done on the child device. Figure 2 Internally, it represents parallelized information, such as sub-information. Figure 2 The parallelization information within is used to identify data parallelism and the number of devices required for data parallelism, for example, 3.

[0080] S603. Determine the distributed computing graph based on the second computing graph and computing resources.

[0081] For example Figure 8 Physical devices 0, 1, 2, 3, and 4 are specific hardware computing resources. These devices can originate from the same machine or from different machines. In this embodiment, the physical devices can be computing devices such as GPUs or CPUs as described above; alternatively, they can be virtual machines. The distributed training framework can divide the hardware computing resources into multiple virtual devices, and this embodiment does not specifically limit the division method. For example, physical devices 0 and 1 can be divided into virtual device 1, and physical devices 2, 3, and 4 can be divided into virtual device 2. Further, a distributed computing graph 83 is obtained based on the second computing graph 82 and the physical devices included in each virtual device. Distributed computing graph 83 indicates that physical devices 0 and 1 are used for sub-processing. Figure 1 Operator splitting calculations are performed; physical devices 2, 3, and 4 are used for sub-operator splitting calculations. Figure 2 Perform parallel data computation.

[0082] S604. Train the model to be trained according to the distributed computing graph.

[0083] For example, the distributed computing graph 83 can be input into a training engine such as Tensorflow or PyTorch, which will then execute the training process. Tensorflow is an open-source machine learning platform for tasks such as image, speech, and language understanding. PyTorch is an open-source Python machine learning library based on Torch, used in fields of artificial intelligence, such as natural language processing.

[0084] In other possible application scenarios, terminal 71 can send computing resource information and the model to be trained, or the first computing graph corresponding to the model to be trained, to cloud server 70. Cloud server 70 can determine a distributed computing graph based on the computing resource information and the model to be trained, or the first computing graph corresponding to the model to be trained. The training process of the model to be trained based on this distributed computing graph can be executed on other servers or training engines provided by other servers. For example, cloud server 70 can send the distributed computing graph to terminal 71. After receiving the distributed computing graph through terminal 71, the user can train the model to be trained in the training engine provided by other servers or other servers based on the distributed computing graph. This embodiment of the disclosure obtains a first computation graph corresponding to the model to be trained and the parallelization strategy of the model to be trained. The parallelization strategy of the model to be trained includes at least one of pipeline parallelism, model parallelism, data parallelism, and operator splitting. Parallelization information is added to the first computation graph according to the parallelization strategy of the model to be trained to obtain a second computation graph. Based on the second computation graph and computing resources, a distributed computation graph is determined. The model to be trained is trained according to the distributed computation graph. This realizes the technology of computation graph editing to support multiple parallelization strategies, so that multiple parallelization strategies can be integrated into a system, thereby realizing a distributed training framework that can support multiple parallelization strategies.

[0085] Based on the above embodiments, and according to the second computation graph and computational resources, the distributed computation graph is determined to include, as follows: Figure 9 The following steps are shown:

[0086] S901. The computing resources are divided to obtain one or more virtual devices, wherein the virtual devices include one or more physical devices.

[0087] For example, physical device 0 and physical device 1 are classified as virtual device 1, and physical device 2, physical device 3 and physical device 4 are classified as virtual device 2.

[0088] Optionally, the computing resources can be partitioned to obtain one or more virtual devices, including: partitioning the computing resources according to the parallelization information to obtain one or more virtual devices.

[0089] For example, when partitioning physical device 0, physical device 1, physical device 2, physical device 3, and physical device 4, the partitioning can be specifically based on the parallelization information in the second computation graph 82. For instance, the parallelization information in the second computation graph 82 indicates that the first computation graph 81 is divided into two subgraphs. Therefore, physical device 0, physical device 1, physical device 2, physical device 3, and physical device 4 can be divided into two virtual devices, such as virtual device 1 and virtual device 2.

[0090] S902. Convert the second computation graph into a third computation graph according to the parallelization information.

[0091] like Figure 10 As shown, when the second computation graph 82 is obtained, the second computation graph 82 can also be converted into a third computation graph 84 based on the parallelization information in the second computation graph 82.

[0092] Optionally, converting the second computation graph into a third computation graph based on the parallelization information includes: converting the first subgraph into a distributed second subgraph based on the parallelization information of each of the plurality of first subgraphs; and connecting the distributed second subgraphs corresponding to each first subgraph based on the connection relationship between the plurality of first subgraphs to obtain the third computation graph.

[0093] For example, child Figure 1 Kazuko Figure 2 Each of these is denoted as a first subgraph. Based on the parallelization information of each first subgraph, each first subgraph can be transformed into a distributed second subgraph. For example, sub... Figure 11 Kazuko Figure 12 It is the son Figure 1 The resulting distributed second subgraph. Subgraphs 21, 22, and 23 are the subgraphs obtained after the transformation. Figure 2 The resulting distributed second subgraph. Further, based on the sub... Figure 1 Kazuko Figure 2 The connection between them will make the child Figure 11 Connect to subgraphs 21, 22, and 23 respectively, and connect the subgraphs... Figure 12 Connect it to subgraphs 21, 22 and 23 respectively to obtain the third computation graph 84.

[0094] S903. Map the third computation graph to the physical device to obtain a distributed computation graph.

[0095] For example, mapping the third computation graph 84 to physical devices yields the distributed computation graph 83.

[0096] Optionally, mapping the third computation graph to a physical device includes mapping each second subgraph in the third computation graph to a physical device.

[0097] For example, each second subgraph in the third computation graph 84 is mapped to a physical device.

[0098] Optionally, mapping each second subgraph in the third computation graph to a physical device includes: mapping each first subgraph to a virtual device; and mapping each second subgraph corresponding to the first subgraph to a physical device included in the virtual device corresponding to the first subgraph.

[0099] For example, the child Figure 1 Mapped to virtual device 1, further, the child Figure 1 Corresponding sub Figure 11 Kazuko Figure 12 Map them respectively to the physical devices included in virtual device 1, for example, map the sub-devices... Figure 11 Mapped to physical device 0 in virtual device 1, the child Figure 12 Mapped to physical device 1 in virtual device 1. Similarly, map the child... Figure 2 Mapped to virtual device 2, further, the child Figure 2 The corresponding subgraphs 21, 22 and 23 are mapped to the physical devices included in the virtual device 2. For example, subgraph 21 is mapped to physical device 2, subgraph 22 is mapped to physical device 3 and subgraph 23 is mapped to physical device 4.

[0100] Additionally, in other embodiments, the sub Figure 1 It can also be split into 3 devices, such as Figure 11 As shown. Optionally, physical device 0 and physical device 3 can be the same device or different devices. Similarly, physical device 1 and physical device 4 can be the same device or different devices. Physical device 2 and physical device 5 can be the same device or different devices.

[0101] This embodiment divides the computing resources to obtain one or more virtual devices, each including one or more physical devices. Based on the parallelization information, the second computing graph is converted into a third computing graph, and the third computing graph is mapped to the physical devices to obtain a distributed computing graph. This allows the computing resources to be fully utilized, improving the utilization rate of computing resources.

[0102] like Figure 12 The diagram 120 illustrates the structure of the distributed training framework. The input to the distributed training framework 120 can be the first computation graph as described above. The output of the distributed training framework 120 can be the training result.

[0103] like Figure 12 As shown, the distributed training framework 120 includes an interface layer, which includes a user interface. This user interface includes scopes and a cluster. Users can configure parallelization strategies for the model to be trained through scopes and the cluster.

[0104] The `scopes` interface identifies the parallelization strategies for different parts of the model to be trained. For example, a `scopes` can be at least one of the following: `replica` (data parallelism), `split` (operator splitting), `pipeline` (pipeline parallelism), and `stage` (model parallelism). In other words, a `scopes` can be any one of these four parallelization strategies, or a combination of two or more. Different `scopes` specify different parallelization strategies. Furthermore, the `scopes` interface supports nested use, allowing different parallelization strategies to be nested to achieve various hybrid parallelism strategies to accelerate distributed training. Users can use the `scopes` interface to divide the model to be trained into multiple subgraphs and configure a `scopes` for each subgraph.

[0105] like Figure 12 The cluster shown is used to partition computing resources, which can also be referred to as hardware resources. These computing resources can specifically be GPUs or CPUs, etc. For example, a cluster is used to divide computing resources into multiple virtual computing devices. Furthermore, based on the parallelization strategy of the model to be trained, the subgraph partitioned by the user through scopes is mapped to the virtual computing devices; this mapping process is completely transparent to the user.

[0106] The following examples illustrate how to build various parallelization strategies through the user interface.

[0107] 1) The data parallelism construction method is as follows:

[0108] withwhale.cluster():

[0109] withwhale.replica():

[0110] USER_MODEL_DEFINATION()

[0111] Here, USER_MODEL_DEFINATION() represents the user's original code, i.e., the code corresponding to the model to be trained. with whale.replica(): represents the data parallelism strategy configured by the user for the model to be trained. withwhale.cluster(): represents calling the cluster interface. In other words, the user does not need to modify the original code; they only need to add replica scope and cluster to the outer layer of the original code to enable the distributed training framework to perform data parallel distributed training on the model to be trained.

[0112] 2) The construction method of the hybrid parallel strategy of data parallelism nested with pipeline parallelism and model parallelism is as follows:

[0113] withwhale.cluster():

[0114] withwhale.replica():

[0115] withwhale.pipeline(num_micro_batch=4):

[0116] withwhale.stage():

[0117] USER_MODEL_DEFINATION_PART_1()

[0118] withwhale.stage():

[0119] USER_MODEL_DEFINATION_PART_2()

[0120] Here, `USER_MODEL_DEFINATION_PART_1()` represents the first part of the model to be trained, and `USER_MODEL_DEFINATION_PART_2()` represents the second part. The first and second parts can be user-defined. The two `with whale.stage()` methods represent the model parallelism strategies configured by the user for the first and second parts respectively. `with whale.pipeline(num_micro_batch=4)` represents the pipeline parallelism strategy configured by the user for the first and second parts respectively. `with whale.replica()` represents the data parallelism strategy configured by the user for the first and second parts respectively. In other words, the user does not need to modify the original code; they only need to add a stage scope to the original code to partition the model to be trained, for example, dividing it into a first part and a second part. The first part can correspond to one subgraph, and the second part can correspond to another subgraph. Adding a pipeline scope outside the first and second parts enables the distributed training framework to perform pipeline parallel training of the model to be trained. Furthermore, if data parallel training of the model to be trained is required, a replica scope can be added outside the pipeline scope.

[0121] 3) The construction method of the hybrid parallel strategy of operator decomposition and data parallelism is as follows:

[0122] withwhale.cluster():

[0123] withwhale.replica():

[0124] USER_MODEL_DEFINATION_PART_1()

[0125] with whale.split(split_dim="length"):

[0126] USER_MODEL_DEFINATION_PART_2()

[0127] Here, `with whale.replica()` represents the data parallelism strategy configured by the user for the first part of the model to be trained. `with whale.split(split_dim="length")` represents the operator splitting strategy configured by the user for the second part of the model to be trained. In other words, for operator splitting, the user can add `splitscope` to the model part that needs to be split. For data parallelism, the user can add `replicascope` to the model part that requires data parallelism.

[0128] It is understood that the methods for constructing the parallelization strategies described above are merely illustrative and not specifically limited. For example, other parallelization strategies can be constructed in other embodiments. That is, replica (data parallelism), split (operator splitting), pipeline (pipeline parallelism), and stage (model parallelism) can be used individually or in combination. In scenarios where they are used in combination, the specific combination method is not limited. In addition, replica (data parallelism), split (operator splitting), pipeline (pipeline parallelism), and stage (model parallelism) can also be nested, and the specific nesting method or nesting order is not limited.

[0129] In some other embodiments, when the distributed training framework 120 receives the first computation graph, the distributed training framework 120 can automatically add a parallelization strategy to the first computation graph through the scopes in the interface layer.

[0130] This embodiment allows users to construct a wide variety of parallelization strategies by using replica (data parallelism), split (operator splitting), pipeline (pipeline parallelism), and stage (model parallelism) individually, in combination, or nested, thereby improving the flexibility of parallelization strategies. Furthermore, as the above example shows, the user's original code, i.e., the code for the user's model definition, can be based on native interfaces, such as Tensorflow and PyTorch interfaces, without needing to change the programming interface for the model definition. Therefore, users do not need to modify the original code; they only need to add a few lines of Application Programming Interface (API) calls to easily combine the parallelization strategies they want.

[0131] like Figure 12As shown, the distributed training framework 120 also includes a model and a parallelized intermediate representation layer. The parallelized representation layer within this layer can parse the parallelization strategy of the model to be trained to obtain corresponding abstractions. For example, this embodiment provides three types of abstractions: Multi-Dimensional Resource, SubgraphGroup, and Virtual Device. These three types of abstractions can be used to unify and express different parallelization strategies. After unifying and abstracting the parallelization strategy, the parallelization strategy can be implemented based on computational graph editing technology.

[0132] For example, data parallelism and operator partitioning can be expressed using a Multi-Dimensional Resource. Typically, model parameters have multiple dimensions, such as data sample dimension, channel dimension, height dimension, width dimension, and length dimension, where the data sample dimension is denoted as N, the channel dimension as C, the height dimension as H, the width dimension as W, and the length dimension as L. Data parallelism specifically involves partitioning the data sample dimension N. Operator partitioning specifically involves partitioning other dimensions besides the data sample dimension N. For example, operator partitioning can partition one of the channel dimension C, height dimension H, width dimension W, and length dimension L, or it can partition multiple dimensions of the channel dimension C, height dimension H, width dimension W, and length dimension L. This abstraction of Multi-Dimensional Resource supports arbitrary partitioning or splitting across different dimensions. Figure 13 As shown, Batch Sample represents the data sample dimension, Channel represents the channel dimension, and Length represents the length dimension. For example, when splitting the data sample dimension, Multi-DimensionalResource can represent data parallelism. When splitting one or more dimensions of the channel dimension C, height dimension H, width dimension W, and length dimension L, Multi-Dimensional Resource can represent operator splitting. When splitting the data sample dimension and other dimensions simultaneously, Multi-DimensionalResource can represent a combination of data parallelism and operator splitting.

[0133] The Subgraph Group abstraction supports dividing a complete computational graph of a model, such as the first computational graph described in the above embodiment, into multiple subgraphs. Each subgraph can implement the same or different parallelization strategies. Subgraphs can communicate with each other. For example, Subgraph Groups can be used to represent model parallelism and / or pipelined parallelism. Specifically, model parallelism and / or pipelined parallelism can be parallelization strategies between subgraphs, while data parallelism and / or operator splitting can be parallelization strategies within a subgraph. Figure 14 As shown, the first computational graph 81 can be divided in various ways. For example, the first computational graph 81 can be divided into the second computational graph 82 described in the above embodiment. Alternatively, the first computational graph 81 can also be divided into... Figure 14 As shown in 140, the first computation graph 81 can be divided into 4 subgraphs, each of which includes one layer of the model to be trained.

[0134] The Virtual Device abstraction supports abstracting multiple physical devices into a single virtual device. These multiple physical devices can originate from the same machine (single-machine multi-GPU) or from multiple different machines (multi-machine multi-GPU). In some embodiments, the physical device is specifically a GPU, and the multiple physical devices are, for example,... Figure 15 The GPUs 0-GPU 5 shown can be partitioned in various ways. For example, when GPUs 0-GPU 2 come from machine A and GPUs 3-GPU 5 come from machine B, GPUs 0-GPU 3 can be partitioned into virtual device 0, and GPUs 4 and 5 into virtual device 1. It is understandable that... Figure 15 The partitioning method shown is merely illustrative and is not specifically limited in this embodiment. Specifically, the user only needs to perceive the virtual devices and allocate corresponding virtual devices to the subgraph. The distributed training framework 120 can associate virtual devices with specific physical devices based on the network topology of the hardware computing resources.

[0135] In addition, such as Figure 12 The execution layer in the distributed training framework 120 shown can be used to rewrite the second computation graph, construct a third computation graph that can be parallelized, and then convert the third computation graph into a distributed computation graph. Furthermore, the execution layer can send the distributed computation graph to a training engine, such as Tensorflow or PyTorch.

[0136] This embodiment uses three abstractions—Multi-Dimensional Resource, Subgraph Group, and Virtual Device—to unify and express various parallelization strategies, enabling the distributed training framework to support arbitrary parallelization strategies and various hybrid parallelization strategies, thus solving the problem of a single parallelization strategy. Furthermore, this embodiment also utilizes computational graph editing technology to implement various parallelization strategies, allowing multiple parallelization strategies to be integrated into a single system, improving the flexibility and diversity of parallelization strategies.

[0137] Based on the above embodiments, the model to be trained is trained according to the distributed computing graph, including as follows: Figure 16 The following steps are shown:

[0138] S1601. Convert the distributed computing graph into a distributed computing graph that the training engine can recognize.

[0139] For example, in Figure 8 , Figure 10 , Figure 11 Based on this, the distributed computation graph 83 can be converted into a distributed computation graph recognizable by training engines such as TensorFlow or PyTorch. Specifically, the process of converting the distributed computation graph 83 into a distributed computation graph recognizable by TensorFlow or PyTorch can be as follows: Figure 12 The parallelized computation graph transformation component shown is used to perform the operation.

[0140] S1602. Input the distributed computing graph that the training engine can recognize into the training engine, and train the model to be trained through the training engine.

[0141] like Figure 12 As shown, the distributed training framework 120 also includes a training engine. After the parallelized computation graph conversion component converts the distributed computation graph 83 into a distributed computation graph that can be recognized by Tensorflow or PyTorch, the distributed computation graph that can be recognized by Tensorflow or PyTorch can be input into the training engine, and the training engine can train the model to be trained.

[0142] This embodiment converts the distributed computation graph into a distributed computation graph that the training engine can recognize, achieving cross-platform compatibility with different training engines such as TensorFlow or PyTorch, thereby improving the compatibility of distributed training frameworks. Furthermore, by converting the distributed computation graph into a training engine-recognizable one, the coupling between the training engine and parallelization strategies can be reduced, thus ensuring compatibility with existing training engines and improving the compatibility of user models.

[0143] Figure 17 This is a schematic diagram of the structure of a model processing apparatus provided in an embodiment of this disclosure. The model processing apparatus provided in an embodiment of this disclosure can execute the processing flow provided in the model processing method embodiment, such as... Figure 17 As shown, the model processing device 170 includes:

[0144] The acquisition module 171 is used to acquire the first computation graph corresponding to the model to be trained and the parallelization strategy of the model to be trained. The parallelization strategy of the model to be trained includes at least one of pipeline parallelism, model parallelism, data parallelism and operator splitting.

[0145] Add module 172, which is used to add parallelization information to the first computation graph according to the parallelization strategy of the model to be trained, to obtain the second computation graph;

[0146] The determination module 173 is used to determine the distributed computing graph based on the second computing graph and computing resources;

[0147] Training module 174 is used to train the model to be trained based on the distributed computing graph.

[0148] Optionally, the addition module 172 is specifically used to: divide the first computation graph into multiple first subgraphs according to the parallelization strategy of the model to be trained; and add parallelization information to each of the multiple first subgraphs according to the parallelization strategy of the model to be trained to obtain a second computation graph.

[0149] Optionally, the parallelization information includes parallelization information between different first subgraphs and parallelization information within each first subgraph.

[0150] Optionally, parallelization information between different first subgraphs includes: the parallelization strategy adopted between different first subgraphs.

[0151] Optionally, the parallelization information between different first subgraphs may also include: parameter information of the parallelization strategy used between different first subgraphs.

[0152] Optionally, parallelization strategies employed between different first subgraphs include pipeline parallelism and / or model parallelism.

[0153] Optionally, the parallelization information within each first subgraph includes: the parallelization strategy within each first subgraph.

[0154] Optionally, the parallelization information within each first subgraph may also include: parameter information of the parallelization strategy within each first subgraph.

[0155] Optionally, parallelization strategies within each first subgraph include: data parallelism and / or operator splitting.

[0156] Optionally, the determining module 173 includes:

[0157] The partitioning unit 1731 is used to partition the computing resources to obtain one or more virtual devices, wherein the virtual devices include one or more physical devices;

[0158] Conversion unit 1732 is used to convert the second computation graph into a third computation graph according to the parallelization information;

[0159] The mapping unit 1733 is used to map the third computation graph to a physical device to obtain a distributed computation graph.

[0160] Optionally, the partitioning unit 1731 is specifically used to: partition the computing resources according to the parallelization information to obtain one or more virtual devices.

[0161] Optionally, the conversion unit 1732 is specifically used to: convert the first subgraph into a distributed second subgraph according to the parallelization information of each first subgraph in the plurality of first subgraphs; and connect the distributed second subgraphs corresponding to each first subgraph according to the connection relationship between the plurality of first subgraphs to obtain a third computation graph.

[0162] Optionally, the mapping unit 1733 is specifically used to map each of the second subgraphs in the third computation graph to a physical device.

[0163] Optionally, the mapping unit 1733 is specifically used to: map each first subgraph to a virtual device; and map each second subgraph corresponding to the first subgraph to a physical device included in the virtual device corresponding to the first subgraph.

[0164] Optionally, the acquisition module 171 is specifically used to: determine the parallelization strategy of the model to be trained based on the first computation graph corresponding to the model to be trained.

[0165] Optionally, the acquisition module 171 is specifically used to: acquire the parallelization strategy of the model to be trained selected by the user.

[0166] Optionally, the training module 174 is specifically used to: convert the distributed computation graph into a distributed computation graph that can be recognized by the training engine; input the distributed computation graph that can be recognized by the training engine into the training engine, and train the model to be trained through the training engine.

[0167] Figure 17 The model processing apparatus of the illustrated embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0168] Figure 18 This is a schematic diagram of the structure of a model processing device provided in an embodiment of this disclosure. The model processing device provided in this embodiment of the disclosure can execute the processing flow provided in the model processing method embodiment, such as... Figure 18 As shown, the model processing device 180 includes: a memory 181, a processor 182, a computer program, and a communication interface 183; wherein the computer program is stored in the memory 181 and configured to be executed by the processor 182 as described above in the model processing method.

[0169] In addition, this disclosure also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the model processing method described in the above embodiments.

[0170] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0171] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model processing method, characterized in that, The method is executed by a distributed training framework, and the method includes: Obtain the first computation graph corresponding to the model to be trained, and the parallelization strategy of the model to be trained, wherein the parallelization strategy of the model to be trained includes at least one of pipeline parallelism, model parallelism, data parallelism and operator splitting. Parallelization information is added to the first computation graph according to the parallelization strategy of the model to be trained, to obtain the second computation graph; Based on the second computation graph and the computational resources, a distributed computation graph is determined, wherein the computational resources include at least one of the following: GPU and CPU; The model to be trained is trained according to the distributed computing graph; The process of obtaining the parallelization strategy of the model to be trained includes at least one of the following: analyzing the first computation graph to determine the feature information of the model to be trained, determining the parallelization strategy based on the feature information; and obtaining the parallelization strategy selected by the user. Specifically, determining the distributed computing graph based on the second computing graph and computing resources includes: dividing the computing resources to obtain one or more virtual devices, each virtual device including one or more physical devices; and obtaining the distributed computing graph based on the parallelization information, the second computing graph, and the physical devices. The distributed training framework includes a model and a parallelized intermediate representation layer. The parallelization layer in the model and parallelized intermediate representation layer is used to parse the parallelization strategy of the model to be trained to obtain the corresponding abstraction, and to implement the parallelization strategy of the model to be trained based on computational graph editing technology.

2. The method according to claim 1, characterized in that, Parallelization information is added to the first computation graph according to the parallelization strategy of the model to be trained, resulting in a second computation graph, including: The first computation graph is divided into multiple first subgraphs according to the parallelization strategy of the model to be trained; Parallelization information is added to each of the plurality of first subgraphs according to the parallelization strategy of the model to be trained, to obtain a second computation graph.

3. The method according to claim 2, characterized in that, The parallelization information includes parallelization information between different first subgraphs, as well as parallelization information within each first subgraph.

4. The method according to claim 3, characterized in that, Parallelization information between different first subgraphs includes: the parallelization strategy adopted between different first subgraphs.

5. The method according to claim 4, characterized in that, Parallelization information between different first subgraphs also includes parameter information of the parallelization strategies used between different first subgraphs.

6. The method according to claim 4 or 5, characterized in that, Parallelization strategies employed between different first subgraphs include pipeline parallelism and / or model parallelism.

7. The method according to claim 3, characterized in that, Parallelization information within each first subgraph includes: the parallelization strategy within each first subgraph.

8. The method according to claim 7, characterized in that, The parallelization information within each first subgraph also includes parameter information for the parallelization strategy within each first subgraph.

9. The method according to claim 7 or 8, characterized in that, Parallelization strategies within each first subgraph include: data parallelism and / or operator splitting.

10. The method according to claim 1, characterized in that, Based on the parallelization information, the second computation graph, and the physical device, the distributed computation graph is obtained, including: The second computation graph is converted into a third computation graph based on the parallelization information; The third computation graph is mapped to physical devices to obtain a distributed computation graph.

11. The method according to claim 10, characterized in that, The computing resources are partitioned to obtain one or more virtual devices, including: The computing resources are divided according to the parallelization information to obtain one or more virtual devices.

12. The method according to claim 10 or 11, characterized in that, The second computation graph is converted into a third computation graph based on the parallelization information, including: Based on the parallelization information of each of the multiple first subgraphs, the first subgraph is transformed into a distributed second subgraph; Based on the connection relationships between the multiple first subgraphs, the distributed second subgraphs corresponding to each first subgraph are connected to obtain a third computation graph.

13. The method according to claim 12, characterized in that, Mapping the third computation graph to a physical device includes: Each second subgraph in the third computation graph is mapped to a physical device.

14. The method according to claim 13, characterized in that, Mapping each second subgraph in the third computation graph to a physical device includes: Map each first subgraph to a virtual device; Map each second subgraph corresponding to the first subgraph to the physical device included in the virtual device corresponding to the first subgraph.

15. The method according to claim 1, characterized in that, Training the model to be trained based on the distributed computing graph includes: The distributed computation graph is converted into a distributed computation graph that the training engine can recognize; The distributed computation graph that the training engine can recognize is input into the training engine, and the training engine trains the model to be trained.

16. A model processing apparatus, characterized in that, The device employs a distributed training framework for execution, and the device includes: The acquisition module is used to acquire the first computation graph corresponding to the model to be trained and the parallelization strategy of the model to be trained. The parallelization strategy of the model to be trained includes at least one of pipeline parallelism, model parallelism, data parallelism and operator splitting. An addition module is used to add parallelization information to the first computation graph according to the parallelization strategy of the model to be trained, so as to obtain a second computation graph. The determining module is configured to determine a distributed computing graph based on the second computing graph and computing resources, wherein the computing resources include at least one of the following: GPU and CPU; The training module is used to train the model to be trained based on the distributed computing graph; The acquisition module is further configured to perform at least one of the following steps: analyze the first computation graph to determine the feature information of the model to be trained, determine the parallelization strategy based on the feature information, and acquire the parallelization strategy selected by the user. The determining module is further configured to partition the computing resources to obtain one or more virtual devices, each virtual device including one or more physical devices; and to obtain the distributed computing graph based on the parallelization information, the second computing graph, and the physical devices. The distributed training framework includes a model and a parallelized intermediate representation layer. The parallelization layer in the model and parallelized intermediate representation layer is used to parse the parallelization strategy of the model to be trained to obtain the corresponding abstraction, and to implement the parallelization strategy of the model to be trained based on computational graph editing technology.

17. A model processing device, characterized in that, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1-15.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-15.

Images