Model training method, model training device and electronic device

Abstract

The embodiment of the invention provides a model training method, a model training device and an electronic device. The method is applied to electronic equipment loaded with a parallel computing framework and heterogeneous computing equipment. The method comprises the following steps: segmenting a model training task into a plurality of computing tasks by using the parallel computing framework; using the parallel computing framework to distribute the plurality of computing tasks to a plurality of virtual devices, the plurality of virtual devices being virtual devices created based on the heterogeneous computing device; and executing the distributed calculation tasks by the plurality of virtual devices so as to carry out iterative calculation of model training. According to the method provided by the embodiment of the invention, the parallel execution logic of the computing task is placed in the virtual device of the reusable computing resource, and the binding relationship among the computing task, the communication task and the hardware is decoupled, so that the effects of multi-process parallelism, communication and computing parallelism can be achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method, a model training device and an electronic device. Background Art

[0002] Deep learning is an artificial intelligence (AI) method used to teach computers to process data in ways inspired by the human brain. Deep learning models can recognize complex patterns in pictures, text, sound, and other data to generate accurate insights and predictions. Users can use deep learning methods to automatically perform tasks that usually require artificial intelligence, such as describing an image or transcribing a sound file into text.

[0003] In deep learning application scenarios, the computing power required for model training is huge. However, based on existing communication solutions, the communication overhead during model training is very high, there is a serious waste of computing power resources, and the resource utilization rate of deep learning model training is not high.

[0004] Therefore, a model training method for deep learning model training is needed to reduce the communication overhead in the deep learning model training process, reduce idle computing resources, and improve the resource utilization of deep learning model training. Summary of the invention

[0005] In order to solve the problem of how to reduce communication overhead in the process of deep learning model training, reduce idle computing resources, and improve resource utilization of deep learning model training, the present application provides a model training method, a model training device and an electronic device. The present application also provides a computer-readable storage medium and a server cluster.

[0006] The present application embodiment adopts the following technical solutions:

[0007] In a first aspect, the present application provides a model training method, which is applied to an electronic device and a heterogeneous computing device loaded with a parallel computing framework, and the method includes:

[0008] The parallel computing framework divides the model training task into multiple computing tasks;

[0009] The parallel computing framework distributes the multiple computing tasks to multiple virtual devices, wherein the multiple virtual devices correspond to the multiple computing tasks one by one, and the multiple virtual devices are virtual devices created based on the heterogeneous computing devices;

[0010] The multiple virtual devices execute the distributed computing tasks to perform iterative calculations for model training.

[0011] According to the method of the first aspect, the parallel execution logic of computing tasks is placed in a virtual device of reusable computing resources, and the binding relationship between computing tasks, communication tasks and hardware (heterogeneous computing devices) is decoupled, achieving the effect of multi-process parallelism, communication and computing parallelism on a macro level. In theory, the time required to complete communication is completely hidden.

[0012] Furthermore, since the parallel execution logic of computing tasks is placed in a virtual device that can reuse computing resources, during the iterative calculation of model training, computing tasks and communication tasks do not need to rely on the hardware topology of heterogeneous computing devices, but rely on the topology of virtual devices in the computing task execution device set. Therefore, when maximizing the computing-to-communication ratio, there is no need to understand the hardware topology of heterogeneous computing devices in detail, which reduces the difficulty of achieving the maximization of the computing-to-communication ratio.

[0013] Further, according to the method of the first aspect, a parallel computing framework for distributing tasks to multiple heterogeneous computing devices is used to implement task distribution to multiple virtual devices. Since the virtual devices are presented as heterogeneous computing devices at the parallel computing framework, there is no need to adjust the parallel computing framework for distributing tasks to multiple heterogeneous computing devices, thereby reducing the difficulty of implementing task distribution to virtual devices.

[0014] In an implementation of the first aspect, the method further includes:

[0015] A communication virtualization device is deployed on the electronic device, and the communication virtualization device is used to present the multiple virtual devices to the parallel computing framework in the form of the multiple heterogeneous computing devices, wherein the communication virtualization device is deployed outside the parallel computing framework.

[0016] According to the above implementation method, a communication virtualization device is deployed outside the parallel computing framework, and the communication between the parallel computing framework and the virtual device is realized through the communication virtualization device. The parallel computing framework does not need to perceive the deployment and implementation status of the virtual device on the heterogeneous computing device, thereby reducing the difficulty of implementing communication between the parallel computing framework and the virtual device.

[0017] In an implementation of the first aspect:

[0018] Implementing communication between the parallel computing framework and the plurality of virtual devices, and between the plurality of virtual devices based on collective communication;

[0019] The communication virtualization device is also used to: configure collective communication attribute resources for the virtual device, present the collective communication attributes of the virtual device to the parallel computing framework; and provide virtualization support for the collective communication physical resources.

[0020] In an implementation of the first aspect, the method further includes:

[0021] A routing device is deployed outside the parallel computing framework, and the routing device is used to generate a collective communication operator for collective communication according to the virtual device topology relationship of the multiple virtual devices.

[0022] According to the above implementation method, the parallel computing framework is not responsible for managing device topology and inter-device communication, and the collective communication capability is transferred to the virtual device. During the iterative computing process, the implementation of collective communication does not require the participation of the parallel computing framework, which improves the flexibility and efficiency of collective communication.

[0023] In an implementation of the first aspect, the routing device is used to save a virtual device routing table, and the virtual device routing table is used to record all virtual devices and topological connection relationships that require data copying for the current computing task.

[0024] In an implementation manner of the first aspect, the routing device is deployed on the electronic device, and the routing device includes instance objects corresponding one-to-one to the multiple virtual devices.

[0025] In an implementation manner of the first aspect, the routing device is deployed in each of the virtual devices.

[0026] In an implementation of the first aspect, the multiple computing tasks include at least a first computing task and a second computing task; the parallel computing framework distributes the multiple computing tasks to multiple virtual devices, including distributing the first computing task to a first virtual device and distributing the second computing task to a second virtual device;

[0027] The virtual device executes the distributed computing tasks to perform iterative computing of model training, including:

[0028] The first virtual device performs the first computing task, and the second virtual device performs the second computing task;

[0029] In the case of triggering task migration from the first virtual device to the third virtual device, terminating the first virtual device from executing the first computing task, and maintaining the second virtual device from continuing to execute the second computing task;

[0030] Executing the task migration includes synchronizing the first computing task and the current execution state of the first computing task in the first virtual device to the third virtual device;

[0031] The third virtual device continues to execute the first computing task.

[0032] In an implementation of the first aspect, the virtual device executes the distributed computing tasks to perform iterative computing of model training, further comprising:

[0033] After the task migration, the virtual device topology relationship in the routing device is updated according to the task migration.

[0034] In an implementation of the first aspect, the virtual device executes the distributed computing tasks to perform iterative computing of model training, further comprising:

[0035] After the task migration, no information indicating a change in the topological relationship of the virtual devices is fed back to the parallel computing framework.

[0036] According to the above implementation method, the parallel computing framework is not responsible for managing device topology and inter-device communication. It sinks the collective communication capabilities to virtual devices, supports dynamic migration of computing tasks during iterative computing, and dynamically updates virtual device link information. There is no need for the computing framework layer to perceive and recompile to generate communication operators, which reduces the waste of computing power waiting.

[0037] In an implementation of the first aspect, the multiple computing tasks further include at least a third computing task, and execution of the third computing task requires reference to a computing result of the first computing task;

[0038] The parallel computing framework distributes the plurality of computing tasks to a plurality of virtual devices, further comprising distributing the third computing task to a fourth virtual device;

[0039] The multiple virtual devices execute the distributed computing tasks to perform iterative calculations for model training, including: the fourth virtual device subscribes to the task execution status of the first virtual device according to the third computing task; after the first virtual device completes the execution of the first computing task, the fourth virtual device obtains the calculation result of the first computing task, and executes the third computing task according to the calculation result of the first computing task.

[0040] According to the above implementation, the virtual device senses the real-time status of each computing task, and a communication task can be initiated in real time after a single computing task is completed. Compared with the parallel computing framework that uniformly manages the status of all computing tasks, after a single computing task is completed, it is necessary to wait for the computing framework to send a communication task to implement communication. The method according to the embodiment of the present application can reduce computing power consumption.

[0041] In an implementation of the first aspect, the virtual device executes the distributed computing tasks to perform iterative computing of model training, further comprising:

[0042] After the task migration, the fourth virtual device is updated to subscribe to the task execution state of the third virtual device.

[0043] In an implementation of the first aspect:

[0044] Executing the task migration further includes synchronizing the virtual device topology relationship in the routing device of the first virtual device to the routing device of the third virtual device;

[0045] After the task migration, updating the virtual device topology relationship in the routing device according to the task migration includes:

[0046] The virtual device topology relationship in the routing device of the third virtual device is updated, and the virtual device topology relationship in the routing device of the fourth virtual device is updated.

[0047] In a second aspect, the present application provides a model training device, the device comprising a parallel computing framework and a plurality of virtual devices, wherein:

[0048] The multiple virtual devices are virtual devices created based on heterogeneous computing devices;

[0049] The parallel computing framework is loaded on the electronic device, and is used to divide the model training task into multiple computing tasks, and distribute the multiple computing tasks to the multiple virtual devices, wherein the multiple computing tasks correspond to the multiple virtual devices one by one;

[0050] The multiple virtual devices are used to execute the distributed computing tasks to perform iterative calculations for model training.

[0051] In an implementation of the second aspect, the device further includes a communication virtualization device, wherein:

[0052] The communication virtualization device is deployed on the electronic device, wherein the communication virtualization device is deployed outside the parallel computing framework;

[0053] The communication virtualization device is used to present the multiple virtual devices to the parallel computing framework in the form of multiple heterogeneous computing devices.

[0054] In an implementation of the second aspect, the communication between the parallel computing framework and the virtual device, and between different virtual devices, is collective communication;

[0055] The communication virtualization device comprises:

[0056] A collective communication resource virtualization module, which is used to configure collective communication attribute resources for the virtual device and present the collective communication attributes of the virtual device to the parallel computing framework;

[0057] The virtual device super-resolution module is used to support virtualization of collective communication physical resources.

[0058] In an implementation manner of the second aspect, the device further includes a routing device, wherein:

[0059] The routing device is deployed outside the parallel computing framework, and is used to generate a collective communication operator for collective communication according to the virtual device topology relationship of the multiple virtual devices.

[0060] In an implementation of the second aspect, the routing device is used to save a virtual device routing table, and the virtual device routing table is used to record all virtual devices and topological connection relationships that require data copying for the current computing task.

[0061] In an implementation manner of the second aspect, the routing device is deployed on the electronic device, and the routing device includes instance objects corresponding one-to-one to the multiple virtual devices.

[0062] In an implementation manner of the second aspect, the routing device is deployed in each of the virtual devices.

[0063] In an implementation of the second aspect: the multiple computing tasks include at least a first computing task and a second computing task; the multiple virtual devices include at least a first virtual device and a second virtual device; the parallel computing framework distributes the first computing task to the first virtual device, and distributes the second computing task to the second virtual device; the first virtual device executes the first computing task, and the second virtual device executes the second computing task;

[0064] The apparatus further comprises a task migration module and a third virtual device, wherein:

[0065] The task migration module is used to: in the case of triggering task migration from the first virtual device to the third virtual device, terminate the first virtual device from executing the first computing task, and maintain the second virtual device to continue executing the second computing task; perform the task migration, including synchronizing the first computing task in the first virtual device and the current execution state of the first computing task to the third virtual device;

[0066] The third virtual device is used to continue executing the first computing task after the task migration.

[0067] In an implementation manner of the second aspect, the routing device is further configured to, after the task migration, update a virtual device topology relationship in the routing device according to the task migration.

[0068] In an implementation of the second aspect, the routing device is further configured to, after the task migration, not feed back information indicating a change in a topological relationship of the virtual devices to the parallel computing framework.

[0069] In an implementation of the second aspect, the multiple computing tasks further include at least a third computing task, and execution of the third computing task requires reference to a computing result of the first computing task; the multiple virtual devices further include at least a fourth virtual device;

[0070] The parallel computing framework is used to distribute the third computing task to the fourth virtual device;

[0071] The fourth virtual device is used to: subscribe to the task execution status of the first virtual device according to the third computing task; after the first virtual device completes the execution of the first computing task, the fourth virtual device obtains the calculation result of the first computing task; and execute the third computing task according to the calculation result of the first computing task.

[0072] In an implementation manner of the second aspect, the fourth virtual device is further used to:

[0073] After the task migration, the task execution state of subscribing to the first virtual device is updated to the task execution state of subscribing to the third virtual device.

[0074] In an implementation of the second aspect:

[0075] The task migration module is used to synchronize the virtual device topology relationship in the routing device of the first virtual device to the routing device of the third virtual device;

[0076] The routing device of the third virtual device is used to update the virtual device topology relationship in the routing device of the third virtual device according to the task migration after the task migration;

[0077] The routing device of the fourth virtual device is used for updating the virtual device topology relationship in the routing device of the fourth virtual device according to the task migration after the task migration.

[0078] In a third aspect, the present application provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method steps described in the first aspect.

[0079] In an implementation of the third aspect, the electronic device is the electronic device described in the first aspect, and the electronic device is used to load a parallel computing framework, and the parallel computing framework loaded by the electronic device is used to execute the method steps executed by the parallel computing framework in the method described in the embodiment of the present application.

[0080] Specifically, when the computer program instructions in the memory of the electronic device are executed by the processor of the electronic device, the electronic device is triggered to execute the method steps executed by the electronic device in the first aspect.

[0081] In an implementation of the third aspect, the electronic device is the heterogeneous computing device described in the first aspect. The electronic device is used to create a virtual device, and the virtual device created in the electronic device is used to execute the method steps executed by the virtual device in the first aspect.

[0082] Specifically, when the computer program instructions in the memory of the electronic device are executed by the processor of the electronic device, the electronic device is triggered to execute the method steps executed by the heterogeneous computing device in the first aspect.

[0083] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method described in the first aspect.

[0084] In a fifth aspect, the present application provides a server cluster, wherein the electronic device comprises a memory for storing computer program instructions and a processor for executing computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method steps described in the first aspect.

[0085] In an implementation manner of the fifth aspect, the server cluster includes the electronic device described in the first aspect and a heterogeneous computing device. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 Shown is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application;

[0087] Figure 2 Shown is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application;

[0088] Figure 3 Shown is a schematic diagram of a model training process according to an embodiment of the present application;

[0089] Figure 4 Shown is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application;

[0090] Figure 5 Shown is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application;

[0091] Figure 6 Shown is a schematic diagram of a model training process according to an embodiment of the present application;

[0092] Figure 7 The figure is a schematic diagram of a partial process of model training according to an embodiment of the present application;

[0093] Figure 8 FIG. 1 is a schematic diagram of a virtual device structure according to an embodiment of the present application;

[0094] Fig. 9 The figure is a schematic diagram of a partial process of model training according to an embodiment of the present application;

[0095] Fig.10 Shown is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application;

[0096] Fig.11 Shown is a schematic diagram of a virtual device according to an embodiment of the present application;

[0097] Fig.12 The figure is a schematic diagram of a computing task migration process according to an embodiment of the present application;

[0098] Fig.13 The figure is a schematic diagram of a computing task migration process according to an embodiment of the present application;

[0099] Fig.14 Shown is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application;

[0100] Fig.15 Shown is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application;

[0101] Fig.16 FIG. 4 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0102] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0103] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0104] For deep learning model training, a feasible training solution is to split the data and model according to the parallel strategy, and then distribute the computing tasks to a heterogeneous cluster with multiple machines and multiple cards for iterative training.

[0105] Figure 1 Shown is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application.

[0106] The hardware device for implementing model training includes an electronic device 100 and a plurality of heterogeneous computing devices, and the plurality of heterogeneous computing devices include at least a heterogeneous computing device 110 and a heterogeneous computing device 120. In another embodiment, the plurality of heterogeneous computing devices may include more than two heterogeneous computing devices.

[0107] In the embodiment of the present application, the electronic device is a device for managing the model training process. The embodiment of the present application does not specifically limit the type of electronic device. For example, the electronic device can be a local device (e.g., a personal desktop computer) or a server device (e.g., a local server or a cloud server).

[0108] A heterogeneous computing device is a physical device used to perform computing tasks, which is used to receive computing tasks, calculate computing tasks, and output computing results. The embodiments of the present application do not specifically limit the types of heterogeneous computing devices. Exemplarily, the heterogeneous computing device can be a computing card, such as a graphics processing unit (GPU) card, a neural network processing unit (NPU); the heterogeneous computing device can be other types of computing devices, such as an embedded microcomputer and a micro computing terminal.

[0109] Exemplarily, a heterogeneous computing device may be an electronic device independent of an electronic device, for example, a heterogeneous computing device is a computing card installed on a terminal device, and the terminal device interacts with the electronic device through a data network. A heterogeneous computing device may also be an electronic device installed on an electronic device, for example, the electronic device is a host, and the heterogeneous computing device is a computing card installed on the host. A heterogeneous computing device includes a computing unit, and the computing unit is used to perform computing tasks. For example, Figure 1 As shown, the heterogeneous computing device 110 includes a computing unit 111 , and the heterogeneous computing device 120 includes a computing unit 121 .

[0110] The communication between the electronic device and multiple heterogeneous computing devices, or between multiple heterogeneous computing devices can be implemented based on protocols such as PCIe links or NVLINK. In some embodiments, it can also be implemented through other links or protocols, which is not specifically limited in the embodiments of the present application.

[0111] like Figure 1 As shown, before starting model training, the electronic device 100 obtains a training script, a sample data set, and a collective communication topology table (RankTable). The training script contains the original model to be trained; the RankTable file is used to describe the topology information of the cluster devices participating in collective communication, including servers (Server), devices (Device), network information, etc.

[0112] The electronic device 100 is loaded with a parallel computing framework 101 , and the training script, sample data set, and collective communication topology table are input into the parallel computing framework 101 , which starts the computing task (model training task).

[0113] The parallel computing framework 101 includes a task scheduler, which divides the computing tasks of the model training according to a specified parallel strategy (specified by the user or based on a default configuration). In another embodiment, the computing tasks can also be divided according to a specified parallel strategy based on a third-party task scheduler independent of the electronic device 100.

[0114] The split computing tasks are distributed to different heterogeneous computing devices. The heterogeneous computing devices use the computing power unit 111 to execute the computing tasks to perform an iterative calculation. The iterative calculations are repeated multiple times until the computing task (model training task) is completed.

[0115] An iterative calculation includes forward propagation and backward update.

[0116] Specifically, forward propagation is the process of calculating the output based on the input of each neuron in sequence from the input layer to the output layer in the neural network and according to the corresponding rules, and finally obtaining the loss function under the current parameters; backward update is the process of calculating the gradient of the loss function for each parameter from the output layer to the input layer of the neural network on the basis of the forward propagation algorithm, and updating the relevant parameters at a given learning rate.

[0117] In one iterative calculation, the heterogeneous computing device executes the following process.

[0118] Forward propagation:

[0119] a) Heterogeneous computing devices perform forward calculation of local error values; (Calculation process)

[0120] b) The local error values ​​generated by all heterogeneous computing devices are integrated according to certain rules to generate a global error value. (Communication process)

[0121] Backward update:

[0122] c) The model calculates the error information of each layer from back to front; (calculation process)

[0123] d) After synchronizing all error information, update the model parameters from front to back. (Communication process)

[0124] In the above model training process, since the computing tasks are divided into multiple heterogeneous computing devices, a single computing iteration introduces a communication process, for example, process b and process d.

[0125] Data communication is achieved between multiple heterogeneous computing devices based on collective communication. Collective communication was first proposed by the message passing interface (MPI), which defines a standard message transmission interface to solve the global communication problem within the process group composed of different processes in parallel computing.

[0126] Specifically, the heterogeneous computing devices are configured with collective communication implementation devices, for example, the heterogeneous computing device 110 includes a collective communication implementation device 112, and the heterogeneous computing device 120 includes a collective communication implementation device 122. The parallel computing framework 101 generates a collective communication operator for collective communication according to the device topology relationship of the heterogeneous computing devices; the collective communication implementation device implements collective communication between different heterogeneous computing devices and between heterogeneous computing devices and electronic devices according to the collective communication operator.

[0127] During the model training process, the computing process and communication process executed by the heterogeneous computing devices are determined by the computing tasks they receive and the network topology between the heterogeneous computing devices. That is to say, after the computing tasks are distributed, the computing process and the communication process are bound to the heterogeneous computing devices; and, based on the network topology, there is a sequence of execution of the computing process and the communication process (the execution of a computing process or communication process may need to wait for the execution results of other computing processes or communication processes). This makes it impossible for the computing process and the communication process to be effectively pipelined and parallel. When the computing process and the communication process cannot be effectively pipelined and parallel, it will lead to idle and wasted computing resources during the training process, resulting in low effective computing power for the entire model training. For example, according to the measured values ​​during the training process of Pengcheng Pangu, in the operator-level model parallel scenario, the communication overhead that cannot be hidden is as high as 40%.

[0128] In view of the above problems, an embodiment of the present application provides a model training method. During the model training process, multiple virtual devices are constructed based on heterogeneous computing devices, and the multiple virtual devices are used to process the segmented computing tasks in parallel to achieve parallel training of the model.

[0129] Figure 2 Shown is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application.

[0130] like Figure 2 As shown, the hardware device for implementing model training includes an electronic device 500 and multiple virtual devices created based on heterogeneous computing devices. Figure 2 As shown, the multiple virtual devices include at least a virtual device 511 and a virtual device 512. In another embodiment, the multiple virtual devices may include more than two virtual devices.

[0131] A parallel computing framework 501 is loaded on the electronic device 500 , and the parallel computing framework 501 includes a task scheduler 502 .

[0132] The parallel computing framework 501 is used to divide the model training task into multiple computing tasks, and distribute the multiple computing tasks obtained by dividing to multiple virtual devices (for example, to virtual device 511 and virtual device 512), so as to use multiple virtual devices to process the model training task in parallel.

[0133] exist Figure 2 In the illustrated embodiment, each of the multiple virtual devices appears as a heterogeneous computing device at the parallel computing framework 501 , that is, the multiple virtual devices appear as multiple heterogeneous computing devices at the parallel computing framework 501 .

[0134] It should be noted here that multiple virtual devices are presented as multiple heterogeneous computing devices in the parallel computing framework 501. This means that from the perspective of the parallel computing framework 501, the parallel computing framework 501 perceives the multiple virtual devices as multiple heterogeneous computing devices, not that there are multiple heterogeneous computing devices in the actual sense.

[0135] Since multiple virtual devices are presented as multiple heterogeneous computing devices at the parallel computing framework 501, when the parallel computing framework 501 distributes the multiple computing tasks obtained by segmentation to multiple virtual devices, the function performed by the parallel computing framework 501 can be regarded as distributing the multiple computing tasks obtained by segmentation to multiple heterogeneous computing devices. Figure 1 The parallel computing framework 101 shown implements the parallel computing framework 501 .

[0136] Figure 3 Shown is a schematic diagram of the model training process according to an embodiment of the present application.

[0137] In one embodiment, executing Figure 3 The following process is shown to achieve parallel model training.

[0138] S300, the parallel computing framework 501 divides the model training task into multiple computing tasks, for example, divides the model training task into a first computing task and a second computing task.

[0139] S310, the parallel computing framework 501 distributes the multiple computing tasks obtained by segmentation to multiple virtual devices, wherein the multiple virtual devices correspond one-to-one to the multiple computing tasks, that is, each of the multiple computing tasks is distributed to a virtual device, and different computing tasks are distributed to different virtual devices.

[0140] For example, the first computing task is distributed to virtual device 511 , and the second computing task is distributed to virtual device 512 .

[0141] S320, multiple virtual devices execute the distributed computing tasks to perform iterative calculations for model training.

[0142] For example, virtual device 511 performs a first computing task, and virtual device 512 performs a second computing task.

[0143] According to the method of the embodiment of the present application, since the parallel execution logic of the computing task is placed in the virtual device of the reusable computing resource, the binding relationship between the computing task, the communication task and the hardware (heterogeneous computing device) is decoupled, and the effect of multi-process parallelism, communication and computing parallelism is achieved in a macro sense, and the time to complete the communication is theoretically completely hidden.

[0144] Furthermore, since the parallel execution logic of computing tasks is placed in a virtual device that can reuse computing resources, during the iterative calculation of model training, computing tasks and communication tasks do not need to rely on the hardware topology of heterogeneous computing devices, but rely on the topology of virtual devices in the computing task execution device set. Therefore, when maximizing the computing-to-communication ratio, there is no need to understand the hardware topology of heterogeneous computing devices in detail, which reduces the difficulty of achieving the maximization of the computing-to-communication ratio.

[0145] Furthermore, according to the method of the embodiment of the present application, a parallel computing framework for distributing tasks to multiple heterogeneous computing devices is used to implement task distribution to multiple virtual devices. Since the virtual devices are presented as heterogeneous computing devices at the parallel computing framework, there is no need to adjust the parallel computing framework for distributing tasks to multiple heterogeneous computing devices, thereby reducing the difficulty of implementing task distribution to virtual devices.

[0146] exist Figure 2 In the illustrated embodiment, the virtual device is a virtual device created based on a heterogeneous computing device, wherein the multiple virtual devices may be virtual devices created based on a single heterogeneous computing device or may be virtual devices created based on multiple heterogeneous computing devices.

[0147] For example, Figure 4 Shown is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application.

[0148] like Figure 4 As shown, the hardware device for implementing model training includes an electronic device 600 and a heterogeneous computing device 610.

[0149] The electronic device 600 may refer to the electronic device 100 , and the heterogeneous computing device 610 may refer to the heterogeneous computing device 110 or the heterogeneous computing device 120 .

[0150] A virtual device 611 and a virtual device 612 are created based on the heterogeneous computing device 610 .

[0151] The hardware resources of the heterogeneous computing device 610 include a shared memory 613 , a communication link 614 , and a computing unit 615 .

[0152] The shared memory 613 may be all memory resources of the heterogeneous computing device 610 or part of the memory resources of the heterogeneous computing device 610. The virtual devices 611 and 612 may synchronize data based on the shared memory 613.

[0153] The communication link 614 is a data transmission link in the heterogeneous computing device 610 that can be provided to the virtual device for use, for example, a PCIe link or a network device link.

[0154] The computing unit 615 is a computing resource of the heterogeneous computing device 610 . The virtual device 611 and the virtual device 612 can share the computing unit 615 , for example, by time-sharing the computing unit 615 .

[0155] For example, Figure 5 Shown is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application.

[0156] like Figure 5 As shown, the hardware devices for implementing model training include an electronic device 700, a heterogeneous computing device 710, and a heterogeneous computing device 720.

[0157] The electronic device 700 may refer to the electronic device 100 , and the heterogeneous computing device 710 and the heterogeneous computing device 720 may refer to the heterogeneous computing device 110 or the heterogeneous computing device 120 .

[0158] The heterogeneous computing device 710 and the heterogeneous computing device 720 may be heterogeneous computing devices of the same electronic device. For example, the heterogeneous computing device 710 and the heterogeneous computing device 720 are both installed on the electronic device 700. For another example, the heterogeneous computing device 710 and the heterogeneous computing device 720 are both installed on another electronic device other than the electronic device 700, and the electronic device on which the heterogeneous computing device 710 and the heterogeneous computing device 720 are installed can communicate and interact with the electronic device 700.

[0159] The heterogeneous computing device 710 and the heterogeneous computing device 720 may be heterogeneous computing devices of different electronic devices. For example, the heterogeneous computing device 710 is installed on the electronic device 700, and the heterogeneous computing device 720 is installed on another electronic device other than the electronic device 700, and the electronic device on which the heterogeneous computing device 720 is installed can communicate and interact with the electronic device 700. For another example, the heterogeneous computing device 710 and the heterogeneous computing device 720 are respectively installed on two other electronic devices other than the electronic device 700, and the other two electronic devices on which the heterogeneous computing device 710 and the heterogeneous computing device 720 are installed can communicate and interact with the electronic device 700.

[0160] A virtual device 711 and a virtual device 712 are created based on the heterogeneous computing device 710 ; a virtual device 721 and a virtual device 722 are created based on the heterogeneous computing device 720 .

[0161] The hardware resources of the heterogeneous computing device 710 include a shared memory 713 , a communication link 714 , and a computing unit 715 .

[0162] The shared memory 713 may be all memory resources of the heterogeneous computing device 710 or part of the memory resources of the heterogeneous computing device 710. The virtual devices 711 and 712 may synchronize data based on the shared memory 713.

[0163] The communication link 714 is a data transmission link in the heterogeneous computing device 710 that can be provided to the virtual device for use, for example, a PCIe link or a network device link.

[0164] The computing unit 715 is a computing resource of the heterogeneous computing device 710 . The virtual device 711 and the virtual device 712 can share the computing unit 715 , for example, by time-division multiplexing the computing unit 717 .

[0165] The hardware resources of the heterogeneous computing device 720 include a shared memory 723 , a communication link 724 , and a computing unit 725 .

[0166] The shared memory 723 may be all memory resources of the heterogeneous computing device 720 , or may be part of the memory resources of the heterogeneous computing device 720 .

[0167] The communication link 724 is a data transmission link in the heterogeneous computing device 720 that can be provided to the virtual device for use, for example, a PCIe link or a network device link.

[0168] The computing unit 725 is a computing resource of the heterogeneous computing device 720 . The virtual device 721 and the virtual device 722 can share the computing unit 725 , for example, by time-division multiplexing the computing unit 725 .

[0169] Furthermore, in order to enable the parallel computing framework 501 to distribute parallel computing tasks to virtual devices, a communication virtualization device 503 is also deployed in the electronic device 500.

[0170] The communication virtualization device 503 is deployed outside the parallel computing framework 501. The communication virtualization device 503 is used to present multiple virtual devices in the form of multiple heterogeneous computing devices to the parallel computing framework 501, so that the parallel computing framework 501 can distribute computing tasks to the virtual devices according to the communication mode for heterogeneous computing devices, without the need to perform communication settings for the virtual devices.

[0171] Those skilled in the art may construct the communication virtualization device 503 in combination with the properties of the virtual device. The present application does not specifically limit the specific implementation of the communication virtualization device 503.

[0172] According to the method of the embodiment of the present application, a communication virtualization device is deployed outside the parallel computing framework, and the communication between the parallel computing framework and the virtual device is realized through the communication virtualization device. The parallel computing framework does not need to perceive the deployment and implementation status of the virtual device on the heterogeneous computing device, thereby reducing the difficulty of realizing communication between the parallel computing framework and the virtual device.

[0173] Figure 6 Shown is a schematic diagram of the model training process according to an embodiment of the present application.

[0174] S800 , creating multiple virtual devices, where the multiple virtual devices at least include a virtual device 511 and a virtual device 512 .

[0175] S801, the electronic device 500 queries all available virtual devices.

[0176] S802: The electronic device 500 generates a virtual device topology relationship table according to available virtual devices.

[0177] S803, the electronic device 500 obtains a training script and a sample data set.

[0178] S804, the electronic device 500 sends the training script, the sample data set and the virtual device topology relationship table to the parallel computing framework 501.

[0179] S805, the parallel computing framework 501 divides the sample data set according to the user configuration and generates multi-path parallel computing tasks according to the training script.

[0180] S806, the task scheduler 502 selects a virtual device for executing the computing task from multiple virtual devices according to the virtual device topology relationship table to form a computing task execution device set, which includes multiple virtual devices (at least virtual device 511 and virtual device 512).

[0181] According to the method of the embodiment of the present application, a virtual device is created before dividing the training task, and a virtual device for executing the computing task is selected from multiple virtual devices based on the divided computing tasks. This can ensure the flexibility of computing task distribution and reduce the time consumption of computing task distribution.

[0182] S807, the task scheduler 502 sends the computing task to the virtual devices in the computing task execution device set (including at least the virtual device 511 and the virtual device 512).

[0183] The communication between the parallel computing framework 501 and the virtual device is implemented through the communication virtualization device 502 .

[0184] S810, the virtual devices (including at least virtual device 511 and virtual device 512) calculate the distributed computing tasks to perform iterative calculations for model training.

[0185] Furthermore, during the process of iterative calculations performed by multiple virtual devices, there may be a need for communication between the virtual devices.

[0186] For example, in one embodiment, there is a connection relationship between computing tasks. The execution of one computing task requires the introduction of the computing result of another computing task. For example, the parallel computing framework 501 divides the model training task into multiple computing tasks, and the multiple computing tasks include at least computing task A and computing task B. The execution of computing task B requires the introduction of the computing result of computing task A. The parallel computing framework 501 distributes computing task A and computing task B to virtual device 511 and virtual device 512 respectively.

[0187] Figure 7 The figure is a schematic diagram of the model training part process according to an embodiment of the present application.

[0188] S900, the virtual device 511 receives computing task A.

[0189] S901, the virtual device 511 executes computing task A. After computing task A is executed, the virtual device 511 generates a computing result of computing task A.

[0190] S910, the virtual device 512 receives computing task B.

[0191] For example, calculation task A and calculation task B are the error value calculation steps of two adjacent layers in the process of forward calculation of local error value in step a. The layer corresponding to calculation task A is the previous layer of the layer corresponding to calculation task B. The execution of calculation task B needs to be calculated based on the calculation result of calculation task A.

[0192] For another example, calculation task A is the step of calculating the error value of a certain layer in the process of forward calculating the local error value in step a. Calculation task B is the step of integrating and generating the global error value in step b. The execution of calculation task B requires the collection of the calculation results of calculation task A.

[0193] S911, the virtual device 512 determines that the communication object to be communicated is the virtual device 511 according to the virtual device topology relationship table of the computing task execution device set and the computing task B.

[0194] S912 , the virtual device 512 subscribes to the task execution status of the virtual device 511 .

[0195] S902, after the virtual device 511 completes the computing task A, it issues a computing task completion event.

[0196] S913, after receiving the task completion event, the virtual device 512 requests the virtual device 511 to obtain the calculation result of the calculation task A.

[0197] S914, the virtual device 512 obtains the calculation result of the calculation task A.

[0198] Specifically, in one embodiment, virtual device 511 and virtual device 512 are virtual devices created in the same heterogeneous computing device, and virtual device 512 obtains the first calculation result through internal communication of the heterogeneous computing device. In another embodiment, virtual device 511 and virtual device 512 are virtual devices created in different heterogeneous computing devices, and virtual device 512 obtains the first calculation result through communication between heterogeneous computing devices. For example, through network channel communication between heterogeneous computing devices, or PCIe data channel communication between heterogeneous computing devices.

[0199] Furthermore, those skilled in the art may use a variety of methods to enable the virtual device 512 to obtain the first calculation result.

[0200] For example, virtual device 511 and virtual device 512 are virtual devices created in the same heterogeneous computing device. Virtual device 511 saves the first calculation result to the shared memory; virtual device 511 publishes the storage location (memory address) of the first calculation result to virtual device 512; virtual device 512 directly reads the first calculation result in the shared memory according to the storage location of the first calculation result.

[0201] For another example, virtual device 511 and virtual device 512 are virtual devices created in different heterogeneous computing devices. Virtual device 511 sends the first calculation result to the heterogeneous computing device where virtual device 512 is located through the communication channel between heterogeneous computing devices. Alternatively, virtual device 511 sends the storage address of the first calculation result to the heterogeneous computing device where virtual device 512 is located through the communication channel between heterogeneous computing devices; virtual device 512, based on the communication channel between heterogeneous computing devices, directly accesses the memory of the heterogeneous computing device where virtual device 511 is located (Remote Direct Memory Access (RDMA)).

[0202] S915, the virtual device 512 executes computing task B according to the computing result of computing task A.

[0203] According to the method of the embodiment of the present application, the virtual device perceives the real-time status of each computing task, and a communication task can be initiated in real time after a single computing task is completed. Compared with the parallel computing framework that uniformly manages the status of all computing tasks, after a single computing task is completed, it is necessary to wait for the computing framework to issue a communication task to implement communication. The method according to the embodiment of the present application can reduce computing power consumption.

[0204] Furthermore, when the first virtual device and the second virtual device are virtual devices created in the same heterogeneous computing device, the communication between the first virtual device and the second virtual device is essentially the internal communication of the heterogeneous computing device. The internal communication of the heterogeneous computing device does not need to involve processes such as interface calling and communication link establishment, and the communication time consumption can be ignored in theory.

[0205] Therefore, in one embodiment, when distributing computing tasks, computing tasks with communication requirements are preferentially distributed to virtual devices created in the same heterogeneous computing device, thereby reducing the waiting time when the virtual device executes the computing task.

[0206] According to the method of the embodiment of the present application, multiple virtual devices are deployed in a single heterogeneous computing device to implement multi-process (multi-channel data) distributed parallel training in a single heterogeneous computing device. Theoretically, it is expected that the BatchSize deployed in a single heterogeneous computing device will be increased by more than double.

[0207] Figure 8 Shown is a schematic diagram of a virtual device structure according to an embodiment of the present application.

[0208] like Figure 8 As shown, in one embodiment, the virtual device 1000 includes a routing device 1010 and a task tracking device 1020 .

[0209] Fig. 9 The figure is a schematic diagram of the model training part process according to an embodiment of the present application.

[0210] In the iterative calculation, Figure 8 The virtual device shown is executed as Fig. 9 The following process is shown.

[0211] S1100, the task tracking device 1020 obtains a computing task.

[0212] S1101, the routing device 1010 obtains a virtual device topology relationship table of a computing task execution device set.

[0213] S1110, the task tracking device 1020 determines, according to the computing task, a virtual device that needs to subscribe to the task execution status (a virtual device that needs to copy data).

[0214] When the task tracking device 1020 does not need to subscribe to the task execution status of other virtual devices, S1120 is executed.

[0215] S1120, the task tracking device 1020 sends the computing task to the computing power unit of the heterogeneous computing device, executes the computing task, and obtains the computing result.

[0216] S1121, the task tracking device 1020 monitors the execution status of the computing task, and after the computing task is completed, publishes a computing task completion event.

[0217] When the task tracking device 1020 does not need to subscribe to the task execution status of other virtual devices, S1130 is executed.

[0218] S1130, the task tracking device 1020 sends a device list of virtual devices that need to subscribe to the task execution status to the routing device 1010, and notifies the routing device 1010 to establish a communication connection.

[0219] S1131, the routing device 1010 establishes a communication connection with the virtual device that needs to subscribe to the task execution status according to the virtual device topology relationship table of the computing task execution device set and the device list of the virtual device that needs to subscribe to the task execution status, so as to realize the task execution status of the virtual device being subscribed.

[0220] S1132, the task tracking device 1020 waits for a computing task completion event of the virtual device that has subscribed to the task execution status, and after receiving the computing task completion event, obtains the computing result of the virtual device that has subscribed to the task execution status.

[0221] After S1132, S1120 is executed.

[0222] Furthermore, in one embodiment, the communication between the electronic device and the virtual device, and between the virtual devices, is implemented based on collective communication.

[0223] Collective Communications is a global communication operation in which all processes in a process group participate. The most basic operations are: send, receive, copy, intra-group process barrier synchronization, and inter-node process synchronization (signal+wait). These basic operations are combined to form a set of communication templates, also called communication primitives (or collective communication operators).

[0224] For example, collective communication operators include: 1-to-many broadcast, many-to-1 gather, many-to-many gather, 1-to-many scatter, many-to-1 reduce, many-to-many reduce, combined reduce and scatter, many-to-all, etc.

[0225] Specifically, a routing device is deployed outside the parallel computing framework, and the routing device is used to generate a collective communication operator for collective communication according to the virtual device topological relationship of the plurality of virtual devices. In this way, in the iterative computing process, the implementation of collective communication does not require the participation of the parallel computing framework, thereby improving the flexibility and communication efficiency of collective communication.

[0226] Fig.10 Shown is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application.

[0227] like Fig.10 As shown, the hardware device for implementing model training includes an electronic device 1200 and multiple virtual devices created based on heterogeneous computing devices, and the multiple virtual devices include at least a virtual device 1210 and a virtual device 1220.

[0228] The electronic device 1200 includes a parallel computing framework 1201 and a communication virtualization device 1202 . The parallel computing framework 1201 includes a task scheduler 1205 .

[0229] The parallel computing framework 1201 sends a collective communication command to the virtual device through the communication virtualization device 1202 .

[0230] The communication virtualization device 1202 includes a collective communication resource virtualization module 1203 and a virtual device super-resolution module 1204 .

[0231] The collective communication resource virtualization module 1203 is used to present the collective communication attributes of virtual devices to upper-level users and parallel computing framework 1201. Specifically, collective communication attribute resources are configured for each virtual device, including but not limited to Rank, Group, etc.; tools for querying communication topology attributes are supported and modified to provide support for the topology and communication attributes of virtual devices.

[0232] The collective communication resource virtualization module 1203 is also used to support virtualization of collective communication physical resources. Specifically, it supports the control flow notification mechanism between virtual devices, and splits and super-divides hardware communication resources (including but not limited to network channels (RDMA), PCIe data channels, etc.).

[0233] The virtual device super-resolution module 1204 is used to optimize the virtualization technology, supporting spatial division multiplexing of resources between multiple virtual devices on a single card and time division multiplexing of the same resources; for example, virtual device 1211 occupies computing resources to perform computing tasks, while virtual device 1212 occupies network resources to perform communication tasks.

[0234] The routing device is deployed in each virtual device. Specifically, each of the multiple virtual devices is configured with a routing device, a task tracking device, and a diversity link implementation device. Taking virtual device 1210 as an example, virtual device 1210 includes a routing device 1211, a task tracking device 1212, and a diversity link implementation device 1213.

[0235] The routing device 1211 stores a virtual device routing table, which is used to record all virtual devices and topological connection relationships of the current computing task that need to copy data. In one embodiment, the routing device 1211 generates a virtual device routing table based on the virtual device topological relationship table of the computing task execution device set and the device list of virtual devices that need to subscribe to the task execution status determined by the task tracking device 1212.

[0236] For example, in an application scenario, in a routing device of a virtual device, the content of the virtual device routing table is as shown in Table 1.

[0237] Table 1

[0238] Rank(Device ID) Memory Address state protocol Process 0 0x1000 Completed Shared Memory Memcpy 1 0x2000 Receiving RDMA Recv() N 0xffff Wait for calculation to complete PCIe / NVLINK DMA / …

[0239] The routing device 1211 is used to generate a collective communication operator according to the virtual device routing table, and based on the collective communication operator, drive the diversity link implementation device to select a communication link to establish a communication connection. The routing device 1211 is triggered by the task tracking device 1212.

[0240] The diversity link implementation device 1213 is used to select a communication link according to the collective communication operator, so as to establish a communication connection based on the selected communication link. For example, the diversity link implementation device 1213 selects a corresponding hardware unit (DMA controller, network controller) to establish a communication connection and implement data synchronization according to the type of the collective communication operator.

[0241] Specifically, in one embodiment, the diversity link implementation device 1213 implements data synchronization specifically as follows:

[0242] The virtual devices of the same heterogeneous computing device communicate data by passing memfd through shared memory;

[0243] Support link resource segmentation between virtual devices of different heterogeneous computing devices of the same electronic device according to hardware type (PCIE / HCCS);

[0244] Between virtual devices of different heterogeneous computing devices across electronic devices, the RDMA / TCP protocol is selected for implementation, and multi-process port isolation is adopted between virtual devices.

[0245] According to the method of the embodiment of the present application, the parallel computing framework is not responsible for managing the device topology and communication between devices, and the collective communication capabilities are transferred to the virtual devices.

[0246] Furthermore, in the iterative computing process, there is an application scenario where a computing task of a virtual device (task migration device) is migrated to another virtual device (task migration device) for execution. After the computing task is migrated, the task migration device in the computing task execution device set is replaced by the task migration device.

[0247] In the above application scenario, in one embodiment, computing task migration is completed on the virtual device side, and the front-end user interface and parallel computing framework on the electronic device side do not need to perceive and respond to computing task migration. That is, after computing task migration occurs, information indicating changes in the topological relationship of the virtual device is not sent to the parallel computing framework, so that the parallel computing framework does not perceive changes in the topological relationship of the virtual device. In this way, computing task migration does not need to trigger the re-segmentation and distribution of computing tasks, and computing task migration will not interrupt the execution of a single iterative calculation.

[0248] Fig.11 Shown is a schematic diagram of a virtual device according to an embodiment of the present application.

[0249] like Fig.11 As shown, virtual device 1301 (first virtual device), virtual device 1302 (fourth virtual device) and virtual device 1303 (second virtual device) are virtual devices of the computing task execution device set. Virtual device 1304 (third virtual device) is a virtual device outside the computing task execution device set.

[0250] Virtual device 1301, virtual device 1302, and virtual device 1303 are respectively distributed with computing task S1 (first computing task), computing task S2 (third computing task), and computing task S3 (second computing task). The execution of computing task S2 requires the introduction of the computing result of computing task S1, so after the computing task is distributed, virtual device 1302 subscribes to the task execution status of virtual device 1301. The execution of computing task S3 does not require the introduction of the computing result of computing task S1, so after the computing task is distributed, virtual device 1303 does not subscribe to the task execution status of virtual device 1301.

[0251] Fig.12 Shown is a schematic diagram of a computing task migration process according to an embodiment of the present application.

[0252] S1400, the virtual device 1301 receives the computing task S1.

[0253] S1401, the virtual device 1302 receives the computing task S2.

[0254] S1402, the virtual device 1303 receives the computing task S3.

[0255] S1403, the virtual device 1301 executes the computing task S1.

[0256] S1404 , the virtual device 1302 subscribes to the task execution status of the virtual device 1301 .

[0257] S1405, the virtual device 1303 executes the computing task S3 and generates a computing result of the computing task S3.

[0258] S1406, triggering computing task migration, the computing task migration is computing task S1 of virtual device 1301 migrating to virtual device 1304.

[0259] Specifically, the present application does not impose any specific restrictions on the triggering conditions for computing task migration. For example, in one embodiment, the execution of a computing task is abnormal, which triggers the computing task migration. In another embodiment, the dynamic scheduling of a virtual device triggers the computing task migration.

[0260] S1410, the virtual device 1301 terminates the execution of the computing task S1.

[0261] S1410, the virtual device 1301 issues a task termination event.

[0262] S1411, the virtual device 1302 receives a task termination event.

[0263] Since the virtual device 1303 does not subscribe to the task execution status of the virtual device 1301 , the virtual device 1303 does not receive the task termination event, or the virtual device 1303 does not respond after receiving the task termination event.

[0264] S1412 , the virtual device 1302 subscribes to the task execution status of the virtual device 1304 according to the task termination event.

[0265] S1420 , synchronizing the computing data of the virtual device 1301 and the virtual device routing table in the routing apparatus to the virtual device 1304 .

[0266] S1421, after data synchronization is completed, the virtual device 1304 publishes a task migration event.

[0267] During the data synchronization process, the virtual device 1304 updates the synchronized virtual device routing table.

[0268] S1422, the virtual device 1302 receives a task migration event.

[0269] Since the virtual device 1303 does not subscribe to the task execution status of the virtual device 1304, the virtual device 1303 does not receive the task termination event, or the virtual device 1303 does not respond after receiving the task termination event.

[0270] S1423, virtual device 1302 cancels the subscription to virtual device 1301, and updates the virtual device topology relationship table of the computing task execution device set (updates the virtual device routing table).

[0271] Specifically, in one embodiment, after receiving the task termination event, if the virtual device 1302 does not receive the task migration event, the virtual device 1302 determines that the computing task migration fails, reports an error, terminates the current iterative calculation, or triggers the computing task migration to be re-executed.

[0272] After S1423, the information indicating the change in the topological relationship of the virtual devices is not fed back to the parallel computing framework, so that the parallel computing framework does not perceive the change in the topological relationship of the virtual devices.

[0273] After S1421 , the virtual device 1304 executes S1430 .

[0274] S1430, the virtual device 1304 continues to execute the computing task S1 and generates the computing result of the computing task S1.

[0275] S1431, after the virtual device 1304 completes the computing task S1, it issues a computing task completion event.

[0276] S1432, after receiving the computing task completion event, the virtual device 1302 obtains the computing result of the computing task S1.

[0277] S1433, the virtual device 1302 executes the computing task S2 according to the computing result of the computing task S1, and generates the computing result of the computing task S2.

[0278] S1430, after the end of this iteration, the virtual devices of the computing task execution device set (including virtual device 1302, virtual device 1303, virtual device 1304, excluding virtual device 1301) synchronously update the virtual device topology relationship table of the computing task execution device set (update the virtual device routing table in the routing device), wherein the virtual device topology relationship table of the virtual device routing table in the routing device of virtual device 1303 is synchronously updated to the virtual device topology relationship table of the virtual device routing table in the routing devices of virtual devices 1302 and 1304.

[0279] In one embodiment, after S1430, no information indicating a change in the topological relationship of the virtual devices is fed back to the parallel computing framework, and the communication virtualization device presents an unchanged topological relationship of the virtual devices at the parallel computing framework by replacing descriptive information such as the virtual device name. For example, before the computing task of virtual device 1301 is migrated to virtual device 1304, at the parallel computing framework, the communication virtualization device presents virtual device 1301 as heterogeneous computing device A. After the computing task of virtual device 1301 is migrated to virtual device 1304, at the parallel computing framework, the communication virtualization device presents virtual device 1304 as heterogeneous computing device A. In this way, at the parallel computing framework, the device topological relationship of multiple heterogeneous computing devices presented by multiple virtual devices remains unchanged.

[0280] In another embodiment, after S1430, information indicating the change in the virtual device topology is fed back to the parallel computing framework, so that the parallel computing framework can perform computing task segmentation and distribution for the next iteration according to the new virtual device topology.

[0281] Fig.13 Shown is a schematic diagram of a computing task migration process according to an embodiment of the present application.

[0282] S1500, the virtual device 1301 receives the computing task S1.

[0283] S1501, the virtual device 1302 receives the computing task S2.

[0284] S1502, the virtual device 1303 receives the computing task S3.

[0285] S1503, the virtual device 1301 executes the computing task S1.

[0286] S1504 , the virtual device 1302 subscribes to the task execution status of the virtual device 1301 .

[0287] S1505, the virtual device 1303 executes the computing task S3 and generates a computing result of the computing task S3.

[0288] S1506, triggering computing task migration, the computing task migration is computing task S1 of virtual device 1301 migrating to virtual device 1304.

[0289] S1510, the virtual device 1301 terminates the execution of the computing task S1.

[0290] S1511, the virtual device 1301 broadcasts a task termination event to the computing task execution device set.

[0291] The virtual devices (including virtual device 1302 and virtual device 1303) of the computing task execution device set receive a task termination event, wherein virtual device 1303 may receive the task termination event during the execution of computing task S3 or after the execution of computing task S3.

[0292] S1520 , synchronize the computing data of virtual device 1301 and the virtual device routing table to virtual device 1304 .

[0293] S1521, after data synchronization is completed, the virtual device 1304 broadcasts a task migration event to the computing task execution device set.

[0294] The virtual devices (including virtual device 1302 and virtual device 1303) of the computing task execution device set receive a task migration event, wherein virtual device 1303 may receive the task migration event during the execution of computing task S3 or after the execution of computing task S3.

[0295] Specifically, in one embodiment, after the virtual devices (including virtual devices 1302 and virtual devices 1303) of the computing task execution device set receive a task termination event but do not receive a task migration event, it is determined that the computing task migration has failed, and the current iterative calculation is terminated or the computing task migration is triggered to be re-executed.

[0296] S1522 , the virtual device 1302 cancels the subscription to the virtual device 1301 , and subscribes to the task execution status of the virtual device 1304 .

[0297] S1523, the virtual devices (including virtual device 1302 and virtual device 1303) of the computing task execution device set update the virtual device topology relationship table of the computing task execution device set (update the virtual device routing table in the routing device) after receiving the task migration event.

[0298] After S1523, the information indicating the change in the topological relationship of the virtual devices is not fed back to the parallel computing framework, so that the parallel computing framework does not perceive the change in the topological relationship of the virtual devices.

[0299] After S1523 , the virtual device 1304 executes S1530 .

[0300] S1530, the virtual device 1304 continues to execute the computing task S1 and generates the computing result of the computing task S1.

[0301] S1531, after the virtual device 1304 completes the computing task S1, it issues a computing task completion event.

[0302] S1532, after receiving the computing task completion event, the virtual device 1302 obtains the computing result of the computing task S1.

[0303] S1533, the virtual device 1302 executes the computing task S2 according to the computing result of the computing task S1, and generates the computing result of the computing task S2.

[0304] According to the method of the embodiment of the present application, the parallel computing framework is not responsible for managing the device topology and communication between devices. The collective communication capabilities are transferred to the virtual devices, supporting the dynamic migration of computing tasks during iterative computing and the dynamic update of virtual device link information. There is no need for the computing framework layer to perceive and recompile the generated communication operators, thus reducing the waste of computing power waiting.

[0305] In the large-model distributed training scenario, the product implementation form of the present invention is contained in the machine learning and deep learning platform software and is deployed as program code on the server hardware. The program code of the present invention exists inside the runtime engine and communication management module of the platform software, as well as outside the existing modules. During operation, the program code of the present invention runs in the electronic device memory of the server and the memory of heterogeneous devices such as NPU / GPU / Ascend.

[0306] Reference Figure 8 The illustrated embodiments and Fig.10 In the embodiment shown, in one embodiment of the present application, the routing device is deployed in each virtual device. Specifically, the routing device and the task tracking device are deployed in each virtual device of the computing task execution device set.

[0307] For example, in one embodiment, the Mindspor distributed training framework is used to implement large model distributed training based on vNPU virtual devices.

[0308] Fig.14 Shown is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application.

[0309] like Fig.14 As shown, the hardware device for implementing model training includes an electronic device 1600. The electronic device 1600 includes a Mindspor distributed training framework 1601. The Mindspor distributed training framework 1601 includes a task scheduler 1602.

[0310] The electronic device 1600 is also equipped with a communication virtualization device 1603. The communication virtualization device 1603 virtualizes the heterogeneous computing device (NPU device) into multiple virtual devices (vNPU devices) for the Mindspor distributed training framework 1601 and the user to present the heterogeneous computing device with a "one virtual multiple" effect. Fig.12As shown, the multiple virtual devices include at least virtual device 1610, virtual device 1620, ...

[0311] The user specifies one or more virtual devices for model training by configuring the RankTable; the Mindspor distributed training framework 1501 compiles and splits the training tasks according to the parallel strategy specified by the user, generates multi-path computing tasks, and schedules virtual devices to complete the training.

[0312] exist Fig.12 In the illustrated embodiment, corresponding routing means, task tracking means and diversity link implementation means are deployed in each virtual device. For example, virtual device 1610 includes routing means 1611, task tracking means 1612 and diversity link implementation means 1613; virtual device 1620 includes routing means 1621, task tracking means 1622 and diversity link implementation means 1623.

[0313] In another embodiment, the routing device is deployed in the electronic device. Specifically, the routing device and the task tracking device are deployed in the electronic device, and the routing device and the task tracking device deployed in the electronic device manage each virtual device of the computing task execution device set.

[0314] For example, in one embodiment, the PyTorch distributed training framework is used to implement large model distributed training based on vNPU or vGPU devices.

[0315] Fig.15 Shown is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application.

[0316] like Fig.15 As shown, the hardware device for implementing model training includes an electronic device 1700. The electronic device 1700 includes a PyTorch distributed training framework 1701. The PyTorch distributed training framework 1701 includes a task scheduler 1702.

[0317] The electronic device 1700 is also equipped with a communication virtualization device 1703. The communication virtualization device 1703 virtualizes the heterogeneous computing device into multiple virtual devices (vNPU devices or vGPU devices) for the PyTorch distributed training framework 1701 and the user to present the heterogeneous computing device with a "one virtual multiple" effect. Fig.13 As shown, the multiple virtual devices include at least virtual device 1710, virtual device 1720, ...

[0318] The PyTorch distributed training framework 1701 is a single communication operator delivery mode, and all collective communication operators are generated and delivered in the electronic device. Specifically, a routing device 1704 and a task tracking device 1705 are deployed in the electronic device 1700. Multiple instance objects are created in the routing device 1704 and the task tracking device 1705. The instance objects of the routing device 1704 and the task tracking device 1705 maintain a one-to-one relationship with the virtual device, and each instance object manages a virtual device. A diversity link implementation device is deployed in each virtual device. For example, virtual device 1710 includes a diversity link implementation device 1713; virtual device 1720 includes a diversity link implementation device 1723.

[0319] In the computing task migration application scenario, the routing device 1704 and the task tracking device 1705 jointly complete the dynamic switching of links when the topology changes without the need for the PyTorch distributed training framework 1701 to perceive and intervene.

[0320] According to the method of the embodiment of the present application, an embodiment of the present application further proposes a model training device, which includes a parallel computing framework and multiple virtual devices, wherein:

[0321] The multiple virtual devices are virtual devices created based on heterogeneous computing devices;

[0322] The parallel computing framework is loaded on the electronic device, and is used to divide the model training task into multiple computing tasks, and distribute the multiple computing tasks to multiple virtual devices, wherein the multiple virtual devices correspond to the multiple computing tasks one by one;

[0323] Multiple virtual devices are used to execute distributed computing tasks to perform iterative calculations for model training.

[0324] Optionally, in one embodiment, the device further includes a communication virtualization device, wherein:

[0325] The communication virtualization device is deployed on the electronic device, wherein the communication virtualization device is deployed outside the parallel computing framework;

[0326] The communication virtualization device is used to present multiple virtual devices to a parallel computing framework in the form of multiple heterogeneous computing devices.

[0327] Optionally, in one embodiment, the communication between the parallel computing framework and the virtual device, and between different virtual devices, is collective communication;

[0328] The communication virtualization device includes:

[0329] A collective communication resource virtualization module, which is used to configure collective communication attribute resources for virtual devices and present the collective communication attributes of virtual devices to the parallel computing framework;

[0330] The virtual device super-resolution module is used to support virtualization of collective communication physical resources.

[0331] Optionally, in one embodiment, the model training device further includes a routing device, wherein:

[0332] The routing device is deployed outside the parallel computing framework and is used to generate a collective communication operator for collective communication according to the virtual device topological relationship of multiple virtual devices.

[0333] Optionally, in one embodiment, the routing device is used to save a virtual device routing table, and the virtual device routing table is used to record all virtual devices and topological connection relationships that require data copying for the current computing task.

[0334] Optionally, in one embodiment, the routing device is deployed on an electronic device, and the routing device includes instance objects corresponding one-to-one to multiple virtual devices.

[0335] Optionally, in one embodiment, the routing device is deployed in each virtual device.

[0336] Optionally, in one embodiment: the plurality of computing tasks include at least a first computing task and a second computing task; the plurality of virtual devices include at least a first virtual device and a second virtual device; the parallel computing framework distributes the first computing task to the first virtual device, and distributes the second computing task to the second virtual device; the first virtual device executes the first computing task, and the second virtual device executes the second computing task;

[0337] The apparatus further includes a task migration module and a third virtual device, wherein:

[0338] The task migration module is used to: in the case of triggering task migration from the first virtual device to the third virtual device, terminate the first virtual device from executing the first computing task, and maintain the second virtual device from continuing to execute the second computing task; perform task migration, including synchronizing the first computing task in the first virtual device and the current execution state of the first computing task to the third virtual device;

[0339] The third virtual device is used to continue executing the first computing task after the task migration.

[0340] Optionally, in one embodiment, the routing device is further configured to, after the task migration, update the topological relationship of the virtual devices in the routing device according to the task migration.

[0341] Optionally, in one embodiment, the routing device is further configured to, after task migration, not feed back information indicating changes in the topological relationship of the virtual devices to the parallel computing framework.

[0342] Optionally, in one embodiment, the plurality of computing tasks further include at least a third computing task, and execution of the third computing task requires reference to a computing result of the first computing task; the plurality of virtual devices further include at least a fourth virtual device;

[0343] The parallel computing framework is used to distribute the third computing task to the fourth virtual device;

[0344] The fourth virtual device is used to: subscribe to the task execution status of the first virtual device according to the third computing task; after the first virtual device completes the execution of the first computing task, the fourth virtual device obtains the calculation result of the first computing task; and execute the third computing task according to the calculation result of the first computing task.

[0345] Optionally, in one embodiment, the fourth virtual device is further used for:

[0346] After the task migration, the task execution state of subscribing to the first virtual device is updated to the task execution state of subscribing to the third virtual device.

[0347] Optionally, in one embodiment:

[0348] The task migration module is used to synchronize the virtual device topology relationship in the routing device of the first virtual device to the routing device of the third virtual device;

[0349] The routing device of the third virtual device is used to update the virtual device topology relationship in the routing device of the third virtual device according to the task migration after the task migration;

[0350] The routing device of the fourth virtual device is used to update the virtual device topology relationship in the routing device of the fourth virtual device according to the task migration after the task migration.

[0351] In the description of the embodiments of the present application, for the convenience of description, the device is described by dividing it into various modules according to its functions. The division of each module is merely a division of logical functions. When implementing the embodiments of the present application, the functions of each module can be implemented in the same or one or more software and / or hardware.

[0352] Specifically, the device proposed in the embodiment of the present application can be fully or partially integrated into a physical entity during actual implementation, or it can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the detection module can be a separately established processing element, or it can be integrated in a chip of an electronic device. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. In the implementation process, each step of the above method or each of the above modules can be completed by an integrated logic circuit of hardware in a processor element or instructions in the form of software.

[0353] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0354] The application also provides an electronic device.

[0355] Fig.16 FIG. 4 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application.

[0356] like Fig.16 As shown, the electronic device 2300 includes a memory 2302 for storing computer program instructions and a processor 2301 for executing program instructions, wherein when the computer program instructions are executed by the processor 2301, the electronic device 2300 is triggered to execute the method steps of the method described in the embodiment of the present application.

[0357] Specifically, in one embodiment of the present application, the above-mentioned one or more computer programs are stored in the above-mentioned memory 2302, and the above-mentioned one or more computer programs include instructions. When the above-mentioned instructions are executed by the above-mentioned electronic device 2300, the above-mentioned electronic device 2300 executes the method steps described in the embodiment of the present application.

[0358] It is to be understood that the structural description of the electronic device 2300 in the embodiment of the present application does not constitute a specific limitation on the electronic device 2300. In other embodiments of the present application, the electronic device 2300 may include other components besides the processor 2301 and the memory 2302.

[0359] The processor 2301 may be a device on a chip (SOC), and the processor 2301 may include a central processing unit (CPU), and may further include other types of processors.

[0360] The processor involved in the processor 2301 may include, for example, a CPU, a DSP, a microcontroller or a digital signal processor, and may also include a GPU, an embedded neural network processor (Neural-network Process Units, NPU) and an image signal processor (Image Signal Processing, ISP). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASICs, or one or more integrated circuits for controlling the execution of the program of the technical solution of the present application. In addition, the processor may have the function of operating one or more software programs, and the software programs may be stored in a storage medium.

[0361] The processor 2301 may include one or more processing units. For example, the processor may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent components or integrated into one or more processors. In some embodiments, the electronic device 2300 may also include one or more processors 2301. Among them, the controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

[0362] In some embodiments, the processor 2301 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface, etc. Among them, the USB interface is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface can be used to connect a charger to charge an electronic device, and can also be used to transmit data between an electronic device and a peripheral device.

[0363] The electronic device 2300 may also include an external memory interface, which is used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 2301 through the external memory interface to implement a data storage function. For example, files such as music and videos are stored in the external memory card.

[0364] The memory 2302 may include a code storage area and a data storage area. The code storage area may store an operating system. The data storage area may store data created during the use of the electronic device 2300, etc. In addition, the memory 2302 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage components, flash memory components, universal flash storage (UFS), etc.

[0365] The memory 2302 may be a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or may be any computer-readable medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer.

[0366] The processor 2301 and the memory 2302 may be integrated into one processing device, or more commonly, are independent components.

[0367] In one embodiment, the electronic device proposed in the present application may be the electronic device described in the above embodiment. Specifically, the electronic device is used to load a parallel computing framework, and the parallel computing framework loaded by the electronic device is used to execute the method steps executed by the parallel computing framework in the method described in the embodiment of the present application. Specifically, when the computer program instructions in the memory 2302 are executed by the processor 2301, the electronic device 2300 is triggered to execute the method steps executed by the electronic device in the method described in the embodiment of the present application.

[0368] In another embodiment, the electronic device proposed in the present application may be the heterogeneous computing device described in the above embodiment. Specifically, the electronic device is used to create a virtual device, and the virtual device created in the electronic device is used to execute the method steps executed by the virtual device in the method described in the embodiment of the present application. Specifically, when the computer program instructions in the memory 2302 are executed by the processor 2301, the electronic device 2300 is triggered to execute the method steps executed by the heterogeneous computing device in the method described in the embodiment of the present application.

[0369] The present application also proposes a server cluster, which includes a memory for storing computer program instructions and a processor for executing computer program instructions, wherein when the computer program instructions are executed by the processor, the server cluster is triggered to execute the method steps as described in the embodiments of the present application. Specifically, in one embodiment, the server cluster includes the electronic device and heterogeneous computing device described in the embodiments of the present application.

[0370] Furthermore, the devices, apparatuses, and modules described in the embodiments of the present application may be implemented by computer chips or entities, or by products having certain functions.

[0371] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0372] In the several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0373] Specifically, in one embodiment of the present application, a computer-readable storage medium is further provided, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided in the embodiment of the present application.

[0374] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is run on a computer, the computer executes the method provided by the embodiment of the present application.

[0375] The embodiments described in this application are described with reference to the flowcharts and / or block diagrams of the methods, devices (apparatus), and computer program products according to the embodiments of the application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0376] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0377] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0378] It should also be noted that in the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0379] In the embodiments of the present application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, commodity or device including the element.

[0380] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0381] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0382] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments of the present application can be implemented by a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0383] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, apparatuses and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0384] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A model training method, characterized in that: The method is applied to an electronic device and a heterogeneous computing device loaded with a parallel computing framework, and the method includes: The parallel computing framework divides the model training task into multiple computing tasks; The parallel computing framework distributes the multiple computing tasks to multiple virtual devices, wherein the multiple virtual devices correspond to the multiple computing tasks one by one, and the multiple virtual devices are virtual devices created based on the heterogeneous computing devices; The multiple virtual devices execute the distributed computing tasks to perform iterative calculations for model training.

2. The method according to claim 1, characterized in that: A communication virtualization device is deployed on the electronic device, and the communication virtualization device presents the multiple virtual devices to the parallel computing framework in the form of multiple heterogeneous computing devices, wherein the communication virtualization device is deployed outside the parallel computing framework.

3. The method according to claim 2, characterized in that: Implementing communication between the parallel computing framework and the plurality of virtual devices, and between the plurality of virtual devices based on collective communication; The communication virtualization device is also used to: configure collective communication attribute resources for the virtual device, present the collective communication attributes of the virtual device to the parallel computing framework; and provide virtualization support for the collective communication physical resources.

4. The method according to claim 3, characterized in that A routing device is deployed outside the parallel computing framework, wherein the routing device is used to generate a collective communication operator for collective communication according to the virtual device topology relationship of the multiple virtual devices.

5. The method according to claim 4, characterized in that The routing device is used to store a virtual device routing table, and the virtual device routing table is used to record all virtual devices and topological connection relationships that need to copy data for the current computing task.

6. The method according to claim 4 or 5, characterized in that: The routing device is deployed on the electronic device, and the routing device includes instance objects corresponding to a plurality of the virtual devices in a one-to-one manner.

7. The method according to claim 4 or 5, characterized in that: The routing device is deployed in each of the virtual devices.

8. The method according to any one of claims 4 to 7, characterized in that: The multiple computing tasks include at least a first computing task and a second computing task; The parallel computing framework distributes the multiple computing tasks to multiple virtual devices, including distributing the first computing task to a first virtual device and distributing the second computing task to a second virtual device; The virtual device executes the distributed computing task to perform iterative computing of model training, including: The first virtual device performs the first computing task, and the second virtual device performs the second computing task; In the case of triggering task migration from the first virtual device to the third virtual device, terminating the first virtual device from executing the first computing task, and maintaining the second virtual device to continue executing the second computing task; Executing the task migration includes synchronizing the first computing task and the current execution state of the first computing task in the first virtual device to the third virtual device; The third virtual device continues to execute the first computing task.

9. The method according to claim 8, characterized in that The virtual device executes the distributed computing task to perform iterative computing of model training, and further includes: After the task migration, the virtual device topology relationship in the routing device is updated according to the task migration.

10. The method according to claim 8, characterized in that The virtual device executes the distributed computing tasks to perform iterative computing of model training, and also includes: After the task migration, no information indicating a change in the topological relationship of the virtual devices is fed back to the parallel computing framework.

11. The method according to claim 9, characterized in that The multiple computing tasks further include at least a third computing task, and execution of the third computing task requires reference to a computing result of the first computing task; The parallel computing framework distributes the plurality of computing tasks to a plurality of virtual devices, further comprising distributing the third computing task to a fourth virtual device; The multiple virtual devices execute the distributed computing tasks to perform iterative computing of model training, including: the fourth virtual device subscribes to the task execution status of the first virtual device according to the third computing task; After the first virtual device completes executing the first computing task, the fourth virtual device obtains the computing result of the first computing task, and executes the third computing task according to the computing result of the first computing task.

12. The method according to claim 11, characterized in that The virtual device executes the distributed computing task to perform iterative computing of model training, and further includes: After the task migration, the fourth virtual device is updated to subscribe to the task execution state of the third virtual device.

13. The method according to claim 12, characterized in that: Executing the task migration further includes synchronizing the virtual device topology relationship in the routing device of the first virtual device to the routing device of the third virtual device; After the task migration, updating the virtual device topology relationship in the routing device according to the task migration includes: The virtual device topology relationship in the routing device of the third virtual device is updated, and the virtual device topology relationship in the routing device of the fourth virtual device is updated.

14. A model training device, characterized in that: The apparatus comprises a parallel computing framework and a plurality of virtual devices, wherein: The multiple virtual devices are virtual devices created based on heterogeneous computing devices; The parallel computing framework is loaded on the electronic device, and is used to divide the model training task into multiple computing tasks, and distribute the multiple computing tasks to the multiple virtual devices, wherein the multiple virtual devices correspond to the multiple computing tasks one by one; The multiple virtual devices are used to execute the distributed computing tasks to perform iterative calculations for model training.

15. The device according to claim 14, characterized in that The device also includes a communication virtualization device, wherein: The communication virtualization device is deployed on the electronic device, wherein the communication virtualization device is deployed outside the parallel computing framework; The communication virtualization device is used to present the multiple virtual devices to the parallel computing framework in the form of multiple heterogeneous computing devices.

16. The device according to claim 15, characterized in that The communication between the parallel computing framework and the virtual device, and between different virtual devices, is collective communication; The communication virtualization device comprises: A collective communication resource virtualization module, which is used to configure collective communication attribute resources for the virtual device and present the collective communication attributes of the virtual device to the parallel computing framework; The virtual device super-resolution module is used to support virtualization of collective communication physical resources.

17. The device according to claim 16, characterized in that The device also includes a routing device, wherein: The routing device is deployed outside the parallel computing framework, and is used to generate a collective communication operator for collective communication according to the virtual device topology relationship of the multiple virtual devices.

18. The device according to claim 17, characterized in that The routing device is used to store a virtual device routing table, and the virtual device routing table is used to record all virtual devices and topological connection relationships that need to copy data for the current computing task.

19. The device according to claim 17 or 18, characterized in that The routing device is deployed on the electronic device, and the routing device includes instance objects corresponding to a plurality of the virtual devices in a one-to-one manner.

20. The device according to claim 17 or 18, characterized in that The routing device is deployed in each of the virtual devices.

21. The device according to any one of claims 17 to 20, characterized in that: The multiple computing tasks include at least a first computing task and a second computing task; the multiple virtual devices include at least a first virtual device and a second virtual device; the parallel computing framework distributes the first computing task to the first virtual device, and distributes the second computing task to the second virtual device; The first virtual device performs the first computing task, and the second virtual device performs the second computing task; The apparatus further comprises a task migration module and a third virtual device, wherein: The task migration module is used to: in the case of triggering task migration from the first virtual device to the third virtual device, terminate the first virtual device from executing the first computing task, and maintain the second virtual device to continue executing the second computing task; perform the task migration, including synchronizing the first computing task in the first virtual device and the current execution state of the first computing task to the third virtual device; The third virtual device is used to continue executing the first computing task after the task migration.

22. The device according to claim 21, characterized in that The routing device is further configured to update the virtual device topology relationship in the routing device according to the task migration after the task migration.

23. The device according to claim 21, characterized in that The routing device is also used for not feeding back information indicating changes in the topological relationship of the virtual devices to the parallel computing framework after the task migration.

24. The device according to claim 22, characterized in that The plurality of computing tasks further include at least a third computing task, and the execution of the third computing task needs to refer to the computing result of the first computing task; the plurality of virtual devices further include at least a fourth virtual device; The parallel computing framework is used to distribute the third computing task to the fourth virtual device; The fourth virtual device is used to: subscribe to the task execution status of the first virtual device according to the third computing task; after the first virtual device completes the execution of the first computing task, the fourth virtual device obtains the calculation result of the first computing task; and execute the third computing task according to the calculation result of the first computing task.

25. The device according to claim 24, characterized in that The fourth virtual device is further used for: After the task migration, the task execution state of subscribing to the first virtual device is updated to the task execution state of subscribing to the third virtual device.

26. The device according to claim 25, characterized in that: The task migration module is used to synchronize the virtual device topology relationship in the routing device of the first virtual device to the routing device of the third virtual device; The routing device of the third virtual device is used to update the virtual device topology relationship in the routing device of the third virtual device according to the task migration after the task migration; The routing device of the fourth virtual device is used for updating the virtual device topology relationship in the routing device of the fourth virtual device according to the task migration after the task migration.

27. An electronic device, characterized in that: The electronic device comprises a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method steps as described in any one of claims 1-13.

28. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 13.

29. A server cluster, characterized in that: The server cluster comprises a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the server cluster is triggered to execute the method steps as described in any one of claims 1-13.

Images