Model training method, model training apparatus, and electronic apparatus

Abstract

Provided in the embodiments of the present application are a model training method, a model training apparatus, and an electronic apparatus. The method is applied to an electronic device on which a parallel computing framework is loaded, and heterogeneous computing devices. The method comprises: using a parallel computing framework to divide a model training task into a plurality of computing tasks; using the parallel computing framework to distribute the plurality of computing tasks to a plurality of virtual devices, wherein the plurality of virtual devices are virtual devices that are created on the basis of heterogeneous computing devices; and the plurality of virtual devices executing the distributed computing tasks, so as to perform iterative computing during model training. According to the method in the embodiments of the present application, parallel execution logic for the computing tasks is placed in the virtual devices, which are capable of multiplexing computing resources, and binding relationships between the computing tasks, communication tasks and hardware are decoupled; thus, the effects of a plurality of processes being in parallel and communication and computing being in parallel can be achieved.

Description

Model training method, model training device and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on October 25, 2023, with application number 2023113928192 and application name “A model training method, model training device and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a model training method, a model training device, and an electronic device. Background Art

[0003] Deep learning is an artificial intelligence (AI) method used to teach computers to process data in ways inspired by the human brain. Deep learning models can identify complex patterns in images, text, sound, and other data, generating accurate insights and predictions. Users can use deep learning methods to automate tasks that typically require AI, such as describing an image or transcribing a sound file into text.

[0004] In deep learning applications, model training requires enormous computing power. However, existing communication solutions result in high communication overhead during model training, leading to significant waste of computing resources and low resource utilization for deep learning model training.

[0005] Therefore, a model training method for deep learning model training is needed to reduce the communication overhead during deep learning model training, reduce idle computing resources, and improve resource utilization of deep learning model training.

[0006] Summary of the Invention

[0007] In order to solve the problem of how to reduce communication overhead in the process of deep learning model training, reduce idle computing resources, and improve resource utilization of deep learning model training, the present application provides a model training method, a model training device and an electronic device. The present application also provides a computer-readable storage medium and a server cluster.

[0008] The embodiments of this application adopt the following technical solutions:

[0009] In a first aspect, the present application provides a model training method, which is applied to electronic devices and heterogeneous computing devices loaded with a parallel computing framework, and the method includes:

[0010] The parallel computing framework divides the model training task into multiple computing tasks;

[0011] The parallel computing framework distributes the plurality of computing tasks to a plurality of virtual devices, wherein the plurality of virtual devices correspond one-to-one to the plurality of computing tasks, and the plurality of virtual devices are virtual devices created based on the heterogeneous computing devices;

[0012] The multiple virtual devices execute the distributed computing tasks to perform iterative calculations for model training.

[0013] According to the method of the first aspect, the parallel execution logic of computing tasks is placed in a virtual device of reusable computing resources, and the binding relationship between computing tasks, communication tasks and hardware (heterogeneous computing devices) is decoupled, achieving the effect of multi-process parallelism, communication and computing parallelism on a macro level. In theory, the time required to complete communication is completely hidden.

[0014] Furthermore, because the parallel execution logic for computing tasks is placed in virtual devices that reuse computing resources, during iterative computations for model training, computing and communication tasks do not rely on the hardware topology of heterogeneous computing devices, but rather on the topology of virtual devices within the computing task execution device set. Therefore, maximizing the compute-to-communication ratio eliminates the need for detailed understanding of the hardware topology of heterogeneous computing devices, reducing the difficulty of achieving this goal.

[0015] Furthermore, according to the method of the first aspect, a parallel computing framework for distributing tasks to multiple heterogeneous computing devices is used to implement task distribution to multiple virtual devices. Because the virtual devices appear as heterogeneous computing devices to the parallel computing framework, there is no need to adjust the parallel computing framework for distributing tasks to multiple heterogeneous computing devices, thereby reducing the difficulty of implementing task distribution to virtual devices.

[0016] In an implementation of the first aspect, the method further includes:

[0017] A communication virtualization device is deployed on the electronic device, and the communication virtualization device is used to present the multiple virtual devices to the parallel computing framework in the form of the multiple heterogeneous computing devices, wherein the communication virtualization device is deployed outside the parallel computing framework.

[0018] According to the above implementation method, a communication virtualization device is deployed outside the parallel computing framework, and the communication between the parallel computing framework and the virtual device is realized through the communication virtualization device. The parallel computing framework does not need to perceive the deployment and implementation status of the virtual device on the heterogeneous computing device, which reduces the difficulty of implementing communication between the parallel computing framework and the virtual device.

[0019] In an implementation of the first aspect:

[0020] Implementing communication between the parallel computing framework and the plurality of virtual devices, and between the plurality of virtual devices based on collective communication;

[0021] The communication virtualization device is further used to: configure collective communication attribute resources for the virtual device, present the collective communication attributes of the virtual device to the parallel computing framework; and provide virtualization support for the collective communication physical resources.

[0022] In an implementation of the first aspect, the method further includes:

[0023] A routing device is deployed outside the parallel computing framework, and the routing device is used to generate a collective communication operator for collective communication according to the virtual device topology relationship of the multiple virtual devices.

[0024] According to the above implementation method, the parallel computing framework is not responsible for managing device topology and inter-device communication, and the collective communication capability is transferred to the virtual device. During the iterative computing process, the implementation of collective communication does not require the participation of the parallel computing framework, which improves the flexibility and efficiency of collective communication.

[0025] In an implementation of the first aspect, the routing device is used to save a virtual device routing table, which is used to record all virtual devices and topological connection relationships that require data copying for the current computing task.

[0026] In an implementation manner of the first aspect, the routing device is deployed on the electronic device, and the routing device includes instance objects corresponding one-to-one to the multiple virtual devices.

[0027] In an implementation manner of the first aspect, the routing device is deployed in each of the virtual devices.

[0028] In an implementation of the first aspect, the multiple computing tasks include at least a first computing task and a second computing task; the parallel computing framework distributes the multiple computing tasks to multiple virtual devices, including distributing the first computing task to a first virtual device and distributing the second computing task to a second virtual device;

[0029] The virtual device executes the distributed computing tasks to perform iterative calculations for model training, including:

[0030] The first virtual device performs the first computing task, and the second virtual device performs the second computing task;

[0031] In the case of triggering task migration from the first virtual device to the third virtual device, terminating the first virtual device from executing the first computing task, and maintaining the second virtual device to continue executing the second computing task;

[0032] Executing the task migration includes synchronizing the first computing task and the current execution state of the first computing task in the first virtual device to the third virtual device;

[0033] The third virtual device continues to execute the first computing task.

[0034] In an implementation of the first aspect, the virtual device executes the distributed computing tasks to perform iterative computing of the model training, further comprising:

[0035] After the task migration, the virtual device topology relationship in the routing apparatus is updated according to the task migration.

[0036] In an implementation of the first aspect, the virtual device executes the distributed computing tasks to perform iterative computing of the model training, further comprising:

[0037] After the task migration, no information indicating a change in the topological relationship of the virtual devices is fed back to the parallel computing framework.

[0038] According to the above implementation method, the parallel computing framework is not responsible for managing device topology and inter-device communication. It sinks the collective communication capabilities to virtual devices, supports dynamic migration of computing tasks during iterative computing, and dynamically updates virtual device link information. There is no need for the computing framework layer to perceive and recompile to generate communication operators, which reduces the waste of computing power waiting.

[0039] In an implementation of the first aspect, the multiple computing tasks further include at least a third computing task, and execution of the third computing task requires reference to a computing result of the first computing task;

[0040] The parallel computing framework distributes the plurality of computing tasks to a plurality of virtual devices, further comprising distributing the third computing task to a fourth virtual device;

[0041] The multiple virtual devices execute the distributed computing tasks to perform iterative calculations for model training, including: the fourth virtual device subscribes to the task execution status of the first virtual device according to the third computing task; after the first virtual device completes the execution of the first computing task, the fourth virtual device obtains the calculation result of the first computing task, and executes the third computing task according to the calculation result of the first computing task.

[0042] According to the above implementation, the virtual device perceives the real-time status of each computing task, and a communication task can be initiated in real time after a single computing task is completed. Compared to a parallel computing framework that uniformly manages the status of all computing tasks, after a single computing task is completed, it is necessary to wait for the computing framework to issue a communication task to implement communication. The method according to the embodiment of the present application can reduce computing power consumption.

[0043] In an implementation of the first aspect, the virtual device executes the distributed computing tasks to perform iterative computing of the model training, further comprising:

[0044] After the task migration, the fourth virtual device is updated to subscribe to the task execution state of the third virtual device.

[0045] In an implementation of the first aspect:

[0046] Executing the task migration further includes synchronizing the virtual device topology relationship in the routing device of the first virtual device to the routing device of the third virtual device;

[0047] After the task migration, updating the virtual device topology relationship in the routing device according to the task migration includes:

[0048] The virtual device topology relationship in the routing device of the third virtual device is updated, and the virtual device topology relationship in the routing device of the fourth virtual device is updated.

[0049] In a second aspect, the present application provides a model training device, which includes a parallel computing framework and multiple virtual devices, wherein:

[0050] The multiple virtual devices are virtual devices created based on heterogeneous computing devices;

[0051] The parallel computing framework is loaded on the electronic device, and is used to divide the model training task into multiple computing tasks and distribute the multiple computing tasks to the multiple virtual devices, wherein the multiple computing tasks correspond to the multiple virtual devices one by one;

[0052] The multiple virtual devices are used to execute the distributed computing tasks to perform iterative calculations for model training.

[0053] In an implementation of the second aspect, the apparatus further includes a communication virtualization apparatus, wherein:

[0054] The communication virtualization device is deployed on the electronic device, wherein the communication virtualization device is deployed outside the parallel computing framework;

[0055] The communication virtualization device is used to present the multiple virtual devices to the parallel computing framework in the form of multiple heterogeneous computing devices.

[0056] In an implementation of the second aspect, the communication between the parallel computing framework and the virtual device, and between different virtual devices, is collective communication;

[0057] The communication virtualization device includes:

[0058] a collective communication resource virtualization module, configured to configure collective communication attribute resources for the virtual device and present the collective communication attributes of the virtual device to the parallel computing framework;

[0059] The virtual device super-resolution module is used to support virtualization of collective communication physical resources.

[0060] In an implementation of the second aspect, the apparatus further includes a routing device, wherein:

[0061] The routing device is deployed outside the parallel computing framework and is used to generate a collective communication operator for collective communication according to the virtual device topology relationship of the multiple virtual devices.

[0062] In an implementation of the second aspect, the routing device is used to save a virtual device routing table, which is used to record all virtual devices and topological connection relationships that require data copying for the current computing task.

[0063] In an implementation manner of the second aspect, the routing device is deployed on the electronic device, and the routing device includes instance objects corresponding one-to-one to the multiple virtual devices.

[0064] In an implementation manner of the second aspect, the routing device is deployed in each of the virtual devices.

[0065] In an implementation of the second aspect, the multiple computing tasks include at least a first computing task and a second computing task; the multiple virtual devices include at least a first virtual device and a second virtual device; the parallel computing framework distributes the first computing task to the first virtual device and distributes the second computing task to the second virtual device; the first virtual device executes the first computing task, and the second virtual device executes the second computing task;

[0066] The apparatus further includes a task migration module and a third virtual device, wherein:

[0067] The task migration module is configured to: in the case of triggering task migration from the first virtual device to the third virtual device, terminate the first virtual device from executing the first computing task, and maintain the second virtual device from continuing to execute the second computing task; performing the task migration includes synchronizing the first computing task in the first virtual device and the current execution state of the first computing task to the third virtual device;

[0068] The third virtual device is used to continue executing the first computing task after the task migration.

[0069] In an implementation manner of the second aspect, the routing apparatus is further configured to, after the task migration, update a topological relationship of virtual devices in the routing apparatus according to the task migration.

[0070] In an implementation of the second aspect, the routing device is further configured to, after the task migration, not feed back information indicating a change in a topological relationship of virtual devices to the parallel computing framework.

[0071] In an implementation of the second aspect, the multiple computing tasks further include at least a third computing task, and execution of the third computing task requires reference to a computing result of the first computing task; the multiple virtual devices further include at least a fourth virtual device;

[0072] The parallel computing framework is used to distribute the third computing task to the fourth virtual device;

[0073] The fourth virtual device is used to: subscribe to the task execution status of the first virtual device according to the third computing task; after the first virtual device completes the execution of the first computing task, the fourth virtual device obtains the calculation result of the first computing task; and execute the third computing task according to the calculation result of the first computing task.

[0074] In an implementation of the second aspect, the fourth virtual device is further configured to:

[0075] After the task migration, the task execution status of the first virtual device is updated to the task execution status of the third virtual device.

[0076] In an implementation of the second aspect:

[0077] The task migration module is used to synchronize the virtual device topology relationship in the routing device of the first virtual device to the routing device of the third virtual device;

[0078] The routing device of the third virtual device is used to update the virtual device topology relationship in the routing device of the third virtual device according to the task migration after the task migration;

[0079] The routing device of the fourth virtual device is used to update the virtual device topology relationship in the routing device of the fourth virtual device according to the task migration after the task migration.

[0080] In a third aspect, the present application provides an electronic device comprising a memory for storing computer program instructions and a processor for executing computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method steps described in the first aspect.

[0081] In an implementation of the third aspect, the electronic device is the electronic device described in the first aspect, and the electronic device is used to load a parallel computing framework. The parallel computing framework loaded by the electronic device is used to execute the method steps executed by the parallel computing framework in the method described in the embodiment of the present application.

[0082] Specifically, when the computer program instructions in the memory of the electronic device are executed by the processor of the electronic device, the electronic device is triggered to execute the method steps executed by the electronic device in the first aspect.

[0083] In an implementation of the third aspect, the electronic device is the heterogeneous computing device described in the first aspect. The electronic device is used to create a virtual device, and the virtual device created in the electronic device is used to execute the method steps executed by the virtual device in the first aspect.

[0084] Specifically, when the computer program instructions in the memory of the electronic device are executed by the processor of the electronic device, the electronic device is triggered to execute the method steps executed by the heterogeneous computing device in the first aspect.

[0085] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method described in the first aspect.

[0086] In a fifth aspect, the present application provides a server cluster, wherein the electronic device includes a memory for storing computer program instructions and a processor for executing computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method steps described in the first aspect.

[0087] In an implementation of the fifth aspect, the server cluster includes the electronic device and heterogeneous computing devices described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] FIG1 is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application;

[0089] FIG2 is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application;

[0090] FIG3 is a schematic diagram of a model training process according to an embodiment of the present application;

[0091] FIG4 is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application;

[0092] FIG5 is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application;

[0093] FIG6 is a schematic diagram of a model training process according to an embodiment of the present application;

[0094] FIG7 is a schematic diagram showing a partial process of model training according to an embodiment of the present application;

[0095] FIG8 is a schematic diagram showing a virtual device structure according to an embodiment of the present application;

[0096] FIG9 is a schematic diagram showing a partial process of model training according to an embodiment of the present application;

[0097] FIG10 is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application;

[0098] FIG11 is a schematic diagram of a virtual device according to an embodiment of the present application;

[0099] FIG12 is a schematic diagram showing a computing task migration process according to an embodiment of the present application;

[0100] FIG13 is a schematic diagram showing a computing task migration process according to an embodiment of the present application;

[0101] FIG14 is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application;

[0102] FIG15 is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application;

[0103] FIG16 is a schematic structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0104] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0105] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0106] For deep learning model training, a feasible training solution is to split the data and model according to a parallel strategy, and then distribute the computing tasks to a heterogeneous cluster with multiple machines and multiple graphics cards for iterative training.

[0107] FIG1 is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application.

[0108] The hardware device for implementing model training includes an electronic device 100 and multiple heterogeneous computing devices, and the multiple heterogeneous computing devices include at least a heterogeneous computing device 110 and a heterogeneous computing device 120. In another embodiment, the multiple heterogeneous computing devices may include more than two heterogeneous computing devices.

[0109] In the embodiment of the present application, the electronic device is a device for managing the model training process. The embodiment of the present application does not specifically limit the type of electronic device. For example, the electronic device can be a local device (e.g., a personal desktop computer) or a server device (e.g., a local server or a cloud server).

[0110] A heterogeneous computing device is a physical device used to perform computing tasks, which is used to receive computing tasks, calculate the computing tasks, and output the computing results. The embodiments of the present application do not specifically limit the type of heterogeneous computing devices. For example, the heterogeneous computing device can be a computing card, such as a graphics processing unit (GPU) card or a neural network processing unit (NPU); the heterogeneous computing device can be other types of computing devices, such as an embedded microcomputer or a microcomputing terminal.

[0111] Exemplarily, a heterogeneous computing device can be an electronic device independent of an electronic device. For example, a heterogeneous computing device is a computing card installed on a terminal device, and the terminal device interacts with the electronic device via a data network. A heterogeneous computing device can also be an electronic device installed on an electronic device. For example, the electronic device is a host computer, and the heterogeneous computing device is a computing card installed on the host computer. Heterogeneous computing devices include computing units, which are used to perform computing tasks. For example, as shown in Figure 1, heterogeneous computing device 110 includes computing unit 111, and heterogeneous computing device 120 includes computing unit 121.

[0112] Communication between electronic devices and multiple heterogeneous computing devices, or between multiple heterogeneous computing devices, can be implemented based on protocols such as PCIe links or NVLINK. In some embodiments, it can also be implemented through other links or protocols, which are not specifically limited in the embodiments of the present application.

[0113] As shown in Figure 1, before starting model training, the electronic device 100 obtains a training script, a sample data set, and a collective communication topology table (RankTable). The training script contains the original model to be trained; the RankTable file is used to describe the topology information of the cluster devices participating in collective communication, including server (Server), device (Device), network information, etc.

[0114] The electronic device 100 is loaded with a parallel computing framework 101 , and the training script, sample data set, and collective communication topology table are input into the parallel computing framework 101 , which starts the computing task (model training task).

[0115] The parallel computing framework 101 includes a task scheduler that divides the computing tasks of the model training according to a specified parallel strategy (specified by the user or based on a default configuration). In another embodiment, the computing tasks can also be divided according to a specified parallel strategy based on a third-party task scheduler independent of the electronic device 100.

[0116] The split computing tasks are distributed to different heterogeneous computing devices. The heterogeneous computing devices use the computing power unit 111 to execute the computing tasks to perform one iterative calculation. The iterative calculations are repeated multiple times until the computing task (model training task) is completed.

[0117] One iterative calculation includes forward propagation and backward update.

[0118] Specifically, forward propagation is the process of calculating the output of each neuron in sequence from the input layer to the output layer in a neural network according to the corresponding rules, and finally obtaining the loss function under the current parameters; backward update is the process of calculating the gradient of the loss function for each parameter from the output layer to the input layer of the neural network on the basis of the forward propagation algorithm, and updating the relevant parameters at a given learning rate.

[0119] In one iterative calculation, the heterogeneous computing device executes the following process.

[0120] Forward propagation:

[0121] a) Heterogeneous computing devices perform forward calculations on local error values; (Calculation process)

[0122] b) The local error values ​​generated by all heterogeneous computing devices are integrated according to certain rules to generate a global error value. (Communication process)

[0123] Backward update:

[0124] c) The model calculates the error information of each layer from back to front; (calculation process)

[0125] d) After synchronizing all error information, update the model parameters from front to back. (Communication process)

[0126] In the above model training process, since the computing tasks are divided into multiple heterogeneous computing devices, a single computing iteration introduces communication processes, such as process b and process d.

[0127] Collective communication enables data communication between multiple heterogeneous computing devices. Collective communication was first proposed by the message passing interface (MPI), which defines a standard message passing interface to address global communication within a process group composed of different processes in parallel computing.

[0128] Specifically, the heterogeneous computing devices are configured with collective communication implementation devices. For example, heterogeneous computing device 110 includes collective communication implementation device 112, and heterogeneous computing device 120 includes collective communication implementation device 122. Parallel computing framework 101 generates collective communication operators for collective communication based on the device topology relationships of the heterogeneous computing devices. The collective communication implementation devices implement collective communication between different heterogeneous computing devices and between heterogeneous computing devices and electronic devices based on the collective communication operators.

[0129] During model training, the computational and communication processes executed by heterogeneous computing devices are determined by the computational tasks they receive and the network topology between the heterogeneous computing devices. That is, after the computational tasks are distributed, the computational and communication processes are bound to the heterogeneous computing devices; and, based on the network topology, there is a sequence in the execution of the computational and communication processes (the execution of a certain computational or communication process may need to wait for the execution results of other computational or communication processes). This prevents the computational and communication processes from being effectively pipelined and parallelized. When the computational and communication processes cannot be effectively pipelined and parallelized, computing resources will be wasted during training, resulting in low effective computing power for the entire model training. For example, according to actual measurements during Pengcheng Pangu training, in operator-level model parallel scenarios, the communication overhead that cannot be concealed is as high as 40%.

[0130] To address the above issues, an embodiment of the present application provides a model training method. During the model training process, multiple virtual devices are constructed based on heterogeneous computing devices, and the multiple virtual devices are used to parallelly process the divided computing tasks to achieve parallel model training.

[0131] FIG2 is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application.

[0132] As shown in Figure 2, the hardware device for implementing model training includes an electronic device 500 and multiple virtual devices created based on heterogeneous computing devices. As shown in Figure 2, the multiple virtual devices include at least virtual device 511 and virtual device 512. In another embodiment, the multiple virtual devices may include more than two virtual devices.

[0133] A parallel computing framework 501 is loaded on the electronic device 500 , and the parallel computing framework 501 includes a task scheduler 502 .

[0134] The parallel computing framework 501 is used to divide the model training task into multiple computing tasks, and distribute the multiple computing tasks obtained by dividing to multiple virtual devices (for example, to virtual device 511 and virtual device 512) to use multiple virtual devices to process the model training task in parallel.

[0135] In the embodiment shown in FIG. 2 , each of the multiple virtual devices appears as a heterogeneous computing device at the parallel computing framework 501 , that is, the multiple virtual devices appear as multiple heterogeneous computing devices at the parallel computing framework 501 .

[0136] It should be noted here that multiple virtual devices appear as multiple heterogeneous computing devices in the parallel computing framework 501. This means that from the perspective of the parallel computing framework 501, the parallel computing framework 501 perceives multiple virtual devices as multiple heterogeneous computing devices, not that there are multiple heterogeneous computing devices in the actual sense.

[0137] Because multiple virtual devices appear as multiple heterogeneous computing devices in parallel computing framework 501, when parallel computing framework 501 distributes the multiple computing tasks obtained by partitioning to multiple virtual devices, the function performed by parallel computing framework 501 can be considered as distributing the multiple computing tasks obtained by partitioning to multiple heterogeneous computing devices. Therefore, parallel computing framework 501 can be implemented by referring to parallel computing framework 101 shown in FIG. 1 .

[0138] FIG3 is a schematic diagram of a model training process according to an embodiment of the present application.

[0139] In one embodiment, the following process shown in FIG3 is executed to implement parallel model training.

[0140] S300, the parallel computing framework 501 divides the model training task into multiple computing tasks, for example, divides the model training task into a first computing task and a second computing task.

[0141] S310, the parallel computing framework 501 distributes the multiple computing tasks obtained by segmentation to multiple virtual devices, wherein the multiple virtual devices correspond one-to-one to the multiple computing tasks, that is, each of the multiple computing tasks is distributed to a virtual device, and different computing tasks are distributed to different virtual devices.

[0142] For example, the first computing task is distributed to the virtual device 511 , and the second computing task is distributed to the virtual device 512 .

[0143] S320: Multiple virtual devices execute the distributed computing tasks to perform iterative calculations for model training.

[0144] For example, virtual device 511 performs a first computing task, and virtual device 512 performs a second computing task.

[0145] According to the method of the embodiment of the present application, since the parallel execution logic of computing tasks is placed in a virtual device that can reuse computing resources, the binding relationship between computing tasks, communication tasks and hardware (heterogeneous computing devices) is decoupled, achieving the effect of multi-process parallelism, communication and computing parallelism at a macro level, and theoretically completely hiding the time required to complete communication.

[0146] Furthermore, because the parallel execution logic for computing tasks is placed in virtual devices that reuse computing resources, during iterative computations for model training, computing and communication tasks do not rely on the hardware topology of heterogeneous computing devices, but rather on the topology of virtual devices within the computing task execution device set. Therefore, maximizing the compute-to-communication ratio eliminates the need for detailed understanding of the hardware topology of heterogeneous computing devices, reducing the difficulty of achieving this goal.

[0147] Furthermore, according to the method of the embodiment of the present application, a parallel computing framework for distributing tasks to multiple heterogeneous computing devices is used to implement task distribution to multiple virtual devices. Because the virtual devices appear as heterogeneous computing devices to the parallel computing framework, there is no need to adjust the parallel computing framework for distributing tasks to multiple heterogeneous computing devices, thereby reducing the difficulty of implementing task distribution to virtual devices.

[0148] In the embodiment shown in FIG2 , the virtual device is a virtual device created based on a heterogeneous computing device, wherein the multiple virtual devices may be virtual devices created based on a single heterogeneous computing device or virtual devices created based on multiple heterogeneous computing devices.

[0149] For example, Figure 4 shows a schematic diagram of the hardware device architecture for implementing model training according to an embodiment of the present application.

[0150] As shown in FIG4 , the hardware device for implementing model training includes an electronic device 600 and a heterogeneous computing device 610 .

[0151] The electronic device 600 may refer to the electronic device 100 , and the heterogeneous computing device 610 may refer to the heterogeneous computing device 110 or the heterogeneous computing device 120 .

[0152] A virtual device 611 and a virtual device 612 are created based on the heterogeneous computing device 610 .

[0153] The hardware resources of the heterogeneous computing device 610 include shared memory 613 , communication links 614 , and computing power units 615 .

[0154] The shared memory 613 may be all memory resources of the heterogeneous computing device 610 or part of the memory resources of the heterogeneous computing device 610. The virtual devices 611 and 612 may synchronize data based on the shared memory 613.

[0155] The communication link 614 is a data transmission link in the heterogeneous computing device 610 that can be provided to the virtual device for use, for example, a PCIe link or a network device link.

[0156] The computing unit 615 is a computing resource of the heterogeneous computing device 610 . The virtual device 611 and the virtual device 612 can share the computing unit 615 , for example, by time-sharing multiplexing the computing unit 615 .

[0157] For another example, FIG5 shows a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application.

[0158] As shown in FIG5 , the hardware devices for implementing model training include an electronic device 700 , a heterogeneous computing device 710 , and a heterogeneous computing device 720 .

[0159] The electronic device 700 may refer to the electronic device 100 , and the heterogeneous computing device 710 and the heterogeneous computing device 720 may refer to the heterogeneous computing device 110 or the heterogeneous computing device 120 .

[0160] Heterogeneous computing device 710 and heterogeneous computing device 720 may be heterogeneous computing devices of the same electronic device. For example, heterogeneous computing device 710 and heterogeneous computing device 720 are both installed on electronic device 700. For another example, heterogeneous computing device 710 and heterogeneous computing device 720 are both installed on another electronic device other than electronic device 700, and the electronic device on which heterogeneous computing device 710 and heterogeneous computing device 720 are installed can communicate and interact with electronic device 700.

[0161] Heterogeneous computing device 710 and heterogeneous computing device 720 may be heterogeneous computing devices of different electronic devices. For example, heterogeneous computing device 710 is installed on electronic device 700, and heterogeneous computing device 720 is installed on another electronic device other than electronic device 700. The electronic device on which heterogeneous computing device 720 is installed can communicate and interact with electronic device 700. For another example, heterogeneous computing device 710 and heterogeneous computing device 720 are installed on two other electronic devices other than electronic device 700, and the other two electronic devices on which heterogeneous computing device 710 and heterogeneous computing device 720 are installed can communicate and interact with electronic device 700.

[0162] A virtual device 711 and a virtual device 712 are created based on the heterogeneous computing device 710 ; a virtual device 721 and a virtual device 722 are created based on the heterogeneous computing device 720 .

[0163] The hardware resources of the heterogeneous computing device 710 include shared memory 713 , communication links 714 , and computing power units 715 .

[0164] The shared memory 713 may be all memory resources of the heterogeneous computing device 710 or part of the memory resources of the heterogeneous computing device 710. The virtual devices 711 and 712 may synchronize data based on the shared memory 713.

[0165] The communication link 714 is a data transmission link in the heterogeneous computing device 710 that can be provided to the virtual device for use, for example, a PCIe link or a network device link.

[0166] The computing unit 715 is a computing resource of the heterogeneous computing device 710 . The virtual device 711 and the virtual device 712 can share the computing unit 715 , for example, by time-sharing the computing unit 717 .

[0167] The hardware resources of the heterogeneous computing device 720 include a shared memory 723 , a communication link 724 , and a computing power unit 725 .

[0168] The shared memory 723 may be all memory resources of the heterogeneous computing device 720 , or may be part of the memory resources of the heterogeneous computing device 720 .

[0169] The communication link 724 is a data transmission link in the heterogeneous computing device 720 that can be provided to the virtual device for use, for example, a PCIe link or a network device link.

[0170] The computing unit 725 is a computing resource of the heterogeneous computing device 720 . The virtual device 721 and the virtual device 722 can share the computing unit 725 , for example, by time-sharing multiplexing the computing unit 725 .

[0171] Furthermore, in order to enable the parallel computing framework 501 to distribute parallel computing tasks to virtual devices, a communication virtualization device 503 is also deployed in the electronic device 500.

[0172] The communication virtualization device 503 is deployed outside the parallel computing framework 501. The communication virtualization device 503 is used to present multiple virtual devices to the parallel computing framework 501 in the form of multiple heterogeneous computing devices, so that the parallel computing framework 501 can distribute computing tasks to the virtual devices according to the communication method for heterogeneous computing devices, without having to configure communication settings for the virtual devices.

[0173] Those skilled in the art can construct the communication virtualization device 503 in combination with the properties of the virtual device. This application does not specifically limit the specific implementation method of the communication virtualization device 503.

[0174] According to the method of the embodiment of the present application, a communication virtualization device is deployed outside the parallel computing framework, and communication between the parallel computing framework and the virtual device is realized through the communication virtualization device. The parallel computing framework does not need to perceive the deployment and implementation status of the virtual device on the heterogeneous computing device, thereby reducing the difficulty of implementing communication between the parallel computing framework and the virtual device.

[0175] FIG6 is a schematic diagram of a model training process according to an embodiment of the present application.

[0176] S800 , creating multiple virtual devices, where the multiple virtual devices at least include a virtual device 511 and a virtual device 512 .

[0177] S801: The electronic device 500 queries all available virtual devices.

[0178] S802: The electronic device 500 generates a virtual device topology relationship table according to available virtual devices.

[0179] S803, the electronic device 500 obtains a training script and a sample data set.

[0180] S804 , the electronic device 500 sends the training script, the sample data set, and the virtual device topology relationship table to the parallel computing framework 501 .

[0181] S805, the parallel computing framework 501 divides the sample data set according to the user configuration and generates multiple parallel computing tasks according to the training script.

[0182] S806, the task scheduler 502 selects a virtual device for executing the computing task from multiple virtual devices according to the virtual device topology relationship table to form a computing task execution device set, which includes multiple virtual devices (at least virtual device 511 and virtual device 512).

[0183] According to the method of the embodiment of the present application, a virtual device is created before splitting the training task, and a virtual device for executing the computing task is selected from multiple virtual devices based on the split computing task. This can ensure the flexibility of computing task distribution and reduce the time consumption of computing task distribution.

[0184] S807 , the task scheduler 502 sends the computing task to the virtual devices in the computing task execution device set (including at least the virtual device 511 and the virtual device 512 ).

[0185] The communication between the parallel computing framework 501 and the virtual devices is implemented through the communication virtualization device 502 .

[0186] S810, the virtual devices (including at least virtual device 511 and virtual device 512) calculate the distributed computing tasks to perform iterative calculations for model training.

[0187] Furthermore, during the iterative calculation process of multiple virtual devices, there may be a need for communication between the virtual devices.

[0188] For example, in one embodiment, computing tasks are interconnected. The execution of one computing task requires the computational results of another computing task. For example, the parallel computing framework 501 divides the model training task into multiple computing tasks, each of which includes at least computing task A and computing task B. The execution of computing task B requires the computational results of computing task A. The parallel computing framework 501 distributes computing task A and computing task B to virtual devices 511 and 512, respectively.

[0189] FIG7 is a schematic diagram showing a partial process of model training according to an embodiment of the present application.

[0190] S900, the virtual device 511 receives computing task A.

[0191] S901, the virtual device 511 executes computing task A. After computing task A is completed, the virtual device 511 generates a computing result of computing task A.

[0192] S910, the virtual device 512 receives computing task B.

[0193] For example, computation tasks A and B are the steps for calculating the error values ​​of two adjacent layers during the forward calculation of the local error value in step a. The layer corresponding to computation task A is the layer above the layer corresponding to computation task B. The execution of computation task B requires calculations based on the results of computation task A.

[0194] For example, computation task A is the step in step a, during which the local error value is calculated forward, for a particular layer. Computation task B is the step in step b, during which the global error value is generated through integration. The execution of computation task B requires the collection of the computation results of computation task A.

[0195] S911 , the virtual device 512 determines that the communication object to be communicated is the virtual device 511 according to the virtual device topology relationship table of the computing task execution device set and the computing task B.

[0196] S912 , the virtual device 512 subscribes to the task execution status of the virtual device 511 .

[0197] S902: After the virtual device 511 completes the computing task A, it issues a computing task completion event.

[0198] S913, after receiving the task completion event, the virtual device 512 requests the virtual device 511 to obtain the calculation result of the calculation task A.

[0199] S914, the virtual device 512 obtains the calculation result of the calculation task A.

[0200] Specifically, in one embodiment, virtual device 511 and virtual device 512 are virtual devices created in the same heterogeneous computing device, and virtual device 512 obtains the first calculation result through internal communication between the heterogeneous computing devices. In another embodiment, virtual device 511 and virtual device 512 are virtual devices created in different heterogeneous computing devices, and virtual device 512 obtains the first calculation result through communication between the heterogeneous computing devices, for example, through network channel communication between the heterogeneous computing devices or PCIe data channel communication between the heterogeneous computing devices.

[0201] Furthermore, those skilled in the art may use a variety of methods to enable the virtual device 512 to obtain the first calculation result.

[0202] For example, virtual device 511 and virtual device 512 are virtual devices created in the same heterogeneous computing device. Virtual device 511 saves the first calculation result to the shared memory. Virtual device 511 publishes the storage location (memory address) of the first calculation result to virtual device 512. Virtual device 512 directly reads the first calculation result from the shared memory based on the storage location of the first calculation result.

[0203] For another example, virtual device 511 and virtual device 512 are virtual devices created in different heterogeneous computing devices. Virtual device 511 sends the first calculation result to the heterogeneous computing device where virtual device 512 is located via the communication channel between the heterogeneous computing devices. Alternatively, virtual device 511 sends the storage address of the first calculation result to the heterogeneous computing device where virtual device 512 is located via the communication channel between the heterogeneous computing devices; virtual device 512, based on the communication channel between the heterogeneous computing devices, directly accesses the memory of the heterogeneous computing device where virtual device 511 is located (Remote Direct Memory Access (RDMA)).

[0204] S915 , the virtual device 512 executes computing task B according to the computing result of computing task A.

[0205] According to the method of the embodiment of the present application, the virtual device perceives the real-time status of each computing task, and a communication task can be initiated in real time after the completion of a single computing task. Compared to the parallel computing framework that uniformly manages the status of all computing tasks, after a single computing task is completed, it is necessary to wait for the computing framework to issue a communication task to implement communication. The method of the embodiment of the present application can reduce computing power consumption.

[0206] Furthermore, when the first virtual device and the second virtual device are created within the same heterogeneous computing device, communication between the first and second virtual devices is essentially internal communication within the heterogeneous computing device. Internal communication between heterogeneous computing devices does not require interface calls or communication link establishment, and the communication time consumed is theoretically negligible.

[0207] Therefore, in one embodiment, when distributing computing tasks, computing tasks with communication requirements are preferentially distributed to virtual devices created in the same heterogeneous computing device, thereby reducing the waiting time when the virtual device executes the computing task.

[0208] According to the method of the embodiment of the present application, multiple virtual devices are deployed in a single heterogeneous computing device to realize the deployment of multi-process (multi-channel data) distributed parallel training in a single heterogeneous computing device. Theoretically, it is expected that the batch size deployed on a single heterogeneous computing device will be more than doubled.

[0209] FIG8 is a schematic diagram showing a virtual device structure according to an embodiment of the present application.

[0210] As shown in FIG. 8 , in one embodiment, a virtual device 1000 includes a routing device 1010 and a task tracking device 1020 .

[0211] FIG9 is a schematic diagram showing a partial process of model training according to an embodiment of the present application.

[0212] In the iterative calculation, the virtual device shown in FIG8 executes the following process shown in FIG9.

[0213] S1100, the task tracking device 1020 obtains a computing task.

[0214] S1101, the routing device 1010 obtains a virtual device topology relationship table of a computing task execution device set.

[0215] S1110 , the task tracking device 1020 determines, based on the computing task, a virtual device that needs to subscribe to the task execution status (a virtual device that needs to copy data).

[0216] When the task tracking device 1020 does not need to subscribe to the task execution status of other virtual devices, S1120 is executed.

[0217] S1120, the task tracking device 1020 sends the computing task to the computing power unit of the heterogeneous computing device, executes the computing task, and obtains the computing result.

[0218] S1121, the task tracking device 1020 monitors the execution status of the computing task, and publishes a computing task completion event after the computing task is completed.

[0219] When the task tracking device 1020 does not need to subscribe to the task execution status of other virtual devices, S1130 is executed.

[0220] S1130 , the task tracking device 1020 sends a device list of virtual devices that need to subscribe to the task execution status to the routing device 1010 , and notifies the routing device 1010 to establish a communication connection.

[0221] S1131, the routing device 1010 establishes a communication connection with the virtual device that needs to subscribe to the task execution status based on the virtual device topology relationship table of the computing task execution device set and the device list of the virtual device that needs to subscribe to the task execution status, thereby realizing the task execution status subscription of the virtual device.

[0222] S1132, the task tracking device 1020 waits for a computing task completion event of the virtual device that has subscribed to the task execution status, and after receiving the computing task completion event, obtains the computing result of the virtual device that has subscribed to the task execution status.

[0223] After S1132, S1120 is executed.

[0224] Furthermore, in one embodiment, the communication between the electronic device and the virtual device, and between the virtual devices, is implemented based on collective communication.

[0225] Collective communications are global communication operations that all processes in a process group participate in. The most basic operations include send, receive, copy, intra-group process barrier synchronization, and inter-node process synchronization (signal+wait). These basic operations are combined to form a set of communication templates, also known as communication primitives (or collective communication operators).

[0226] For example, collective communication operators include: 1-to-many broadcast, many-to-1 gather, many-to-many gather (all-gather), 1-to-many scatter, many-to-1 reduce, many-to-many reduce (all-reduce), combined reduce and scatter (reduce-scatter), many-to-many (all-to-all), etc.

[0227] Specifically, a routing device is deployed outside the parallel computing framework. The routing device is used to generate a collective communication operator for collective communication based on the virtual device topology relationship of multiple virtual devices. This allows the implementation of collective communication during iterative computation without the involvement of the parallel computing framework, improving the flexibility and efficiency of collective communication.

[0228] FIG10 is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application.

[0229] As shown in FIG10 , the hardware device for implementing model training includes an electronic device 1200 and multiple virtual devices created based on heterogeneous computing devices, and the multiple virtual devices include at least a virtual device 1210 and a virtual device 1220 .

[0230] The electronic device 1200 includes a parallel computing framework 1201 and a communication virtualization device 1202 . The parallel computing framework 1201 includes a task scheduler 1205 .

[0231] The parallel computing framework 1201 sends collective communication commands to the virtual devices through the communication virtualization device 1202 .

[0232] The communication virtualization apparatus 1202 includes a collective communication resource virtualization module 1203 and a virtual device super-resolution module 1204 .

[0233] The aggregate communication resource virtualization module 1203 is used to present the aggregate communication properties of virtual devices to upper-layer users and the parallel computing framework 1201. Specifically, it configures aggregate communication property resources for each virtual device, including but not limited to Rank and Group, and supports tools for querying communication topology properties, providing support for virtual device topology and communication properties.

[0234] The aggregate communication resource virtualization module 1203 is also used to support virtualization of aggregate communication physical resources. Specifically, it supports the control flow notification mechanism between virtual devices and the segmentation and super-division preemption of hardware communication resources (including but not limited to network channels (RDMA), PCIe data channels, etc.).

[0235] The virtual device super-resolution module 1204 is used to optimize virtualization technology, supporting spatial multiplexing of resources between multiple virtual devices on a single card and time multiplexing of the same resources; for example, virtual device 1211 occupies computing resources when performing computing tasks, while virtual device 1212 occupies network resources when performing communication tasks.

[0236] The routing device is deployed in each virtual device. Specifically, each of the multiple virtual devices is configured with a routing device, a task tracking device, and a diversity link implementation device. Taking virtual device 1210 as an example, virtual device 1210 includes a routing device 1211, a task tracking device 1212, and a diversity link implementation device 1213.

[0237] Routing device 1211 stores a virtual device routing table, which records all virtual devices requiring data copying for the current computing task, along with their topological connection relationships and other information. In one embodiment, routing device 1211 generates the virtual device routing table based on the virtual device topology relationship table for the computing task execution device set and the device list of virtual devices requiring task execution status subscription, as determined by task tracking device 1212.

[0238] For example, in an application scenario, in a routing device of a virtual device, the content of the virtual device routing table is as shown in Table 1.

[0239] Table 1

[0240] The routing device 1211 is used to generate a collective communication operator according to the virtual device routing table, and based on the collective communication operator, drive the diversity link implementation device to select a communication link to establish a communication connection. The routing device 1211 is triggered by the task tracking device 1212.

[0241] The diversity link implementation device 1213 is used to select a communication link based on the collective communication operator, thereby establishing a communication connection based on the selected communication link. For example, the diversity link implementation device 1213 selects a corresponding hardware unit (DMA controller, network controller) based on the type of the collective communication operator to establish a communication connection and implement data synchronization.

[0242] Specifically, in one embodiment, the diversity link implementation device 1213 implements data synchronization specifically as follows:

[0243] Data communication is performed between virtual devices of the same heterogeneous computing device by passing memfd through shared memory;

[0244] Supports link resource segmentation between virtual devices of different heterogeneous computing devices of the same electronic device based on hardware type (PCIE / HCCS);

[0245] Between virtual devices of different heterogeneous computing devices across electronic devices, the RDMA / TCP protocol is selected for implementation, and multi-process port isolation is adopted between virtual devices.

[0246] According to the method of the embodiment of the present application, the parallel computing framework is not responsible for managing the device topology and inter-device communication, and the collective communication capabilities are transferred to the virtual devices.

[0247] Furthermore, during the iterative computing process, there is an application scenario where a computing task from one virtual device (the task-migrating device) is migrated to another virtual device (the task-migrating device) for execution. After the computing task is migrated, the task-migrating device in the computing task execution device set is replaced by the task-migrating device.

[0248] In one embodiment, computing task migration is performed on the virtual device side, eliminating the need for the front-end user interface and parallel computing framework on the electronic device side to perceive or respond to the computing task migration. Specifically, after a computing task migration occurs, no information indicating a change in the virtual device topology is sent to the parallel computing framework, rendering the parallel computing framework unaware of the change. This eliminates the need for computing task migration to trigger the resharding and redistribution of computing tasks, and does not disrupt the execution of a single iterative computation.

[0249] FIG11 is a schematic diagram of a virtual device according to an embodiment of the present application.

[0250] As shown in Figure 11, virtual device 1301 (first virtual device), virtual device 1302 (fourth virtual device), and virtual device 1303 (second virtual device) are virtual devices in the computing task execution device set. Virtual device 1304 (third virtual device) is a virtual device outside the computing task execution device set.

[0251] Virtual devices 1301, 1302, and 1303 are assigned computing tasks S1 (the first computing task), S2 (the third computing task), and S3 (the second computing task), respectively. The execution of computing task S2 requires the computation result of computing task S1. Therefore, after the computing tasks are assigned, virtual device 1302 subscribes to the task execution status of virtual device 1301. The execution of computing task S3 does not require the computation result of computing task S1. Therefore, after the computing tasks are assigned, virtual device 1303 does not subscribe to the task execution status of virtual device 1301.

[0252] FIG12 is a schematic diagram showing a computing task migration process according to an embodiment of the present application.

[0253] S1400, the virtual device 1301 receives the computing task S1.

[0254] S1401, the virtual device 1302 receives the computing task S2.

[0255] S1402, the virtual device 1303 receives the computing task S3.

[0256] S1403, the virtual device 1301 executes the computing task S1.

[0257] S1404 , the virtual device 1302 subscribes to the task execution status of the virtual device 1301 .

[0258] S1405, the virtual device 1303 executes the computing task S3 and generates a computing result of the computing task S3.

[0259] S1406 , triggering computing task migration, where computing task S1 of virtual device 1301 is migrated to virtual device 1304 .

[0260] Specifically, this application does not impose any specific restrictions on the triggering conditions for computing task migration. For example, in one embodiment, an abnormal execution of a computing task triggers computing task migration. In another embodiment, dynamic scheduling of a virtual device triggers computing task migration.

[0261] S1410, the virtual device 1301 terminates the execution of the computing task S1.

[0262] S1410 , the virtual device 1301 issues a task termination event.

[0263] S1411 , the virtual device 1302 receives a task termination event.

[0264] Since the virtual device 1303 does not subscribe to the task execution status of the virtual device 1301 , the virtual device 1303 does not receive the task termination event, or the virtual device 1303 does not respond after receiving the task termination event.

[0265] S1412 , the virtual device 1302 subscribes to the task execution status of the virtual device 1304 according to the task termination event.

[0266] S1420 , synchronizing the calculation data of the virtual device 1301 and the virtual device routing table in the routing apparatus to the virtual device 1304 .

[0267] S1421: After data synchronization is completed, the virtual device 1304 issues a task migration event.

[0268] During the data synchronization process, the virtual device 1304 updates the synchronized virtual device routing table.

[0269] S1422 , the virtual device 1302 receives a task migration event.

[0270] Since the virtual device 1303 does not subscribe to the task execution status of the virtual device 1304 , the virtual device 1303 does not receive the task termination event, or the virtual device 1303 does not respond after receiving the task termination event.

[0271] S1423, virtual device 1302 cancels subscription to virtual device 1301, and updates the virtual device topology relationship table of the computing task execution device set (updates the virtual device routing table).

[0272] Specifically, in one embodiment, after receiving the task termination event, if the virtual device 1302 does not receive the task migration event, the virtual device 1302 determines that the computing task migration fails, reports an error, terminates the current iterative calculation, or triggers the computing task migration to be re-executed.

[0273] After S1423 , the information indicating the change in the topology of the virtual devices is not fed back to the parallel computing framework, so that the parallel computing framework does not perceive the change in the topology of the virtual devices.

[0274] After S1421 , the virtual device 1304 executes S1430 .

[0275] S1430, the virtual device 1304 continues to execute the computing task S1 and generates a computing result of the computing task S1.

[0276] S1431, after the virtual device 1304 completes the computing task S1, it issues a computing task completion event.

[0277] S1432, after receiving the computing task completion event, the virtual device 1302 obtains the computing result of the computing task S1.

[0278] S1433, the virtual device 1302 executes the computing task S2 according to the computing result of the computing task S1, and generates the computing result of the computing task S2.

[0279] S1430. After this iteration is completed, the virtual devices of the computing task execution device set (including virtual device 1302, virtual device 1303, virtual device 1304, excluding virtual device 1301) synchronously update the virtual device topology relationship table of the computing task execution device set (update the virtual device routing table in the routing device), wherein the virtual device topology relationship table of the virtual device routing table in the routing device of virtual device 1303 is synchronously updated to the virtual device topology relationship table of the virtual device routing table in the routing devices of virtual devices 1302 and 1304.

[0280] In one embodiment, after S1430, no information indicating a change in the virtual device topology is fed back to the parallel computing framework, and the communication virtualization device presents the unchanged virtual device topology at the parallel computing framework by replacing descriptive information such as the virtual device name. For example, before the computing task of virtual device 1301 is migrated to virtual device 1304, the communication virtualization device presents virtual device 1301 as heterogeneous computing device A at the parallel computing framework. After the computing task of virtual device 1301 is migrated to virtual device 1304, the communication virtualization device presents virtual device 1304 as heterogeneous computing device A at the parallel computing framework. In this way, the device topology of the multiple heterogeneous computing devices presented by the multiple virtual devices at the parallel computing framework remains unchanged.

[0281] In another embodiment, after S1430, information indicating a change in the virtual device topology is fed back to the parallel computing framework so that the parallel computing framework can perform computing task segmentation and distribution for the next iteration based on the new virtual device topology.

[0282] FIG13 is a schematic diagram of a computing task migration process according to an embodiment of the present application.

[0283] S1500, the virtual device 1301 receives the computing task S1.

[0284] S1501, the virtual device 1302 receives the computing task S2.

[0285] S1502, the virtual device 1303 receives the computing task S3.

[0286] S1503, the virtual device 1301 executes the computing task S1.

[0287] S1504 , the virtual device 1302 subscribes to the task execution status of the virtual device 1301 .

[0288] S1505, the virtual device 1303 executes the computing task S3 and generates a computing result of the computing task S3.

[0289] S1506, triggering computing task migration, computing task migration is computing task S1 of virtual device 1301 migrating to virtual device 1304.

[0290] S1510, the virtual device 1301 terminates the execution of the computing task S1.

[0291] S1511 , the virtual device 1301 broadcasts a task termination event to the computing task execution device set.

[0292] The virtual devices of the computing task execution device set (including virtual device 1302 and virtual device 1303) receive a task termination event, wherein virtual device 1303 may receive the task termination event during the execution of computing task S3 or after the execution of computing task S3.

[0293] S1520 , synchronize the computing data and the virtual device routing table of the virtual device 1301 to the virtual device 1304 .

[0294] S1521, after data synchronization is completed, the virtual device 1304 broadcasts a task migration event to the computing task execution device set.

[0295] The virtual devices of the computing task execution device set (including virtual device 1302 and virtual device 1303) receive a task migration event, wherein virtual device 1303 may receive the task migration event during the execution of computing task S3 or after the execution of computing task S3.

[0296] Specifically, in one embodiment, after the virtual devices (including virtual device 1302 and virtual device 1303) of the computing task execution device set receive a task termination event but do not receive a task migration event, it is determined that the computing task migration has failed, and the current iterative calculation is terminated or the computing task migration is triggered to be re-executed.

[0297] S1522 , the virtual device 1302 cancels its subscription to the virtual device 1301 and subscribes to the task execution status of the virtual device 1304 .

[0298] S1523, the virtual devices (including virtual device 1302 and virtual device 1303) of the computing task execution device set update the virtual device topology relationship table of the computing task execution device set (update the virtual device routing table in the routing device) after receiving the task migration event.

[0299] After S1523 , the information indicating the change in the topology of the virtual devices is not fed back to the parallel computing framework, so that the parallel computing framework does not perceive the change in the topology of the virtual devices.

[0300] After S1523 , the virtual device 1304 executes S1530 .

[0301] S1530, the virtual device 1304 continues to execute the computing task S1 and generates a computing result of the computing task S1.

[0302] S1531, after the virtual device 1304 completes the computing task S1, it issues a computing task completion event.

[0303] S1532, after receiving the computing task completion event, the virtual device 1302 obtains the computing result of the computing task S1.

[0304] S1533, the virtual device 1302 executes the computing task S2 according to the computing result of the computing task S1, and generates the computing result of the computing task S2.

[0305] According to the method of the embodiment of the present application, the parallel computing framework is not responsible for managing the device topology and inter-device communication, and the collective communication capabilities are transferred to the virtual device, supporting the dynamic migration of computing tasks during iterative computing and the dynamic update of virtual device link information. There is no need for the computing framework layer to perceive and recompile the generated communication operators, thereby reducing the waste of computing power waiting.

[0306] In large-scale distributed training scenarios, the product implementation of this invention consists of program code contained within the machine learning and deep learning platform software and deployed on server hardware. The program code of this invention resides within the platform software's runtime engine and communication management modules, as well as externally within existing modules. During runtime, the program code of this invention runs in the server's electronic device memory and the memory of heterogeneous devices such as the NPU / GPU / Ascend.

[0307] 8 and 10 , in one embodiment of the present application, a routing device is deployed in each virtual device. Specifically, a routing device and a task tracking device are deployed in each virtual device in a computing task execution device set.

[0308] For example, in one embodiment, the Mindspor distributed training framework is used to implement large-model distributed training based on vNPU virtual devices.

[0309] FIG14 is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application.

[0310] As shown in FIG14 , the hardware device for implementing model training includes an electronic device 1600 . The electronic device 1600 includes a Mindspor distributed training framework 1601 . The Mindspor distributed training framework 1601 includes a task scheduler 1602 .

[0311] Electronic device 1600 also includes a communication virtualization device 1603. This virtualization device 1603 virtualizes a heterogeneous computing device (NPU device) into multiple virtual devices (vNPU devices) for the Mindspor distributed training framework 1601 and the user, creating a "one virtual multiple" effect for heterogeneous computing devices. For example, as shown in FIG12 , the multiple virtual devices include at least virtual device 1610, virtual device 1620, and so on.

[0312] The user specifies one or more virtual devices for model training by configuring the RankTable; the Mindspor distributed training framework 1501 compiles and splits the training tasks according to the parallel strategy specified by the user, generates multi-path computing tasks, and schedules virtual devices to complete the training.

[0313] In the embodiment shown in FIG12 , corresponding routing devices, task tracking devices, and link diversity implementation devices are deployed in each virtual device. For example, virtual device 1610 includes routing devices 1611, task tracking devices 1612, and link diversity implementation devices 1613; virtual device 1620 includes routing devices 1621, task tracking devices 1622, and link diversity implementation devices 1623.

[0314] In another embodiment, the routing device is deployed in the electronic device. Specifically, the routing device and the task tracking device are deployed in the electronic device, and the routing device and the task tracking device deployed in the electronic device manage each virtual device in the computing task execution device set.

[0315] For example, in one embodiment, the PyTorch distributed training framework is used to implement large model distributed training based on vNPU or vGPU devices.

[0316] FIG15 is a schematic diagram of a hardware device architecture for implementing model training according to an embodiment of the present application.

[0317] As shown in FIG15 , the hardware device for implementing model training includes an electronic device 1700 . The electronic device 1700 includes a PyTorch distributed training framework 1701 . The PyTorch distributed training framework 1701 includes a task scheduler 1702 .

[0318] Electronic device 1700 also includes a communication virtualization device 1703. This virtualization device 1703 virtualizes heterogeneous computing devices into multiple virtual devices (vNPU devices or vGPU devices) for the PyTorch distributed training framework 1701 and users, creating a "one virtual multiple" effect. For example, as shown in FIG13 , the multiple virtual devices include at least virtual device 1710, virtual device 1720, and so on.

[0319] The PyTorch distributed training framework 1701 is a single communication operator delivery mode, and all collective communication operators are generated and delivered in the electronic device. Specifically, a routing device 1704 and a task tracking device 1705 are deployed in the electronic device 1700. Multiple instance objects are created in the routing device 1704 and the task tracking device 1705. The instance objects of the routing device 1704 and the task tracking device 1705 maintain a one-to-one relationship with the virtual device, and each instance object manages a virtual device. A diversity link implementation device is deployed in each virtual device. For example, virtual device 1710 includes a diversity link implementation device 1713; virtual device 1720 includes a diversity link implementation device 1723.

[0320] In the computing task migration application scenario, the routing device 1704 and the task tracking device 1705 jointly complete the dynamic switching of links when the topology changes without the need for the PyTorch distributed training framework 1701 to perceive and intervene.

[0321] According to the method of an embodiment of the present application, an embodiment of the present application further proposes a model training device, which includes a parallel computing framework and multiple virtual devices, wherein:

[0322] The multiple virtual devices are virtual devices created based on heterogeneous computing devices;

[0323] A parallel computing framework is loaded on the electronic device. The parallel computing framework is used to divide the model training task into multiple computing tasks and distribute the multiple computing tasks to multiple virtual devices, wherein the multiple virtual devices correspond one-to-one to the multiple computing tasks.

[0324] Multiple virtual devices are used to execute distributed computing tasks to perform iterative calculations for model training.

[0325] Optionally, in one embodiment, the apparatus further includes a communication virtualization device, wherein:

[0326] The communication virtualization device is deployed on the electronic device, wherein the communication virtualization device is deployed outside the parallel computing framework;

[0327] The communication virtualization device is used to present multiple virtual devices to the parallel computing framework in the form of multiple heterogeneous computing devices.

[0328] Optionally, in one embodiment, the communication between the parallel computing framework and the virtual device, and between different virtual devices, is collective communication;

[0329] The communication virtualization device includes:

[0330] A collective communication resource virtualization module, which is used to configure collective communication attribute resources for virtual devices and present the collective communication attributes of virtual devices to the parallel computing framework;

[0331] The virtual device super-resolution module is used to support virtualization of collective communication physical resources.

[0332] Optionally, in one embodiment, the model training device further includes a routing device, wherein:

[0333] The routing device is deployed outside the parallel computing framework and is used to generate a collective communication operator for collective communication according to a virtual device topology relationship of a plurality of virtual devices.

[0334] Optionally, in one embodiment, the routing device is used to save a virtual device routing table, which is used to record all virtual devices and topological connection relationships that require data copying for the current computing task.

[0335] Optionally, in one embodiment, the routing device is deployed on an electronic device, and the routing device includes instance objects corresponding one-to-one to multiple virtual devices.

[0336] Optionally, in one embodiment, the routing device is deployed in each virtual device.

[0337] Optionally, in one embodiment: the plurality of computing tasks include at least a first computing task and a second computing task; the plurality of virtual devices include at least a first virtual device and a second virtual device; the parallel computing framework distributes the first computing task to the first virtual device and distributes the second computing task to the second virtual device; the first virtual device executes the first computing task, and the second virtual device executes the second computing task;

[0338] The apparatus further includes a task migration module and a third virtual device, wherein:

[0339] The task migration module is configured to: upon triggering task migration from the first virtual device to the third virtual device, terminate the execution of the first computing task by the first virtual device, and maintain the second virtual device to continue executing the second computing task; perform task migration, including synchronizing the first computing task in the first virtual device and the current execution state of the first computing task to the third virtual device;

[0340] The third virtual device is used to continue executing the first computing task after the task migration.

[0341] Optionally, in one embodiment, the routing device is further configured to, after the task migration, update the topological relationship of the virtual devices in the routing device according to the task migration.

[0342] Optionally, in one embodiment, the routing device is further configured to, after task migration, not feed back information indicating changes in the topological relationship of the virtual devices to the parallel computing framework.

[0343] Optionally, in one embodiment, the plurality of computing tasks further include at least a third computing task, and execution of the third computing task requires reference to a computing result of the first computing task; the plurality of virtual devices further include at least a fourth virtual device;

[0344] The parallel computing framework is used to distribute the third computing task to the fourth virtual device;

[0345] The fourth virtual device is used to: subscribe to the task execution status of the first virtual device according to the third computing task; after the first virtual device completes the execution of the first computing task, the fourth virtual device obtains the calculation result of the first computing task; and execute the third computing task according to the calculation result of the first computing task.

[0346] Optionally, in one embodiment, the fourth virtual device is further configured to:

[0347] After the task migration, the task execution state of the first virtual device is updated to the task execution state of the third virtual device.

[0348] Optionally, in one embodiment:

[0349] The task migration module is used to synchronize the virtual device topology relationship in the routing device of the first virtual device to the routing device of the third virtual device;

[0350] The routing device of the third virtual device is used to update the virtual device topology relationship in the routing device of the third virtual device according to the task migration after the task migration;

[0351] The routing device of the fourth virtual device is used to update the virtual device topology relationship in the routing device of the fourth virtual device according to the task migration after the task migration.

[0352] In the description of the embodiments of the present application, for the convenience of description, the device is described as being divided into various modules according to their functions. The division of each module is merely a division of logical functions. When implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0353] Specifically, the device proposed in the embodiment of the present application can be fully or partially integrated into a physical entity during actual implementation, or it can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; or they can all be implemented in the form of hardware; or some modules can be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the detection module can be a separately established processing element, or it can be integrated in a chip of an electronic device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. During the implementation process, each step of the above method or each of the above modules can be completed by the hardware integrated logic circuit in the processor element or the instructions in the form of software.

[0354] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0355] The present application also provides an electronic device.

[0356] FIG16 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

[0357] As shown in Figure 16, the electronic device 2300 includes a memory 2302 for storing computer program instructions and a processor 2301 for executing program instructions, wherein, when the computer program instructions are executed by the processor 2301, the electronic device 2300 is triggered to execute the method steps of the method described in the embodiment of the present application.

[0358] Specifically, in one embodiment of the present application, the above-mentioned one or more computer programs are stored in the above-mentioned memory 2302, and the above-mentioned one or more computer programs include instructions. When the above-mentioned instructions are executed by the above-mentioned electronic device 2300, the above-mentioned electronic device 2300 executes the method steps described in the embodiment of the present application.

[0359] It is understood that the structural description of the electronic device 2300 in the embodiment of the present application does not constitute a specific limitation on the electronic device 2300. In other embodiments of the present application, the electronic device 2300 may include other components besides the processor 2301 and the memory 2302.

[0360] The processor 2301 may be a device on a chip (SOC), and the processor 2301 may include a central processing unit (CPU), and may further include other types of processors.

[0361] The processor involved in processor 2301 may include, for example, a CPU, a DSP, a microcontroller, or a digital signal processor, and may also include a GPU, an embedded neural network processor (NPU), and an image signal processor (ISP). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASICs, or one or more integrated circuits for controlling the execution of the program of the technical solution of this application. In addition, the processor may have the function of operating one or more software programs, and the software programs may be stored in a storage medium.

[0362] The processor 2301 may include one or more processing units. For example, the processor may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent components or integrated into one or more processors. In some embodiments, the electronic device 2300 may also include one or more processors 2301. The controller may generate an operation control signal based on the instruction opcode and the timing signal to complete the control of instruction fetching and execution.

[0363] In some embodiments, the processor 2301 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface, etc. Among them, the USB interface is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface can be used to connect a charger to charge the electronic device, and can also be used to transmit data between the electronic device and peripheral devices.

[0364] Electronic device 2300 may also include an external memory interface for connecting an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with processor 2301 via the external memory interface to implement data storage. For example, files such as music and videos can be stored on the external memory card.

[0365] Memory 2302 may include a code storage area and a data storage area. The code storage area may store an operating system. The data storage area may store data created during the use of electronic device 2300. Furthermore, memory 2302 may include high-speed random access memory and non-volatile memory, such as one or more disk storage components, flash memory components, or universal flash storage (UFS).

[0366] The memory 2302 may be a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any computer-readable medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer.

[0367] The processor 2301 and the memory 2302 may be combined into one processing device, or more commonly, they may be independent components.

[0368] In one embodiment, the electronic device proposed in this application may be the electronic device described in the above embodiments. Specifically, the electronic device is used to load a parallel computing framework, and the parallel computing framework loaded by the electronic device is used to execute the method steps performed by the parallel computing framework in the method described in the embodiment of the present application. Specifically, when the computer program instructions in the memory 2302 are executed by the processor 2301, the electronic device 2300 is triggered to execute the method steps performed by the electronic device in the method described in the embodiment of the present application.

[0369] In another embodiment, the electronic device proposed in this application may be the heterogeneous computing device described in the above embodiment. Specifically, the electronic device is used to create a virtual device, and the virtual device created in the electronic device is used to execute the method steps performed by the virtual device in the method described in the embodiment of this application. Specifically, when the computer program instructions in the memory 2302 are executed by the processor 2301, the electronic device 2300 is triggered to execute the method steps performed by the heterogeneous computing device in the method described in the embodiment of this application.

[0370] This application also proposes a server cluster, comprising a memory for storing computer program instructions and a processor for executing the computer program instructions. When the computer program instructions are executed by the processor, the server cluster is triggered to perform the method steps described in the embodiments of this application. Specifically, in one embodiment, the server cluster includes the electronic device described in the embodiments of this application and a heterogeneous computing device.

[0371] Furthermore, the devices, apparatuses, and modules described in the embodiments of the present application may be implemented by computer chips or entities, or by products having certain functions.

[0372] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, apparatus, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0373] In the several embodiments provided in this application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this application.

[0374] Specifically, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, the computer executes the method provided in the embodiment of the present application.

[0375] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program product is run on a computer, it enables the computer to execute the method provided in the embodiment of the present application.

[0376] The embodiment description in this application is described with reference to the flow chart and / or block diagram according to the method, equipment (device) and computer program product of embodiment of the present application.It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of general-purpose computer, special-purpose computer, embedded processing machine or other programmable data processing equipment to produce a machine, so that the instruction executed by the processor of computer or other programmable data processing equipment produces the device for realizing the function specified in one flow chart flow chart or multiple flow charts and / or one block or multiple blocks of block diagram.

[0377] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0378] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0379] It should also be noted that, in the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can represent: a, b, c, a and b, a and c, b and c, or a, b and c, where a, b, c can be single or multiple.

[0380] In the embodiments of the present application, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, commodity, or apparatus comprising the element.

[0381] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0382] The various embodiments in this application are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the partial description of the method embodiments.

[0383] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments of the present application can be implemented using a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0384] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, apparatuses and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0385] The above description is merely a specific embodiment of the present application. Any person skilled in the art may easily conceive of variations or substitutions within the technical scope disclosed in this application, and such variations or substitutions shall be within the scope of protection of this application. The scope of protection of this application shall be subject to the scope of protection of the claims.

Claims

1. A model training method, characterized in that: The method is applied to an electronic device and a heterogeneous computing device loaded with a parallel computing framework, and the method includes: The parallel computing framework divides the model training task into multiple computing tasks; The parallel computing framework distributes the multiple computing tasks to multiple virtual devices, wherein the multiple virtual devices correspond to the multiple computing tasks one by one, and the multiple virtual devices are virtual devices created based on the heterogeneous computing devices; The multiple virtual devices execute the distributed computing tasks to perform iterative calculations for model training.

2. The method according to claim 1, characterized in that A communication virtualization device is deployed on the electronic device, and the communication virtualization device presents the multiple virtual devices to the parallel computing framework in the form of multiple heterogeneous computing devices, wherein the communication virtualization device is deployed outside the parallel computing framework.

3. The method according to claim 2, characterized in that: Implementing communication between the parallel computing framework and the plurality of virtual devices, and between the plurality of virtual devices based on collective communication; The communication virtualization device is also used to: configure collective communication attribute resources for the virtual device, present the collective communication attributes of the virtual device to the parallel computing framework; and provide virtualization support for the collective communication physical resources.

4. The method according to claim 3, characterized in that A routing device is deployed outside the parallel computing framework, wherein the routing device is used to generate a collective communication operator for collective communication according to the virtual device topology relationship of the multiple virtual devices.

5. The method according to claim 4, characterized in that The routing device is used to store a virtual device routing table, and the virtual device routing table is used to record all virtual devices and topological connection relationships that need to copy data for the current computing task.

6. The method according to claim 4 or 5, characterized in that: The routing device is deployed on the electronic device, and the routing device includes instance objects corresponding to a plurality of the virtual devices in a one-to-one manner.

7. The method according to claim 4 or 5, characterized in that: The routing device is deployed in each of the virtual devices.

8. The method according to any one of claims 4 to 7, characterized in that: The multiple computing tasks include at least a first computing task and a second computing task; The parallel computing framework distributes the multiple computing tasks to multiple virtual devices, including distributing the first computing task to a first virtual device and distributing the second computing task to a second virtual device; The virtual device executes the distributed computing task to perform iterative computing of model training, including: The first virtual device performs the first computing task, and the second virtual device performs the second computing task; In the case of triggering task migration from the first virtual device to the third virtual device, terminating the first virtual device from executing the first computing task, and maintaining the second virtual device to continue executing the second computing task; Executing the task migration includes synchronizing the first computing task and the current execution state of the first computing task in the first virtual device to the third virtual device; The third virtual device continues to execute the first computing task.

9. The method according to claim 8, characterized in that The virtual device executes the distributed computing task to perform iterative computing of model training, and further includes: After the task migration, the virtual device topology relationship in the routing device is updated according to the task migration.

10. The method according to claim 8, characterized in that The virtual device executes the distributed computing tasks to perform iterative computing of model training, and also includes: After the task migration, no information indicating a change in the topological relationship of the virtual devices is fed back to the parallel computing framework.

11. The method according to claim 9, characterized in that The multiple computing tasks further include at least a third computing task, and execution of the third computing task requires reference to a computing result of the first computing task; The parallel computing framework distributes the plurality of computing tasks to a plurality of virtual devices, further comprising distributing the third computing task to a fourth virtual device; The multiple virtual devices execute the distributed computing tasks to perform iterative computing of model training, including: the fourth virtual device subscribes to the task execution status of the first virtual device according to the third computing task; After the first virtual device completes executing the first computing task, the fourth virtual device obtains the computing result of the first computing task and performs the computing result of the first computing task according to the computing result of the first computing task. The third computing task is executed based on the computing result.

12. The method according to claim 11, characterized in that The virtual device executes the distributed computing task to perform iterative computing of model training, and further includes: After the task migration, the fourth virtual device is updated to subscribe to the task execution state of the third virtual device.

13. The method according to claim 12, characterized in that: Executing the task migration further includes synchronizing the virtual device topology relationship in the routing device of the first virtual device to the routing device of the third virtual device; After the task migration, updating the virtual device topology relationship in the routing device according to the task migration includes: The virtual device topology relationship in the routing device of the third virtual device is updated, and the virtual device topology relationship in the routing device of the fourth virtual device is updated.

14. A model training device, characterized in that: The apparatus comprises a parallel computing framework and a plurality of virtual devices, wherein: The multiple virtual devices are virtual devices created based on heterogeneous computing devices; The parallel computing framework is loaded on the electronic device, and is used to divide the model training task into multiple computing tasks, and distribute the multiple computing tasks to the multiple virtual devices, wherein the multiple virtual devices correspond to the multiple computing tasks one by one; The multiple virtual devices are used to execute the distributed computing tasks to perform iterative calculations for model training.

15. The device according to claim 14, characterized in that The device also includes a communication virtualization device, wherein: The communication virtualization device is deployed on the electronic device, wherein the communication virtualization device is deployed outside the parallel computing framework; The communication virtualization device is used to present the multiple virtual devices to the parallel computing framework in the form of multiple heterogeneous computing devices.

16. The device according to claim 15, characterized in that The communication between the parallel computing framework and the virtual device, and between different virtual devices, is collective communication; The communication virtualization device comprises: A collective communication resource virtualization module, which is used to configure collective communication attribute resources for the virtual device and present the collective communication attributes of the virtual device to the parallel computing framework; The virtual device super-resolution module is used to support virtualization of collective communication physical resources.

17. The device according to claim 16, characterized in that The device also includes a routing device, wherein: The routing device is deployed outside the parallel computing framework, and is used to generate a collective communication operator for collective communication according to the virtual device topology relationship of the multiple virtual devices.

18. The device according to claim 17, characterized in that The routing device is used to store a virtual device routing table, and the virtual device routing table is used to record all virtual devices and topological connection relationships that need to copy data for the current computing task.

19. The device according to claim 17 or 18, characterized in that The routing device is deployed on the electronic device, and the routing device includes instance objects corresponding to a plurality of the virtual devices in a one-to-one manner.

20. The device according to claim 17 or 18, characterized in that The routing device is deployed in each of the virtual devices.

21. The device according to any one of claims 17 to 20, characterized in that: The multiple computing tasks include at least a first computing task and a second computing task; the multiple virtual devices include at least a first virtual device and a second virtual device; the parallel computing framework distributes the first computing task to the first virtual device, and distributes the second computing task to the second virtual device; The first virtual device performs the first computing task, and the second virtual device performs the second computing task; The apparatus further comprises a task migration module and a third virtual device, wherein: The task migration module is used to: in the case of triggering task migration from the first virtual device to the third virtual device, terminate the first virtual device from executing the first computing task, and maintain the second virtual device to continue executing the second computing task; perform the task migration, including synchronizing the first computing task in the first virtual device and the current execution state of the first computing task to the third virtual device; The third virtual device is used to continue executing the first computing task after the task migration.

22. The device according to claim 21, characterized in that The routing device is further configured to update the virtual device topology relationship in the routing device according to the task migration after the task migration.

23. The device according to claim 21, characterized in that The routing device is also used for not feeding back information indicating changes in the topological relationship of the virtual devices to the parallel computing framework after the task migration.

24. The device according to claim 22, characterized in that The plurality of computing tasks further include at least a third computing task, and the execution of the third computing task needs to refer to the computing result of the first computing task; the plurality of virtual devices further include at least a fourth virtual device; The parallel computing framework is used to distribute the third computing task to the fourth virtual device; The fourth virtual device is used to: subscribe to the task execution status of the first virtual device according to the third computing task; after the first virtual device completes the execution of the first computing task, the fourth virtual device obtains the calculation result of the first computing task; and execute the third computing task according to the calculation result of the first computing task.

25. The device according to claim 24, characterized in that The fourth virtual device is further used for: After the task migration, the task execution state of subscribing to the first virtual device is updated to the task execution state of subscribing to the third virtual device.

26. The device according to claim 25, characterized in that: The task migration module is used to synchronize the virtual device topology relationship in the routing device of the first virtual device to the routing device of the third virtual device; The routing device of the third virtual device is used to update the virtual device topology relationship in the routing device of the third virtual device according to the task migration after the task migration; The routing device of the fourth virtual device is used for updating the virtual device topology relationship in the routing device of the fourth virtual device according to the task migration after the task migration.

27. An electronic device, characterized in that: The electronic device comprises a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method steps as described in any one of claims 1-13.

28. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 13.

29. A server cluster, characterized in that: The server cluster comprises a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the server cluster is triggered to execute the method steps as described in any one of claims 1-13.

Images