Heterogeneous computing platform and model data processing method, device and equipment thereof, and medium

Abstract

The application discloses a kind of heterogeneous computing platforms and its model data processing method, device, equipment, medium, applied to heterogeneous computing field.Therein, method includes the multiple computing power node devices of heterogeneous computing platform are adapted to target deep learning framework, using same target deep learning framework jointly to be trained network model is trained to model.For each computing power node device is uniformly coded, and based on the computing power resource type corresponding to each computing power node device in distributed communication interface Corresponding target communication interface is determined;When based on the coding information of each computing power node device, to each computing power node device for distributed training process in the training network model being issued, the transmission of model data is carried out between each computing power node device through corresponding target communication interface.The application can solve the problem that related technology does not support multiple computing power node devices to carry out heterogeneous computing, and different types of computing power node devices can be used to complete the training of neural network model.

Description

Technical Field

[0001] This invention relates to the field of heterogeneous computing, and in particular to a heterogeneous computing platform and its model data processing method, apparatus, electronic device, and non-volatile storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, the scale of artificial intelligence network models is constantly expanding. In order to meet the computing power resources required for training large-scale neural network models, a distributed training model is adopted using a heterogeneous computing platform to train large-scale neural network models.

[0003] A heterogeneous computing platform is a computing system composed of computing node devices with various different types of computing resources. When using the computing node devices of a heterogeneous computing platform to collaboratively train a large-scale neural network model, the related technologies require that the computing devices participating in the training task be produced by the same manufacturer, or even be the same generation of computing devices from the same manufacturer.

[0004] Therefore, enabling different types of computing node devices to jointly train neural network models is a technical problem that needs to be solved by professionals in this field.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] This invention provides a heterogeneous computing platform and its model data processing method, device, electronic device, and non-volatile storage medium, enabling different types of computing node devices to jointly complete the training of neural network models, and enabling the training of larger-scale neural network models.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] This invention provides a method for processing model data on a heterogeneous computing platform, comprising:

[0009] The heterogeneous computing platform adapts various types of computing power node devices to the target deep learning framework, so that each computing power node device of the heterogeneous computing platform can use the target deep learning framework to jointly train the network model to be trained.

[0010] Each computing node device is uniformly coded, and the corresponding target communication interface is determined in the distributed communication interface based on the computing resource type of each computing node device.

[0011] During the distributed training process, based on the encoding information of each computing node device, the network model to be trained is distributed to each computing node device, and the model data is transmitted between each computing node device through the corresponding target communication interface.

[0012] In a first exemplary embodiment, the computing node devices transmit model data to each other through corresponding target communication interfaces, including:

[0013] When running at the distributed communication interface, obtain the configuration information of the computing node devices;

[0014] If the first and second computing power node devices currently transmitting data have the same computing power resource type, then the first and second computing power node devices transmit data based on the corresponding distributed communication library.

[0015] If the computing power resource types of the first computing power node device and the second computing power node device currently transmitting data are different, the first computing power node device and the second computing power node device will implement data transmission through a pre-defined method.

[0016] In a second exemplary embodiment, the first computing node device and the second computing node device implement data transmission through a pre-defined method, including:

[0017] The input data of the distributed communication interface is copied to the central processing unit in advance;

[0018] The first computing node device and the second computing node device transmit data through the central processing unit.

[0019] In a third exemplary embodiment, the distributed communication interface adopts a ring protocol architecture, and the computing node devices transmit model data through corresponding target communication interfaces, including:

[0020] Obtain the total number of computing node devices that execute the training task of the network model to be trained;

[0021] Based on the total number, the data blocks of the processes corresponding to each computing node device are divided into multiple sub-data blocks;

[0022] For each process executing the training task of the network model to be trained, in the current iteration, the current process performs a reduction operation on the data block received from the adjacent previous process and its own data in the previous iteration to obtain the reduced target data block.

[0023] If the computing power node device corresponding to the next process adjacent to the current process has the same computing power resource type as the computing power node device corresponding to the current process, the current process obtains the target communication library provided by the chip manufacturer of the computing power node device corresponding to the current process, and sends the target data block to the next process through the target communication library.

[0024] If the computing power node device corresponding to the next process adjacent to the current process is not the same as the computing power resource type corresponding to the computing power node device corresponding to the current process, the input data of the distributed communication interface is copied to the central processing unit, and the current process sends the target data block to the next process through the central processing unit;

[0025] Each sub-data block has the same number as the total number, and the number of processes has the same number as the total number of computing node devices.

[0026] In the fourth exemplary implementation, the unified encoding of each computing node device includes:

[0027] Each computing node device is classified according to its computing resource type, and the target computing resource type is selected as the coding benchmark.

[0028] Based on the numbering order of the computing node devices included in the target computing resource type, the computing node devices of other computing resource types are coded sequentially.

[0029] In the fifth exemplary embodiment, the process of distributing the network model to be trained to each computing node device for distributed training based on the encoding information of each computing node device includes:

[0030] After constructing the network model to be trained based on the target deep learning framework, obtain the computing power node devices that execute the training task of the network model to be trained;

[0031] Based on the encoding information of each computing node device, the training data of the network model to be trained is sent to the corresponding computing node device. Each computing node device calls a preset distributed training framework to perform memory allocation and operator calculation for the network model to be trained.

[0032] In a sixth exemplary embodiment, adapting the various types of computing power node devices of the heterogeneous computing platform to the target deep learning framework includes:

[0033] When the target computing power node device of the heterogeneous computing platform needs to be adapted to the target deep learning framework, the target deep learning framework is modified based on the name and runtime interface of the target computing power node device. The target deep learning framework is used as the front end, and the underlying software toolkit of the target computing power node device is used as the back end, so as to enable the target deep learning framework to support the target computing power node device through the front end and back end separation method.

[0034] In the seventh exemplary embodiment, modifying the target deep learning framework based on the name and runtime interface of the target computing node device includes:

[0035] Based on the name and runtime interface of the target computing node device, the converter code and Python library code of the target deep learning framework are modified accordingly;

[0036] Based on the name and runtime interface of the target computing node device, the distributed training framework code corresponding to the network model to be trained is modified accordingly.

[0037] In the eighth exemplary embodiment, modifying the target deep learning framework based on the name and runtime interface of the target computing power node device, using the target deep learning framework as the front end and the underlying software toolkit of the target computing power node device as the back end, includes:

[0038] Add a virtual device as an adapter interface for the target computing power node device to the target deep learning framework;

[0039] After adding the virtual device, register the corresponding distributed communication interface in the target deep learning framework, and use the distributed communication interface to call the computing power task execution framework; the computing power task execution framework is used to train the network model to be trained.

[0040] The underlying software stack of the target computing power node device is abstracted and designed using the computing power task execution framework to obtain the abstracted computing power underlying software stack.

[0041] The underlying computing software stack is configured as the backend, and the target deep learning framework, which serves as the frontend, is separated from the abstracted underlying computing software stack, which serves as the backend.

[0042] In the ninth exemplary embodiment, after configuring the underlying computing power software stack as the backend and separating the target deep learning framework as the frontend from the abstracted underlying computing power software stack as the backend, the method further includes:

[0043] Obtain the first kernel configuration file corresponding to the target deep learning framework;

[0044] Based on preset kernel configuration file construction rules, a second kernel configuration file corresponding to the first kernel configuration file is constructed using the first kernel configuration file; wherein, the second kernel configuration file is a kernel configuration file containing kernel name, parameter type and target correspondence, and the target correspondence is the correspondence between the first kernel configuration file and the second kernel configuration file;

[0045] A parameter conversion module is constructed between the target deep learning framework and the computing power task execution framework, so that when parameter conversion is required, the parameter conversion module is called to complete the automatic conversion operation of the corresponding parameters to be converted.

[0046] The kernel configuration file parsing operation is triggered to parse the first kernel configuration file and the second kernel configuration file corresponding to the target deep learning framework, and obtain the corresponding kernel configuration file parsing results. When kernel registration and implementation are required, the automatic code generation operation is triggered to generate target code for kernel registration and kernel implementation based on the kernel configuration file parsing results and a preset type language program.

[0047] In the tenth exemplary embodiment, after configuring the underlying computing power software stack as the backend and separating the target deep learning framework as the frontend from the abstracted underlying computing power software stack as the backend, the method further includes:

[0048] Determine the target operator that needs to be registered;

[0049] If the backend operator supports the computational operation corresponding to the target operator, and the target operator is a random number operator, then the target operator is registered through the central processing unit.

[0050] Another aspect of the present invention provides a model data processing apparatus for a heterogeneous computing platform, comprising:

[0051] The heterogeneous computing power adaptation module is used to adapt multiple types of computing power node devices of a heterogeneous computing platform to the target deep learning framework, so that each computing power node device of the heterogeneous computing platform can use the target deep learning framework to jointly train the network model to be trained.

[0052] The heterogeneous computing power configuration module is used to uniformly encode each computing power node device and determine the corresponding target communication interface in the distributed communication interface based on the computing power resource type of each computing power node device.

[0053] The data processing module is used to transmit model data between computing nodes through corresponding target communication interfaces during the distributed training process of distributing the network model to be trained to each computing node device based on the encoding information of each computing node device.

[0054] The present invention also provides an electronic device including a processor, the processor being configured to perform the steps of a model data processing method for a heterogeneous computing platform as described in any of the preceding claims when executing a computer program stored in a memory.

[0055] The present invention also provides a non-volatile storage medium storing a computer program, which, when executed by a processor, implements the steps of the model data processing method for the heterogeneous computing platform as described in any of the preceding claims.

[0056] Finally, the present invention also provides a heterogeneous computing platform, including multiple types of computing power node devices and processors, wherein the processors are connected to all types of computing power node devices;

[0057] When the processor executes a computer program stored in memory, it implements the steps of the model data processing method for the heterogeneous computing platform as described in any of the preceding claims.

[0058] The advantage of the technical solution provided by this invention lies in adapting various types of computing power node devices of heterogeneous computing platforms to the deep learning framework. This allows these different computing power node devices to use the same deep learning framework. Furthermore, various computing power node devices are uniformly coded, and communication interfaces are implemented in a distributed communication interface based on the computing power resource type corresponding to each node device. Thus, when the heterogeneous computing platform distributes the training task of the neural network model to each computing power node device, it can directly call various types of computing power node devices to execute the training task. The computing power node devices can communicate with each other through the distributed communication interface, thereby jointly completing the training of the neural network model. The more computing power node devices participating in the training task on the heterogeneous computing platform, the larger the scale of the supported neural network model, enabling the training of larger-scale neural network models without modifying the hardware.

[0059] Furthermore, the present invention also provides corresponding implementation devices, electronic devices, non-volatile storage media, and heterogeneous computing platforms for the model data processing method of heterogeneous computing platforms, further making the method more practical. The devices, electronic devices, non-volatile storage media, and heterogeneous computing platforms have corresponding advantages.

[0060] The technical features mentioned above, those to be mentioned below, and those shown individually in the accompanying drawings can be arbitrarily combined with each other, provided that the combined technical features are not contradictory. All feasible combinations of features are the technical content explicitly described herein. Any one of the multiple sub-features contained in the same statement can be applied independently, without necessarily being applied together with other sub-features. It should be understood that the above general description and the following detailed description are merely exemplary and do not limit the invention. Attached Figure Description

[0061] To more clearly illustrate the technical solutions of the present invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 A flowchart illustrating a model data processing method for a heterogeneous computing platform provided by the present invention;

[0063] Figure 2 A flowchart illustrating another model data processing method for a heterogeneous computing platform provided by the present invention;

[0064] Figure 3 A schematic diagram illustrating the distributed communication implementation process under an exemplary embodiment of the present invention;

[0065] Figure 4 A schematic diagram of a distributed communication structure under an exemplary embodiment of the present invention;

[0066] Figure 5 A structural diagram of an exemplary embodiment of the model data processing device for the heterogeneous computing platform provided by the present invention;

[0067] Figure 6 A structural diagram of an exemplary embodiment of the electronic device provided by the present invention;

[0068] Figure 7 This is a structural diagram of an exemplary embodiment of the heterogeneous computing platform provided by the present invention. Detailed Implementation

[0069] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. In this specification and the aforementioned drawings, the terms "first," "second," "third," "fourth," etc., are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. The term "exemplary" means "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior to or better than other embodiments.

[0070] With the continuous development of deep learning across various industries, the scale and performance of deep learning models have been greatly improved. Deep learning models, by constructing deep neural networks, can automatically learn feature representations of data, thus avoiding the tedious process of manually designing features. They are widely used in technologies such as image recognition, speech recognition, and natural language processing. With the development of AI (Artificial Intelligence) technology, neural network models exhibit characteristics such as massive data volume, model generalization, and centralized application patterns.

[0071] Currently, neural network models are becoming increasingly large, with a growing number of parameters, sometimes reaching billions. In training scenarios, the memory requirements of accelerator cards are more than four times the number of parameters, while current accelerator cards typically only have tens of gigabits of memory. The data parallelism methods of deep learning frameworks like PyTorch are no longer suitable for training large models. To meet the training needs of large-scale neural network models, the DeepSpeed ​​(Microsoft's distributed training framework) uses ZeRO (Zero Redundancy Optimization) to optimize memory usage in data parallelism through this distributed training strategy. In ZeRO data parallelism, the GPU (Graphics Processing Unit) containing the complete network only stores a portion of the parameters, gradients, and optimizer state information. This allows the parameters, gradients, and optimizer state information to be evenly distributed across multiple GPUs, effectively reducing the memory requirements of large-scale network models.

[0072] For DeepSpeed, its ZeRO data parallelism includes three types: one optimizes only the optimizer state, another optimizes both the optimizer state and gradients, and the third optimizes the optimizer state, gradients, and parameters. The optimization process for the optimizer state, gradients, and parameters is as follows:

[0073] For the forward computation process, in the initial stage, each GPU only stores the parameters of the weights / number of accelerator cards. For example, a neural network model has two layers and two GPUs, GPU1 and GPU2, participate in model training. GPU1 only stores the parameters w1 of the first layer, and GPU2 only stores the parameters w2 of the second layer. Before the forward computation of the first layer, GPU2 needs to distribute the parameters w1 once, and then perform the forward computation of the first layer. Similarly, before performing the forward computation of the second layer, the parameters w2 of the second layer need to be distributed once, and then the forward computation of the second layer is completed. For the data-parallel backpropagation process: before the backpropagation computation of the neural network model, the parameters are distributed first. After the backpropagation is completed, gradient collection is performed. For example, GPU2 needs to store the gradient g2 corresponding to w2, so all other GPUs send the g2 gradient to GPU2. After GPU2 receives the g2 gradients from other GPUs, it performs a reduction operation and saves them. Correspondingly, the other GPUs delete w2 and g2. The above process is repeated until all layers have completed the backpropagation computation. Regarding the parameter update process: Since gradient reduction is performed during backpropagation, ZeRO data parallelism can directly update the optimizer's state information and then update the parameters. After optimizing the optimizer state, gradients, and parameters, the memory usage of a large-scale neural network model is directly related to the number of accelerator cards involved in the computation; the more accelerator cards involved, the smaller the memory usage of a single accelerator card. Although the DeepSpeed ​​training framework can significantly reduce the memory usage of accelerator cards during the training of large models using ZeRO, this type of method is only applicable to a large class of accelerator cards. It requires that the accelerator cards involved in the computation be from the same manufacturer, or even the same generation of accelerator card products from the same manufacturer, such as the Ampere architecture and Hopper architecture GPUs provided by NVIDIA. This limits the scale of the neural network model.

[0074] Therefore, this invention adapts various types of computing power node devices to the deep learning framework and unifies the encoding of each computing power node device, enabling heterogeneous computing by different types of computing power nodes to complete the training of large-scale neural network models. After introducing the technical solution of this invention, various non-limiting embodiments are described in detail below. To better illustrate this invention, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this invention can be implemented without these specific details. In other examples, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of this invention.

[0075] Please see first. Figure 1 , Figure 1This is a flowchart illustrating a model data processing method for a heterogeneous computing platform provided in this embodiment. This embodiment may include the following:

[0076] S101: Adapt the various computing power node devices of the heterogeneous computing platform to the target deep learning framework so that the computing power node devices of the heterogeneous computing platform can use the target deep learning framework to jointly train the network model to be trained.

[0077] In this invention, the heterogeneous computing platform is a multi-type heterogeneous computing system, which deploys a group of computing power nodes with different computing performance to the same distributed computing system. It can be a single computing device such as a server with multiple heterogeneous accelerator cards installed, or a data center composed of multiple servers. The servers in the data center have built-in accelerator cards. The computing power node devices are the heterogeneous computing power chips, accelerator cards or computing cards built into each server of the heterogeneous computing platform, including but not limited to GPUs, FPGAs (Field Programmable Gate Arrays), NVIDIA computing accelerator cards, and Cambricon computing accelerator cards. Different types of heterogeneous computing power resources have different advantages in different application scenarios, such as artificial intelligence tasks. In order to meet the training requirements of neural network models, the heterogeneous computing platform includes a variety of computing power resources. Each computing power resource may include multiple computing power node devices. The computing power node devices can be high-concurrency, high-performance AI accelerator cards that can significantly shorten the training time of the model and improve the training efficiency, and are hardware devices used to accelerate the training performance of the network model to be trained.

[0078] The target deep learning framework can be any deep learning framework such as PaddlePaddle or PyTorch, without affecting the implementation of this invention. By adapting various types of computing node devices on a heterogeneous computing platform to the target deep learning framework, the underlying SDK (Software Development Kit) of each computing node device on the heterogeneous computing platform is decoupled from the target deep learning framework. This allows the computing node devices on the heterogeneous computing platform to share the target deep learning framework, i.e., use the same set of training code for the target deep learning framework.

[0079] S102: Perform unified coding on each computing node device, and determine the corresponding target communication interface in the distributed communication interface based on the computing resource type of each computing node device.

[0080] In this invention, the heterogeneous computing platform includes various computing resources. The original number of each computing node device is assigned within the range of its respective computing resource type. For example, the heterogeneous computing platform includes NVIDIA GPU1, NVIDIA GPU2, NVIDIA GPU3, Cambricon accelerator card 1, and Cambricon accelerator card 2. The original numbers of NVIDIA GPU1, NVIDIA GPU2, Cambricon accelerator card 1, and Cambricon accelerator card 2 are device_id (device identifier) ​​= 0, device_id = 1, device_id = 2, device_id = 0, and device_id = 1, respectively. This numbering information... Therefore, it becomes impossible to accurately locate which computing node device. To enable heterogeneous computing platforms to utilize these various types of computing node devices, each node device shares the same target deep learning framework code. This can be achieved by uniformly encoding the computing node devices. Uniform encoding refers to re-encoding based on the original numbering, allowing the heterogeneous computing platform to uniquely identify which computing node device it is when calling upon it. Since this invention does not modify the hardware, the underlying logic still uses the original numbering. Therefore, after uniform encoding, a correspondence needs to be established, meaning the new numbering after uniform encoding must correspond to the original numbering. For example, uniformly encoding a heterogeneous computing platform including NVIDIA GPU1, NVIDIA GPU2, NVIDIA GPU3, Cambricon accelerator card 1, and Cambricon accelerator card 2 can result in new numbers corresponding to device_id=0, device_id=1, device_id=2, device_id=3, and device_id=4. Understandably, in a distributed training model, each computing node shares model parameters during the training of the network model. Taking the Ring-allreduce architecture as an example, each computing node participating in model training is a worker, forming a ring. In one iteration, each worker completes training on its own mini-batch of sample data, calculates the gradient, and passes the gradient to the next worker in the ring. It also receives gradients from the previous worker. To enable communication between the computing nodes, after uniformly encoding each node, a corresponding communication interface needs to be determined for each node. For ease of description, this is defined as the target communication interface. This step determines which communication interface to use between computing nodes based on the computing resource type of each node.

[0081] S103: During the distributed training process of the network model to be trained, based on the encoding information of each computing node device, the computing node devices transmit model data through the corresponding target communication interface.

[0082] In this embodiment, the user issues a distributed training task to the heterogeneous computing platform. The heterogeneous computing platform can automatically determine the computing power node device that matches the distributed training task based on the training parameters of the network model to be trained, or it can determine the computing power node device that matches the distributed training task based on the computing power resources and the number of computing power node devices selected by the user. In this embodiment, each computing power node device includes at least the computing power node device selected to execute the distributed training task. The heterogeneous computing platform can uniquely identify the corresponding computing node device based on the uniformly encoded information. After constructing the network model to be trained based on the target deep learning framework, it obtains the computing node devices that will execute the training task of the network model. Based on the encoding information of each computing node device, the training data of the network model to be trained, such as training sample data and the network model to be trained, is sent to the corresponding computing node device. Each computing node device calls the preset distributed training framework to start memory allocation and operator computation for the network model to be trained. Deep learning algorithms are composed of individual computing units, which are called operators. In the network model to be trained, operators correspond to the computational logic in the layers. For example, a convolutional layer is an operator, the weight summation process in a fully connected layer is an operator, and the activation function is also an operator. After the operator computation is completed, it runs to the distributed communication interface. The computing node devices transmit model data through the corresponding target communication interface until the distributed training task is completed.

[0083] In the technical solution provided in this embodiment, various types of computing power node devices of a heterogeneous computing platform are adapted to the deep learning framework. This allows these different computing power node devices to use the same deep learning framework. Furthermore, various computing power node devices are uniformly coded, and communication interfaces are implemented in a distributed communication interface based on the computing power resource type corresponding to each node device. When the heterogeneous computing platform distributes the training task of the neural network model to each computing power node device, it can directly call various types of computing power node devices to execute the training task. The computing power node devices can communicate with each other through the distributed communication interface, thereby jointly completing the training of the neural network model. The more computing power node devices participating in the training task on the heterogeneous computing platform, the larger the scale of the supported neural network model, enabling the training of larger-scale neural network models without modifying the hardware.

[0084] In the above embodiments, there is no limitation on how to execute step S103. This embodiment provides an exemplary implementation method for communication between computing node devices through corresponding target communication interfaces, which may include the following:

[0085] When running to the distributed communication interface, the configuration information of the computing power node devices is obtained. If the computing power resource types of the first and second computing power node devices currently transmitting data are the same, then the first and second computing power node devices transmit data based on the corresponding distributed communication libraries. If the computing power resource types of the first and second computing power node devices currently transmitting data are different, then the first and second computing power node devices implement data transmission through a pre-defined method.

[0086] In this embodiment, the computing node device configuration information includes at least the computing node type to which each computing node device belongs. When running at the distributed communication interface, such as... Figure 2 As shown, the system determines whether the computing power node devices participating in the training task belong to the same type of computing power resource based on their configuration information. For example, if the first computing power node device is NVIDIA GPU board 1 and the second computing power node device is Cambricon board 1, then the computing power resource types of the first and second computing power node devices are different. Conversely, if the first computing power node device is NVIDIA GPU board 1 and the second computing power node device is NVIDIA GPU board 2, then the computing power resource types of the first and second computing power node devices are the same. Computing power node devices of the same type are implemented using a distributed communication library provided by the chip manufacturer. Computing power node devices of different types can communicate through a custom interface. For example, as a simple and efficient communication method, different types of computing power node devices can achieve distributed communication through the CPU (Central Processing Unit). Accordingly, the input data of the distributed communication interface is pre-copied to the central processing unit. Thus, when it is determined that the first and second computing power node devices belong to different computing power resource types, the first and second computing power node devices can transmit data through the central processing unit.

[0087] As can be seen from the above, after decoupling the underlying software toolkit of the computing power node device from the deep learning framework, this embodiment enables computing power node devices of the same type to communicate directly through the distributed communication library provided by the chip manufacturer, and computing power node devices of different types to communicate through the CPU distributed communication interface, so that different types of boards can perform heterogeneous computing, thereby realizing heterogeneous computing of large-scale models.

[0088] It should be noted that there is no strict order of execution between the steps in this invention. As long as they conform to the logical order, these steps can be executed simultaneously or in a certain preset order. Figures 1-2 This is just an illustrative example and does not mean that this is the only possible execution order.

[0089] In the above embodiments, there are no limitations on how to execute step S103. This embodiment adopts a ring protocol architecture based on the distributed communication interface, and also provides another exemplary implementation method for communication between computing node devices through corresponding target communication interfaces. This embodiment may include the following:

[0090] The process obtains the total number of computing node devices executing the training task of the network model to be trained. Based on the total number, the data blocks of the processes corresponding to each computing node device are divided into multiple sub-data blocks, with each sub-data block having the same number as the total number, and the number of processes being the same as the total number of computing node devices. For each process executing the training task of the network model to be trained, in the current iteration, the current process performs a reduction operation on the data block received from the adjacent previous process and its own data from the previous iteration to obtain the reduced target data block. If the computing node device corresponding to the next adjacent process of the current process has the same computing resource type as the computing node device corresponding to the current process, the process obtains the target communication library provided by the chip manufacturer of the computing node device corresponding to the current process, and the current process sends the target data block to the next process through the target communication library. If the computing node device corresponding to the next adjacent process of the current process has a different computing resource type than the computing node device corresponding to the current process, the input data of the distributed communication interface is copied to the central processing unit, and the current process sends the target data block to the next process through the central processing unit.

[0091] In this embodiment, each computing node device participating in the execution of the network model to be trained has one process. The data block of each process is divided into multiple sub-data blocks, and the total number of each sub-data block is the same as the total number of computing node devices participating in the execution of the network model to be trained. Combined with... Figure 3 and Figure 4For example, taking Ring-AllReduce as an example of a distributed communication interface, the heterogeneous computing platform includes two NVIDIA GPUs and two Cambricon accelerator cards, corresponding to processes 1, 2, 3, and 4. The unified encoding information is as follows: NVIDIA GPUs correspond to device_id = 0, 1; Cambricon accelerator cards correspond to device_id = 2, 3. Each process divides the array into four subarrays, where chunk represents a data block, and chunk[p] represents the data block of the p-th process. Then, process p performs a reduction operation on the received chunk[p-1] and its own chunk[p-1], and sends the reduced chunk to the next process p+1. When the next node in the ring is an accelerator card of the same type, the distributed communication interface provided by the chip manufacturer is used directly. For example, NVIDIA GPUs use nccl (NVIDIA Collective Communications Library, NVIDIA's high-performance distributed computing communication library) for communication, while Cambricon accelerator cards use cnnl (Cambricon Communications Library). When the next node in the ring is an accelerator card of a different type, the data to be communicated is copied from the device to the CPU, and communication is performed through CPU distributed communication. This process repeats the above three data receive-reduction-data send operations, with each process having a different part of the original array.

[0092] As can be seen from the above, this invention can realize heterogeneous computing with various types of accelerator cards for heterogeneous computing platforms using the Ring-AllReduce architecture, which is beneficial to increasing the scale of the network model to be trained.

[0093] The above embodiments do not impose any limitations on the unified coding of each computing power node device. Based on the above embodiments, the present invention also provides a simple coding method, which may include the following: classifying each computing power node device according to the computing power resource type, and selecting the target computing power resource type as the coding basis; and coding the computing power node devices of the other computing power resource types in sequence according to the numbering order of the computing power node devices included in the target computing power resource type.

[0094] In this embodiment, the unified coding involves sequentially numbering each computing node device. Correspondingly, the coding reference refers to the starting position of the number, or the computing node device type that does not change the original coding. For example, a computing node includes four NVIDIA GPU cards and four Cambricon accelerator cards. Using one NVIDIA card as the coding reference, after unified coding, the four NVIDIA GPU cards are numbered 0, 1, 2, and 3, while the four Cambricon cards change from their original numbers 0, 1, 2, and 3 to numbers 5, 6, 7, and 8.

[0095] The above embodiments do not limit how multiple types of computing power node devices are adapted to the target deep learning framework. This invention also provides an implementation method for adapting deep learning frameworks to computing power node devices on heterogeneous computing platforms, i.e., the target computing power node devices, which may include the following:

[0096] When the target computing power node device of the heterogeneous computing platform needs to be adapted to the target deep learning framework, the target deep learning framework is modified based on the name and runtime interface of the target computing power node device. The target deep learning framework is used as the front end, and the underlying software toolkit of the target computing power node device is used as the back end. This front-end and back-end separation approach enables the target deep learning framework to support the target computing power node device.

[0097] In this embodiment, adapting the target computing power node device to the target deep learning framework involves integrating and optimizing the target computing power node device and the target deep learning framework. This allows the target deep learning framework to better utilize the computing power of the target computing power node device, accelerating the training and inference process of the deep learning model. This embodiment achieves the adaptation between the two through front-end and back-end isolation. It is understood that, in order to support the target computing power node device and provide computational support, the target deep learning framework uses the same computing power node device name, adapts the operator implementation and runtime interface, and modifies the converter code and Python library code of the target deep learning framework accordingly based on the target computing power node device name and runtime interface. This ensures the training function of the network model to be trained is implemented. Furthermore, based on the target computing power node device name and runtime interface, the distributed training framework code corresponding to the network model to be trained is modified accordingly to complete the support for the operator implementation and runtime interface. Taking PyTorch as the target deep learning framework and DeepSpeed ​​as the distributed training framework as an example, it is necessary to modify the code in the transformers and accelerate libraries according to the newly added device names and runtime interfaces in PyTorch to ensure the functionality of large model training; and modify the DeepSpeed ​​code according to the newly added device names and runtime interfaces in PyTorch to complete the support for runtime and custom operators.

[0098] For example, a virtual device is added to the target deep learning framework as an adapter interface for the target computing power node device; the corresponding distributed communication interface is registered in the target deep learning framework after the virtual device is added, and the computing power task execution framework is called using the distributed communication interface; the underlying software stack of the target computing power node device is abstracted using the computing power task execution framework to obtain the abstracted underlying software stack of computing power; the underlying software stack of computing power is configured as the backend, and the target deep learning framework as the frontend and the abstracted underlying software stack of computing power as the backend are separated.

[0099] The virtual device serves as the adaptation interface for the target computing power node device, while the computing power task execution framework is used to train the network model to be trained. The computing power task execution framework includes a unified Runtime interface, a unified kernel interface, and a unified distributed communication interface. The unified Runtime interface calls the target computing power node device's Runtime library, the unified kernel interface calls the target computing power node device's operator acceleration library, and the unified distributed communication interface calls the target computing power node device's distributed communication library. After adding the virtual device to the target deep learning framework, a distributed communication interface is registered in the target deep learning framework with the added virtual device, and then the computing power task execution framework is called using the distributed communication interface. In essence, the corresponding virtual distributed communication interface is registered in the target deep learning framework, and the computing power task execution framework is called using the distributed communication interface. The computing power task execution framework abstracts the underlying software stack of the target computing power node devices provided by various vendors. This not only shields the differences in the underlying software stacks of different target computing power node devices but also allows the actual target computing power node device to be run based on the computing power task execution framework. By separating the target deep learning framework content as the front end and the content related to the underlying software stack of the target computing power node device as the back end, the target computing power node device manufacturers do not need to pay attention to the implementation and version updates of the target deep learning framework. This simplifies the work of adapting the target deep learning framework to the target computing power node device. Furthermore, by separating the front end and back end, hardware-related attributes can be adjusted to the computing power task execution framework. As a result, the kernel registration and implementation of the target deep learning framework no longer depend on the specific underlying software stack of the target computing power node device, and code generation can be achieved automatically.

[0100] Taking PyTorch as the target deep learning framework as an example, the PyTorch framework includes Device, kernel, Runtime interface, and distributed communication interface. It implements Device registration through patching, and implements the Runtime interface, kernel, and distributed communication interface through a framework extension mechanism. Kernel registration and implementation enable automatic code generation. The PyTorch framework adaptation development process is as follows: Device registration is developed within the PyTorch framework source code, and a kernel configuration file is built to achieve automatic kernel code generation, kernel registration, and implementation. Then, Runtime content is developed to support the PyTorch Runtime, followed by the development of the distributed communication interface. Finally, a network model is built for training, achieving faster model training and inference speeds by adapting to the target computing power node device. Correspondingly, the target computing power node device adaptation method may include: determining the Runtime interface corresponding to the general parallel computing architecture runtime in the target deep learning framework, and redirecting the Runtime interface corresponding to the general parallel computing architecture runtime in the target deep learning framework to the Runtime interface corresponding to the backend runtime. By reusing the CUDA (Compute Unified Device Architecture) related interfaces in the Python API (Application Programming Interface) of the PyTorch framework, it integrates into the PyTorch GPU development ecosystem and enables switching between target computing nodes via the backend. This eliminates the need to modify the source code of third-party software stacks developed based on the PyTorch framework, such as torchvision (software library name) and transformers, significantly improving the universality and ease of use for adapting target computing nodes to the PyTorch framework.

[0101] To achieve accurate adaptation, kernel registration and implementation are also required, which may include the following: obtaining a first kernel configuration file corresponding to the target deep learning framework; constructing a second kernel configuration file corresponding to the first kernel configuration file based on preset kernel configuration file construction rules; wherein the second kernel configuration file contains the kernel name, parameter types, and target correspondence, and the target correspondence is the correspondence between the first and second kernel configuration files, and the second kernel configuration file contains the kernel name, parameter types, and the correspondence between the first and second kernel configuration files. A parameter conversion module for the target deep learning framework and the computing task execution framework is constructed, so that when parameter conversion is needed, the corresponding parameters to be converted can be automatically converted by calling the parameter conversion module. For example, a parameter conversion module for the PyTorch framework and the computing task execution framework is constructed to achieve automatic parameter conversion. A kernel configuration file parsing operation is performed to parse the first and second kernel configuration files corresponding to the target deep learning framework, obtain the corresponding kernel configuration file parsing results, and trigger an automatic code generation operation when kernel registration and implementation are needed, generating target code for kernel registration and kernel implementation based on the kernel configuration file parsing results and preset type language programs.

[0102] In addition, operator registration is required during the adaptation process. When registering operators, it is first determined whether the backend operator supports the corresponding operation. If the backend operator supports the corresponding operation, it means that there is a corresponding implementation in the backend operator and operator registration can be performed. In order to avoid inconsistent random weights generated by different boards, which may lead to abnormal gradient updates, random number operators can be implemented using the CPU, that is, the target operator to be registered is determined. If the backend operator supports the calculation operation corresponding to the target operator, and the target operator is a random number operator, then the target operator is registered through the central processing unit.

[0103] To enable those skilled in the art to more clearly understand the adaptation process of this invention, this invention also provides an exemplary implementation method, taking PyTorch as the target deep learning framework and Cambricon MLU-370 as the target computing node device, which may include the following:

[0104] When the Cambricon MLU-370 is added to the PyTorch framework, the underlying software stack SDK-1.15.0 is adapted to the Cambricon MLU-370. That is, when the Cambricon MLU-370 accelerator card needs to be adapted to the PyTorch framework, the computing task execution framework is invoked. Following the computing task execution framework adaptation method in this embodiment, the Cambricon MLU-370 accelerator card is adapted to the PyTorch framework. This allows the PyTorch framework adapted to the Cambricon MLU-370 accelerator card to accelerate the training and inference speed of neural network models. Specifically, the unified Runtime interface provided in the computing task execution framework enables the invocation of the Runtime library CNRT (Compute Network Runtime). The Runtime interface represents the interface used to interact with a specific hardware accelerator. The high-performance operator library CNNL (Convolutional Neural Network) is implemented through the unified kernel interface provided in the computing task execution framework. The library (convolutional neural network library), CNNL_Extra, or custom-developed BANGC (an intelligent programming language) operators can be called. High-performance communication libraries can also be called through the unified distributed communication interface provided in the computing power task execution framework. Finally, the functions of the Runtime interface, kernel interface, and distributed communication interface can be verified based on the corresponding call results. Then, the PyTorch framework adapted to Cambricon MLU-370 can be used to accelerate the training of neural network models, such as using Cambricon MLU-370 to accelerate the training of ResNet50 (a deep learning network model structure).

[0105] As shown above, this embodiment separates the front-end and back-end by using the target deep learning framework content as the front-end and the content related to the underlying software stack of the target computing power node device as the back-end. This allows the target computing power node device manufacturer to avoid focusing on the implementation and version updates of the PyTorch framework, thus avoiding modifications to the PyTorch framework source code. This simplifies the work of adapting PyTorch to AI accelerator cards and improves the ease of use and versatility of AI accelerator cards to the PyTorch framework. Based on the front-end and back-end separation approach, automatic code generation for kernel registration in the PyTorch framework is achieved, thereby simplifying the development workload of kernel registration in the PyTorch framework and enabling rapid adaptation to PyTorch framework version updates. Furthermore, by implementing a random number operator through the CPU, inconsistent random weights generated by different computing power node devices are avoided, which could lead to abnormal gradient updates.

[0106] Furthermore, to ensure the heterogeneous computing platform efficiently completes the training task of the network model to be trained, a thread can be built to monitor whether memory usage is continuously increasing, i.e., whether memory leaks are occurring, during the training process. If a continuous increase in memory is detected, the cause of the memory leak can be determined first. For example, by checking the network structure and the code during training, it can be determined whether there is an unreasonable network structure design or errors in the training process. If the network structure is unreasonable, the number of neural network layers and nodes can be reduced, or more efficient network layers can be used, such as using 1x1 convolutions instead of fully connected layers to reduce memory usage. If the continuous increase in memory is caused by the training process, the optimizer and learning rate can be readjusted to make the model converge better and reduce memory usage. In addition, the `fit_generator` function of Keras (an open-source artificial neural network library) can be used to generate data in batches and release memory after each batch, limiting memory usage during training and reducing memory consumption. Furthermore, the training data scale of each computing node device can be further divided into multiple batches to further reduce memory usage and improve training efficiency.

[0107] This invention also provides a corresponding apparatus for the model data processing method of heterogeneous computing platforms, further enhancing the practicality of the method. The apparatus can be described from both a functional module perspective and a hardware perspective. The following describes the model data processing apparatus for heterogeneous computing platforms provided by this invention. This apparatus is used to implement the model data processing method for heterogeneous computing platforms provided by this invention. In this embodiment, the model data processing apparatus for heterogeneous computing platforms may include or be divided into one or more program modules. These one or more program modules are stored in a storage medium and executed by one or more processors to complete the model data processing method for heterogeneous computing platforms disclosed in Embodiment 1. The program module referred to in this embodiment refers to a series of computer program instruction segments capable of performing specific functions, which are more suitable than the program itself for describing the execution process of the model data processing apparatus for heterogeneous computing platforms in the storage medium. The following description will specifically introduce the functions of each program module in this embodiment. The model data processing apparatus for heterogeneous computing platforms described below can be referred to in correspondence with the model data processing method for heterogeneous computing platforms described above.

[0108] From the perspective of functional modules, see Figure 5 , Figure 5 This is a structural diagram of the model data processing device for the heterogeneous computing platform provided in this embodiment, in one specific implementation. The device may include:

[0109] The heterogeneous computing power adaptation module 501 is used to adapt multiple types of computing power node devices of heterogeneous computing platforms to the target deep learning framework, so that each computing power node device of the heterogeneous computing platform can use the target deep learning framework to train the network model to be trained.

[0110] The heterogeneous computing power configuration module 502 is used to uniformly encode each computing power node device and determine the corresponding target communication interface in the distributed communication interface based on the computing power resource type of each computing power node device.

[0111] The data processing module 503 is used to transmit model data between computing nodes through corresponding target communication interfaces during the distributed training process of distributing the network model to be trained to each computing node device based on the encoding information of each computing node device.

[0112] For example, in some embodiments of this example, the data processing module 503 described above can also be used for:

[0113] When running at the distributed communication interface, obtain the configuration information of the computing node devices;

[0114] If the first and second computing power node devices currently transmitting data have the same computing power resource type, then the first and second computing power node devices transmit data based on the corresponding distributed communication library.

[0115] If the computing power resource types of the first computing power node device and the second computing power node device currently transmitting data are different, the first computing power node device and the second computing power node device will implement data transmission through a pre-defined method.

[0116] As an exemplary implementation of the above embodiments, the data processing module 503 may further be used for:

[0117] The input data of the distributed communication interface is copied to the central processing unit in advance;

[0118] The first and second computing node devices transmit data through the central processing unit.

[0119] For example, in some other embodiments of this example, the data processing module 503 described above can also be used for:

[0120] The distributed communication interface adopts a ring-shaped protocol architecture to obtain the total number of computing node devices that execute the training task of the network model to be trained;

[0121] Based on the total number, the data blocks of the processes corresponding to each computing node device are divided into multiple sub-data blocks;

[0122] For each process executing the training task of the network model to be trained, in the current iteration, the current process performs a reduction operation on the data block received from the adjacent previous process and its own data in the previous iteration to obtain the reduced target data block.

[0123] If the computing power node device of the next process adjacent to the current process has the same computing power resource type as the computing power node device of the current process, the current process obtains the target communication library provided by the chip manufacturer of the computing power node device of the current process, and sends the target data block to the next process through the target communication library;

[0124] If the computing power node device corresponding to the next process adjacent to the current process is different from the computing power resource type corresponding to the computing power node device corresponding to the current process, the input data of the distributed communication interface is copied to the central processing unit, and the current process sends the target data block to the next process through the central processing unit;

[0125] The number of sub-data blocks is the same as the total number, and the number of processes is the same as the total number of computing node devices.

[0126] For example, in some other embodiments of this example, the heterogeneous computing power configuration module 502 described above can also be used for:

[0127] Each computing node device is classified according to its computing resource type, and the target computing resource type is selected as the coding benchmark.

[0128] Based on the numbering order of the computing node devices included in the target computing resource type, the computing node devices of other computing resource types are coded sequentially.

[0129] For example, in some other embodiments of this example, the data processing module 503 described above can also be used for:

[0130] After constructing the network model to be trained based on the target deep learning framework, obtain the computing power node devices that execute the training task of the network model to be trained;

[0131] Based on the encoding information of each computing node device, the training data of the network model to be trained is sent to the corresponding computing node device. Each computing node device calls the preset distributed training framework to allocate memory and perform operator calculations on the network model to be trained.

[0132] For example, in some other embodiments of this example, the heterogeneous computing power adaptation module 501 described above can also be used for:

[0133] When the target computing power node device of the heterogeneous computing platform needs to be adapted to the target deep learning framework, the target deep learning framework is modified based on the name and runtime interface of the target computing power node device. The target deep learning framework is used as the front end, and the underlying software toolkit of the target computing power node device is used as the back end. This front-end and back-end separation approach enables the target deep learning framework to support the target computing power node device.

[0134] As an exemplary implementation of the above embodiments, the heterogeneous computing power adaptation module 501 can be further used for:

[0135] Based on the name and runtime interface of the target computing power node device, the converter code and Python library code of the target deep learning framework are modified accordingly;

[0136] Based on the name and runtime interface of the target computing node device, the distributed training framework code corresponding to the network model to be trained is modified accordingly.

[0137] As another exemplary implementation of the above embodiments, the heterogeneous computing power adaptation module 501 can be further used for:

[0138] Add a virtual device as an adapter interface for the target computing power node device in the target deep learning framework;

[0139] After adding the virtual device, register the corresponding distributed communication interface in the target deep learning framework, and use the distributed communication interface to call the computing power task execution framework; the computing power task execution framework is used to train the network model to be trained.

[0140] The underlying software stack of the target computing power node device is abstracted and designed using a computing power task execution framework to obtain the abstracted computing power underlying software stack;

[0141] Configure the underlying computing software stack as the backend, and separate the target deep learning framework as the frontend from the abstracted underlying computing software stack as the backend.

[0142] As an exemplary implementation of the above embodiments, the heterogeneous computing power adaptation module 501 can be further used for:

[0143] Obtain the first kernel configuration file corresponding to the target deep learning framework;

[0144] Based on the preset kernel configuration file construction rules, a second kernel configuration file corresponding to the first kernel configuration file is constructed using the first kernel configuration file; wherein, the second kernel configuration file is a kernel configuration file containing the kernel name, parameter types and target correspondence, and the target correspondence is the correspondence between the first kernel configuration file and the second kernel configuration file;

[0145] A parameter conversion module is constructed between the target deep learning framework and the computing power task execution framework, so that when parameter conversion is required, the corresponding automatic conversion operation of the parameters to be converted can be completed by calling the parameter conversion module;

[0146] The kernel configuration file parsing operation is triggered to parse the first and second kernel configuration files corresponding to the target deep learning framework, obtain the corresponding kernel configuration file parsing results, and when kernel registration and implementation are required, the automatic code generation operation is triggered to generate target code for kernel registration and kernel implementation based on the kernel configuration file parsing results and preset type language programs.

[0147] As an exemplary implementation of the above embodiments, the heterogeneous computing power adaptation module 501 can be further used for:

[0148] Determine the target operator that needs to be registered;

[0149] If the backend operator supports the computational operation corresponding to the target operator, and the target operator is a random number operator, then the target operator is registered through the central processing unit.

[0150] The functions of each functional module of the model data processing device of the heterogeneous computing platform in this embodiment can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0151] As can be seen from the above, this embodiment enables different types of computing power node devices to jointly complete the training of neural network models, and can train larger-scale neural network models.

[0152] The model data processing device for the heterogeneous computing platform mentioned above is described from the perspective of functional modules. Furthermore, the present invention also provides an electronic device, which is described from the perspective of hardware. Figure 6 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. For example... Figure 6 As shown, the electronic device includes a memory 60 for storing a computer program; and a processor 61 for executing the computer program to implement the steps of the model data processing method for the heterogeneous computing platform as described in any of the above embodiments.

[0153] The processor 61 may include one or more processing cores, such as a 6-core processor or an 8-core processor. The processor 61 may also be a controller, microcontroller, microprocessor, or other data processing chip. The processor 61 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 61 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 61 may integrate a GPU, which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 61 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0154] The memory 60 may include one or more computer non-volatile storage media, which may be non-transitory. The memory 60 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the memory 60 may be an internal storage unit of an electronic device, such as a server hard drive. In other embodiments, the memory 60 may be an external storage device of an electronic device, such as a plug-in hard drive on a server, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Furthermore, the memory 60 may include both internal and external storage units of the electronic device. The memory 60 can be used not only to store application software and various types of data installed on the electronic device, such as code in the process of executing model data processing methods on a heterogeneous computing platform, but also to temporarily store data that has been output or will be output. In this embodiment, the memory 60 is used to store at least the following computer program 601, which, after being loaded and executed by the processor 61, is capable of implementing the relevant steps of the model data processing method for the heterogeneous computing platform disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, and the storage method may be temporary storage or permanent storage. The operating system 602 may include Windows, Unix, Linux, etc. The data 603 may include, but is not limited to, data corresponding to the model data processing results of the heterogeneous computing platform.

[0155] In some embodiments, the aforementioned electronic device may further include a display screen 62, an input / output interface 63, a communication interface 64 (or network interface), a power supply 65, and a communication bus 66. The display screen 62 and input / output interface 63, such as a keyboard, are user interfaces. Exemplary user interfaces may also include standard wired interfaces, wireless interfaces, etc. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a display screen or display unit, used to display information processed in the electronic device and to display a visual user interface. The communication interface 64 may exemplary include wired and / or wireless interfaces, such as a Wi-Fi interface, a Bluetooth interface, etc., typically used to establish communication connections between the electronic device and other electronic devices. The communication bus 66 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0156] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, such as sensors 67 that perform various functions.

[0157] The functions of each functional module of the electronic device described in this embodiment can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0158] As can be seen from the above, this embodiment enables different types of computing power node devices to jointly complete the training of neural network models, and can train larger-scale neural network models.

[0159] It is understood that if the model data processing of the heterogeneous computing platform in the above embodiments is implemented as a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the related technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned storage medium includes, but is not limited to, various media capable of storing program code, such as: USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, removable disk, CD-ROM, magnetic disk, or optical disk. Based on this, the present invention also provides a non-volatile storage medium storing a computer program, which, when executed by a processor, performs the steps of model data processing of the heterogeneous computing platform as described in any of the above embodiments.

[0160] It is understood that if the model data processing of the heterogeneous computing platform in the above embodiments is implemented as a software functional unit and sold or used as an independent product, the computer software product may not need to be stored in a physical storage medium. For example, it can be directly transmitted to a computer or other device with information processing capabilities via a wired or wireless network to execute all or part of the steps of the methods in the various embodiments of the present invention. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the related technology, or all or part of the technical solution, can be embodied in the form of a software product. Based on this, the present invention also provides a computer program product storing a computer program, which, when executed by a processor, performs the steps of model data processing of the heterogeneous computing platform as described in any of the above embodiments.

[0161] This invention also provides a heterogeneous computing platform; please refer to [link / reference]. Figure 7 It may include:

[0162] The heterogeneous computing platform includes multiple types of computing node devices 70 and a processor 61. The processor 61 is connected to all types of computing node devices. When executing the computer program stored in the memory 60, the processor 61 implements the steps of the model data processing method of the heterogeneous computing platform described in any of the above embodiments. The types of computing resources and the number of computing node devices included in the heterogeneous computing platform can be flexibly selected according to actual conditions. The processor 61 and each computing node device 70 can be connected through any communication method, such as wired connection or remote connection. The processor 61 can be deployed on a server node in the heterogeneous computing platform, which can be the node with the highest computing power performance among all computing devices in the heterogeneous computing platform. As an exemplary implementation, the processor 61 can also be deployed on any server outside the heterogeneous computing platform, and the server communicates with each computing node device of the heterogeneous computing platform.

[0163] The functions of each functional module of the heterogeneous computing platform system described in the embodiments of the present invention can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0164] As can be seen from the above, this embodiment enables different types of computing power node devices to jointly complete the training of neural network models, and can train larger-scale neural network models.

[0165] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the hardware disclosed in the embodiments, including devices and electronic equipment, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0166] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0167] The foregoing has provided a detailed description of the heterogeneous computing platform and its model data processing method, apparatus, electronic device, non-volatile storage medium, and heterogeneous computing platform provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that, based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Several improvements and modifications can be made to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A method for processing model data on a heterogeneous computing platform, characterized in that, include: The heterogeneous computing platform adapts various types of computing power node devices to the target deep learning framework, so that each computing power node device of the heterogeneous computing platform can use the target deep learning framework to jointly train the network model to be trained. Each computing node device is uniformly coded, and the corresponding target communication interface is determined in the distributed communication interface based on the computing resource type of each computing node device. During the distributed training process, based on the encoding information of each computing node device, the network model to be trained is distributed to each computing node device. The computing node devices transmit model data through the corresponding target communication interface. When running to the distributed communication interface, the configuration information of the computing node device is obtained. If the computing resource types of the first and second computing node devices currently transmitting data are the same, the first and second computing node devices transmit data based on the corresponding distributed communication library. If the computing power resource types of the first computing power node device and the second computing power node device currently transmitting data are different, then the first computing power node device and the second computing power node device will transmit data through the CPU distributed communication interface.

2. The model data processing method for a heterogeneous computing platform according to claim 1, characterized in that, The first computing node device and the second computing node device implement data transmission through a pre-defined method, including: The input data of the distributed communication interface is copied to the central processing unit in advance; The first computing node device and the second computing node device transmit data through the central processing unit.

3. The model data processing method for a heterogeneous computing platform according to claim 1, characterized in that, The distributed communication interface adopts a ring protocol architecture, and the various computing node devices transmit model data through corresponding target communication interfaces, including: Obtain the total number of computing node devices that execute the training task of the network model to be trained; Based on the total number, the data blocks of the processes corresponding to each computing node device are divided into multiple sub-data blocks; For each process executing the training task of the network model to be trained, in the current iteration, the current process performs a reduction operation on the data block received from the adjacent previous process and its own data in the previous iteration to obtain the reduced target data block. If the computing power node device corresponding to the next process adjacent to the current process has the same computing power resource type as the computing power node device corresponding to the current process, the current process obtains the target communication library provided by the chip manufacturer of the computing power node device corresponding to the current process, and sends the target data block to the next process through the target communication library. If the computing power node device corresponding to the next process adjacent to the current process is not the same as the computing power resource type corresponding to the computing power node device corresponding to the current process, the input data of the distributed communication interface is copied to the central processing unit, and the current process sends the target data block to the next process through the central processing unit; Each sub-data block has the same number as the total number, and the number of processes has the same number as the total number of computing node devices.

4. The model data processing method for a heterogeneous computing platform according to claim 1, characterized in that, The unified coding of each computing node device includes: Each computing node device is classified according to its computing resource type, and the target computing resource type is selected as the coding benchmark. Based on the numbering order of the computing node devices included in the target computing resource type, the computing node devices of other computing resource types are coded sequentially.

5. The model data processing method for a heterogeneous computing platform according to claim 1, characterized in that, The process of distributing the network model to be trained to each computing node device for distributed training based on the encoding information of each computing node device includes: After constructing the network model to be trained based on the target deep learning framework, obtain the computing power node devices that execute the training task of the network model to be trained; Based on the encoding information of each computing node device, the training data of the network model to be trained is sent to the corresponding computing node device. Each computing node device calls a preset distributed training framework to perform memory allocation and operator calculation for the network model to be trained.

6. The model data processing method for a heterogeneous computing platform according to any one of claims 1 to 5, characterized in that, The process of adapting various types of computing power node devices from heterogeneous computing platforms to the target deep learning framework includes: When the target computing power node device of the heterogeneous computing platform needs to be adapted to the target deep learning framework, the target deep learning framework is modified based on the name and runtime interface of the target computing power node device. The target deep learning framework is used as the front end, and the underlying software toolkit of the target computing power node device is used as the back end, so as to enable the target deep learning framework to support the target computing power node device through the front end and back end separation method.

7. The model data processing method for a heterogeneous computing platform according to claim 6, characterized in that, The modification of the target deep learning framework based on the name and runtime interface of the target computing node device includes: Based on the name and runtime interface of the target computing node device, the converter code and Python library code of the target deep learning framework are modified accordingly; Based on the name and runtime interface of the target computing node device, the distributed training framework code corresponding to the network model to be trained is modified accordingly.

8. The model data processing method for a heterogeneous computing platform according to claim 6, characterized in that, The modification of the target deep learning framework based on the name and runtime interface of the target computing power node device, using the target deep learning framework as the front end and the underlying software toolkit of the target computing power node device as the back end, includes: Add a virtual device as an adapter interface for the target computing power node device to the target deep learning framework; After adding the virtual device, register the corresponding distributed communication interface in the target deep learning framework, and use the distributed communication interface to call the computing power task execution framework; the computing power task execution framework is used to train the network model to be trained. The underlying software stack of the target computing power node device is abstracted and designed using the computing power task execution framework to obtain the abstracted computing power underlying software stack. The underlying computing software stack is configured as the backend, and the target deep learning framework, which serves as the frontend, is separated from the abstracted underlying computing software stack, which serves as the backend.

9. The model data processing method for a heterogeneous computing platform according to claim 8, characterized in that, After configuring the underlying computing power software stack as the backend and separating the target deep learning framework as the frontend from the abstracted underlying computing power software stack as the backend, the method further includes: Obtain the first kernel configuration file corresponding to the target deep learning framework; Based on preset kernel configuration file construction rules, a second kernel configuration file corresponding to the first kernel configuration file is constructed using the first kernel configuration file; wherein, the second kernel configuration file is a kernel configuration file containing kernel name, parameter type and target correspondence, and the target correspondence is the correspondence between the first kernel configuration file and the second kernel configuration file; A parameter conversion module is constructed between the target deep learning framework and the computing power task execution framework, so that when parameter conversion is required, the parameter conversion module is called to complete the automatic conversion operation of the corresponding parameters to be converted. The kernel configuration file parsing operation is triggered to parse the first kernel configuration file and the second kernel configuration file corresponding to the target deep learning framework, and obtain the corresponding kernel configuration file parsing results. When kernel registration and implementation are required, the automatic code generation operation is triggered to generate target code for kernel registration and kernel implementation based on the kernel configuration file parsing results and a preset type language program.

10. The model data processing method for a heterogeneous computing platform according to claim 8, characterized in that, After configuring the underlying computing power software stack as the backend and separating the target deep learning framework as the frontend from the abstracted underlying computing power software stack as the backend, the method further includes: Determine the target operator that needs to be registered; If the backend operator supports the computational operation corresponding to the target operator, and the target operator is a random number operator, then the target operator is registered through the central processing unit.

11. A model data processing device for a heterogeneous computing platform, characterized in that, include: The heterogeneous computing power adaptation module is used to adapt multiple types of computing power node devices of a heterogeneous computing platform to the target deep learning framework, so that each computing power node device of the heterogeneous computing platform can use the target deep learning framework to jointly train the network model to be trained. The heterogeneous computing power configuration module is used to uniformly encode each computing power node device and determine the corresponding target communication interface in the distributed communication interface based on the computing power resource type of each computing power node device. The data processing module is used to transmit model data between computing power nodes through corresponding target communication interfaces during the distributed training process of distributing the network model to be trained to each computing power node device based on the encoding information of each computing power node device. When running to the distributed communication interface, the module obtains the configuration information of the computing power node devices. If the computing power resource types of the first and second computing power node devices currently transmitting data are the same, then the first and second computing power node devices transmit data based on the corresponding distributed communication libraries. If the computing power resource types of the first computing power node device and the second computing power node device currently transmitting data are different, then the first computing power node device and the second computing power node device will transmit data through the CPU distributed communication interface.

12. An electronic device, characterized in that, It includes a processor and a memory, wherein the processor is used to implement the steps of the model data processing method of the heterogeneous computing platform as described in any one of claims 1 to 10 when executing a computer program stored in the memory.

13. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a computer program, which, when executed by a processor, implements the steps of the model data processing method for the heterogeneous computing platform as described in any one of claims 1 to 10.

14. A heterogeneous computing platform, characterized in that, It includes various types of computing power node devices and processors, and the processors are connected to all types of computing power node devices; When the processor executes a computer program stored in the memory, it implements the steps of the model data processing method for the heterogeneous computing platform as described in any one of claims 1 to 10.

Images