Data processing method and related product

Abstract

The invention provides a data processing method and related products, the method is applied to an accelerator cluster, the accelerator cluster comprises a first accelerator, a first data processing unit, a second data processing unit and a second accelerator, the first data processing unit is connected with the second data processing unit, and the first accelerator and the second accelerator are different in type. The first accelerator and the second accelerator are respectively used for operating the Nth layer and the (N + 1) th layer of the same model, the first data processing unit obtains a target value based on a first value after obtaining the first value calculated by operating the Nth layer of the first accelerator, and sends the target value to the second accelerator through the second data processing unit, communication between the first accelerator and the second accelerator is achieved, a foundation is laid for effectively utilizing computing power of various accelerators, the heterogeneous accelerators can operate a complete model, and therefore function improvement of the accelerator cluster is achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a data processing method and related products. Background Art

[0002] As the scale of neural network models and training data sets gradually increase, tasks that a single accelerator cannot handle need to be completed by accelerator clusters. For example, a single accelerator can no longer run a complete neural network model independently, and an accelerator cluster consisting of multiple accelerators is required to run large neural network models.

[0003] However, multiple heterogeneous accelerators cannot run a complete neural network model. Summary of the invention

[0004] In a first aspect, an embodiment of the present application provides a data processing method, characterized in that it is applied to an accelerator cluster, the accelerator cluster includes a first accelerator, a first data processing unit, a second data processing unit, and a second accelerator, the first accelerator is connected to the first data processing unit, the second accelerator is connected to the second data processing unit, the first data processing unit is connected to the second data processing unit, the first accelerator and the second accelerator are of different types, the first accelerator and the second accelerator are respectively used to run the Nth layer and the N+1th layer of the same model, where N is a positive integer, and the method includes:

[0005] The first data processing unit obtains a first value in response to the first accelerator running the Nth layer calculation; the first data processing unit sends a target value to the second accelerator through the second data processing unit, and the target value is obtained based on the first value.

[0006] In the embodiment of the present application, a first data processing unit and a second data processing unit are introduced. After obtaining a first value calculated by a first accelerator, the first data processing unit obtains a target value based on the first value, and sends the target value to the second accelerator through the second data processing unit, thereby realizing communication between the first accelerator and the second accelerator, that is, realizing communication between heterogeneous accelerators, laying a foundation for effectively utilizing the computing power of multiple accelerators, so that heterogeneous accelerators can run a complete model.

[0007] In a possible implementation, after the first data processing unit obtains the first value in response to the first accelerator running the Nth layer calculation, the first value is further included: performing precision conversion on the first value to obtain the target value, wherein the precision of the target value is adapted to the precision of the second accelerator. Through precision conversion, the first value is directly converted into a data type adapted to the precision of the second accelerator, thereby achieving efficient data transmission and precision matching between different types of accelerators.

[0008] In a possible implementation, after the first data processing unit obtains the first value in response to the first accelerator running the Nth layer calculation, the first value is further included: performing precision conversion on the first value to obtain the target value, the target value is used to generate a second value, and the precision of the second value is adapted to the precision of the second accelerator. Through precision conversion, the first value is first converted into a target value with a unified and standardized data type to ensure that the data type of the target value is compatible between different accelerators, so as to facilitate the generation of a second value adapted to the precision of the second accelerator based on the target value.

[0009] In a possible implementation, the data processing method further includes: the first data processing unit receives a third value through the second data processing unit, the third value is obtained based on a fourth value, and the fourth value is obtained by the second accelerator running the N+1th layer calculation; the first data processing unit sends the third value to the first accelerator. In this way, the value calculated by the first accelerator can be sent to the second accelerator, and the first accelerator can also receive the value calculated by the second accelerator, so that the heterogeneous accelerator can run a complete model for forward calculation and backward propagation, and effectively utilize the computing power of multiple accelerators.

[0010] In another possible implementation, the data processing method further includes: the first data processing unit responds to the initialization instruction, obtains the first configuration information by reading the first configuration file, and the first configuration information includes: the local network receiving monitoring IP and port, the network receiving IP and port of the second data processing unit; calls the preset network monitoring interface, monitors the local network receiving monitoring IP and port to complete the initialization of the local network receiving function, and waits for the network sending layer connection of the second data processing unit; calls the preset network connection interface, monitors the first peer network receiving IP and port, and establishes a network connection with the second data processing unit. In this way, by initialization, a network connection is established between the first data processing unit and the first peer downstream (the second data processing unit). If the accelerator cluster also includes a second peer upstream of the first data processing unit, a network connection can also be established between the first data processing unit and the second peer upstream through initialization, laying the foundation for communication between heterogeneous accelerators.

[0011] In yet another possible implementation, the first data processing unit is a CPU or a DPU.

[0012] In a second aspect, an embodiment of the present application provides a device transmission agent device, which is applied to a first data processing unit in an accelerator cluster, wherein the accelerator cluster further includes a first accelerator, a second data processing unit, and a second accelerator, wherein the first accelerator is connected to the first data processing unit, the second accelerator is connected to the second data processing unit, the first data processing unit is connected to the second data processing unit, the first accelerator and the second accelerator are of different types, and the first accelerator and the second accelerator are respectively used to run the Nth layer and the N+1th layer of the same model, where N is a positive integer; the device transmission agent device includes:

[0013] An accelerator data handling module, configured to obtain a first value in response to the first accelerator running the Nth layer calculation to obtain the first value;

[0014] A network sending module is used to send a target value to the second accelerator through the second data processing unit, and the target value is obtained based on the first value.

[0015] In a possible implementation manner, the device transmission agent apparatus further includes:

[0016] The precision conversion module is used to perform precision conversion on the first value to obtain the target value, and the precision of the target value is adapted to the precision of the second accelerator.

[0017] In a possible implementation manner, the device transmission agent apparatus further includes:

[0018] The precision conversion module is used to perform precision conversion on the first value to obtain the target value, and the target value is used to generate a second value, and the precision of the second value is adapted to the precision of the second accelerator.

[0019] In a possible implementation manner, the device transmission agent apparatus further includes:

[0020] a network receiving module, configured to receive a third value through the second data processing unit, where the third value is obtained based on a fourth value, where the fourth value is obtained by the second accelerator running the N+1th layer calculation;

[0021] The accelerator data handling module is further used to send the third value to the first accelerator.

[0022] In yet another possible implementation, the device transmission agent apparatus further includes:

[0023] An initialization module is used to respond to an initialization instruction and obtain first configuration information by reading a first configuration file, wherein the first configuration information includes: a local network receiving listening IP and port, and a network receiving IP and port of the second data processing unit; calling a preset network listening interface, listening to the local network receiving listening IP and port to complete the initialization of the local network receiving function, and waiting for the network sending layer of the second data processing unit to be connected; calling a preset network connection interface, listening to the network receiving IP and port of the second data processing unit, and establishing a network connection with the second data processing unit.

[0024] In a third aspect, an embodiment of the present application provides a data processing unit, wherein the data processing unit is a first data processing unit in an accelerator cluster, and the accelerator cluster further includes a first accelerator, a second data processing unit, and a second accelerator, wherein the first accelerator is connected to the first data processing unit, the second accelerator is connected to the second data processing unit, the first data processing unit is connected to the second data processing unit, the first accelerator and the second accelerator are of different types, and the first accelerator and the second accelerator are respectively used to run the Nth layer and the N+1th layer of the same model, where N is a positive integer;

[0025] The data processing unit is used for obtaining a first value in response to the first accelerator running the Nth layer calculation, acquiring the first value, and sending a target value to the second accelerator through the second data processing unit, the target value being obtained based on the first value.

[0026] In a fourth aspect, an embodiment of the present application provides a server, comprising a first accelerator and a first processing unit;

[0027] The first accelerator is connected to the first data processing unit, the first data processing unit is connected to the second data processing unit, the second data processing unit is connected to the second accelerator, the first accelerator and the second accelerator are of different types, and the first accelerator and the second accelerator are respectively used to run the Nth layer and the N+1th layer of the same model, where N is a positive integer;

[0028] The first data processing unit is used for obtaining a first value in response to the first accelerator running the Nth layer calculation, acquiring the first value, and sending a target value to the second accelerator through the second data processing unit, the target value being obtained based on the first value.

[0029] In a possible implementation, the server further includes a first network card, the first data processing unit and the second data processing unit are both CPUs, the first data processing unit is connected to the first network card, the first network card is communicatively connected to the second network card, and the second network card is connected to the second data processing unit;

[0030] Or the first data processing unit and the second data processing unit are both DPUs.

[0031] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer-readable instructions, which, when executed on a data processing unit in an accelerator cluster, enables the data processing unit to implement any one of the methods described in the first aspect.

[0032] In a sixth aspect, an embodiment of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by a data processing unit in an accelerator cluster, the data processing unit can implement any method described in the first aspect.

[0033] In the seventh aspect, an embodiment of the present application provides an accelerator cluster, comprising a first accelerator, a first data processing unit, a second data processing unit, and a second accelerator, the first accelerator being connected to the first data processing unit, the second accelerator being connected to the second data processing unit, the first data processing unit and the second data processing unit being connected, the first accelerator and the second accelerator being of different types, the first accelerator and the second accelerator being respectively used to run the Nth layer and the N+1th layer of the same model, wherein N is a positive integer, the first data processing unit obtaining a first value in response to the first accelerator running the Nth layer by calculating the first value, and obtaining the first value; the first data processing unit sending a target value to the second accelerator through the second data processing unit, the target value being obtained based on the first value. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work.

[0035] Figure 1 A schematic diagram of an accelerator cluster operation logic provided in an embodiment of the present application;

[0036] Figure 2 A schematic diagram of a data processing method system architecture provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of another data processing method system architecture provided in an embodiment of the present application;

[0038] Figure 4 A schematic diagram of another data processing method system architecture provided in an embodiment of the present application;

[0039] Figure 5 A schematic diagram of another data processing method system architecture provided in an embodiment of the present application;

[0040] Figure 6 A schematic diagram of an accelerator cluster architecture provided in an embodiment of the present application;

[0041] Figure 7 A timing diagram of a data processing method provided in an embodiment of the present application;

[0042] Figure 8 A schematic diagram of the software structure of a data processing unit provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0044] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0045] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0046] For the purpose of reference and clarity, the technical terms, abbreviations or acronyms used below are summarized as follows:

[0047] Heterogeneous accelerator cluster: A computing cluster composed of different types of accelerator devices. These accelerators include but are not limited to graphics processing units (GPUs) and Moore's Law accelerators. Each accelerator has its own unique architecture and optimization features, and together they form a diverse computing environment.

[0048] Neural network model: A computational model that simulates the working principle of neurons in the human brain and is used to process and learn complex data patterns. This model consists of multiple layers of interconnected neurons (or nodes), each of which can perform simple mathematical operations such as weighted summation and nonlinear transformations. Neural network models can learn the mapping relationship from input data to output results by passing information between these neuron layers and adjusting the connection weights using algorithms such as backpropagation. These models are widely used in the fields of machine learning and artificial intelligence, especially in tasks such as image recognition, speech processing, and natural language processing.

[0049] Distributed training: The process of training AI models using distributed computing resources. This training method allows large-scale data sets and complex computing tasks to be distributed across multiple computing nodes and accelerates the learning process, thereby improving overall training efficiency and model performance.

[0050] Forward Propagation: Input data is passed through the layers of the model, from the input layer to the hidden layer and finally to the output layer. Each layer applies specific transformations, such as linear transformations (matrix multiplication) and nonlinear activation functions (such as ReLU, sigmoid, etc.). The result of the output layer is the model's prediction of the input data.

[0051] Loss Computation: Use a loss function (such as mean squared error, cross entropy, etc.) to calculate the difference between the model prediction and the true label. The value of the loss function represents the error of the model on the current input.

[0052] Backpropagation: This is a computational process used to update the weights and biases in the model in order to reduce the loss. Starting from the output layer, the gradients of each layer (i.e., the partial derivatives of the loss function with respect to each weight) are calculated backwards. This process involves the chain rule, which uses the gradients of the previous layers to calculate the gradients of the subsequent layers.

[0053] Optimizer: The optimizer uses the gradients calculated by backpropagation to update the parameters (weights and biases) of the model.

[0054] AI training process: input data -> [forward calculation] -> output prediction -> [loss calculation] -> [backward propagation] -> [parameter update] -> repeat -> ... -> training completed. The whole process will be repeated in multiple training iterations until the performance of the model reaches a satisfactory level or no longer improves significantly.

[0055] Activation value: refers to the data transmitted and processed during the calculation process of the neural network. Specifically, when the input data is forward propagated through the layers of the neural network model, the neurons in each layer will weight the received signals and generate output through an activation function. This output signal is the activation value of the neuron. These activation values ​​are then used as inputs to the neurons in the next layer until the output of the network is finally generated. Activation values ​​are basic elements in the learning and decision-making process of neural networks. They reflect the network's response and processing status to the input data.

[0056] Gradient: In the context of neural network training, gradient refers specifically to the partial derivatives of the loss function with respect to the model parameters. These gradients indicate the local changes in the loss function in the parameter space, thus guiding the direction the optimization algorithm should take when adjusting the network weights to reduce prediction errors and improve model performance. The gradient calculated by the backpropagation algorithm is the basis for automatic adjustment of neural network parameters (i.e. learning) and is the core concept of model training in deep learning.

[0057] Parameters: refers to the variables that make up the neural network model, which define the structure of the network and how it processes input data. Parameters usually include the connection weights between neurons, bias terms, and possible hyperparameters (such as learning rate, number of layers, number of neurons in each layer, etc.). These parameters are adjusted by optimization algorithms during model training, with the goal of minimizing the loss function, that is, reducing the difference between the model's predictions and the actual labels. The initial values ​​of the parameters are usually randomly assigned, and as training proceeds, they are gradually adjusted to capture complex patterns and relationships in the data.

[0058] Gradient averaging: It refers to a step used to aggregate and synchronize model parameter updates in distributed machine learning, especially in the training of neural networks using a parameter server architecture. In this process, each worker node (or worker) first independently calculates their own gradients, which reflect the local changes in the loss function on their respective data subsets relative to the model parameters. These gradients are then sent to the parameter server, which is responsible for averaging these gradients to obtain a global gradient average. This global gradient is then used to update the model parameters held by the parameter server.

[0059] Neural network model execution framework: a software library or platform used to define, train, and deploy neural network models. These frameworks provide a set of tools and interfaces that enable researchers and developers to easily build neural network structures, implement algorithms, and perform computing tasks. Typical neural network model execution frameworks include PyTorch and TensorFlow, which provide tensor computing capabilities, automatic differentiation mechanisms, optimizers, and a series of predefined neural network layers and loss functions, greatly simplifying the development process of complex mathematical models. Such frameworks also support high-level API design, allowing models to be efficiently distributed trained and inferred on different devices, thereby promoting the widespread application of deep learning technology.

[0060] Data Processing Unit (DPU): An efficient, intelligent processing chip that is generally used on network cards as the processing chip of the network card. The DPU is not just a network card responsible for data communication in the traditional sense, it also integrates additional computing power, enabling it to perform advanced data processing tasks when data is transmitted into or out of the network. These tasks may include encryption and decryption of data, compression and decompression, and more complex functions such as network optimization and traffic management. The design of the intelligent DPU allows it to perform these operations independently of the main processor, significantly improving overall data processing efficiency and network performance, while also reducing the workload of the central processing unit (CPU). Therefore, this type of DPU is a key component in heterogeneous computing architectures, especially in applications that require high-speed networks and real-time data processing.

[0061] Different types of accelerators, i.e. heterogeneous accelerators, have different chip architectures and cannot communicate directly.

[0062] The embodiment of the present application provides a data processing method, which can effectively utilize the computing power of multiple accelerators by improving the communication between heterogeneous accelerators, so that the heterogeneous accelerators can run a complete model. For example, please refer to Figure 1 , enabling the idle Class B accelerator to run a complete neural network model together with the Class A accelerator.

[0063] The data processing method provided in the embodiment of the present application is applied to an accelerator cluster, and the accelerator cluster includes at least a first accelerator, a first data processing unit, a second data processing unit, and a second accelerator. The first accelerator is connected to the first data processing unit, the second accelerator is connected to the second data processing unit, and the first data processing unit is connected to the second data processing unit. The first accelerator and the second accelerator are of different types. The first accelerator and the second accelerator are respectively used to run the Nth layer and the N+1th layer of the same model, where N is a positive integer. Among them, the model can be a neural network model or a multi-layer architecture machine learning model including multiple layers. The first data processing unit and the second data processing unit can be a CPU (Central Processing Unit) or a DPU (Data Processing Unit), etc. The first accelerator and the second accelerator are of different types, that is, heterogeneous accelerators. The first accelerator and the second accelerator can belong to the same server or different servers.

[0064] In a possible implementation, the first data processing unit and the second data processing unit are both CPUs to implement the data processing method.

[0065] like Figure 2In the system architecture diagram shown, accelerator A and accelerator B are of different types, that is, accelerator A and accelerator B are heterogeneous accelerators, and accelerator A and accelerator B belong to different servers. Accelerator A runs neural network layer N (layerN) of the neural network model, and accelerator B runs neural network layer N+1 (layerN+1) of the neural network model. NIC (Network Interface Controller) is a network card, CPU-A may include a switching unit to provide data movement channels between CPU-A and accelerator A, and between CPU-A and NIC-A. CPU-B may also include a switching unit to provide data movement channels between CPU-B and accelerator B, and between CPU-B and NIC-B.

[0066] In some implementations, the CPU does not include a switching unit, and the system architecture may include a switching chip. The switching chip may be a PCIE (peripheral component interconnect express, a high-speed serial computer expansion bus standard) switching chip. Taking CPU-A as an example, the switching chip Switch-A may be used to connect accelerator A, CPU-A, and NIC-A.

[0067] In the training process of the neural network model, taking the forward calculation stage as an example, the data processing method includes the following steps 201-204:

[0068] 201: Accelerator A runs layerN to perform calculations and obtains a first activation value, and CPU-A obtains the first activation value.

[0069] The data processing method provided in this embodiment is implemented in the form of a neural network layer, that is, CPU-A and CPU-B run the intermediate neural network layer between layerN and layerN+1, CPU-A can respond to the call of layerN, and CPU-B can pass data to layerN+1 by calling the layerN+1 interface.

[0070] After accelerator A runs layerN to perform calculations, it obtains the first activation value and stores it in the memory of accelerator A. It also sends a call request to CPU-A through the interface call between the neural network layers in the neural network. In response to the call request of accelerator A, CPU-A calls the copy function of accelerator A and copies the first activation value in accelerator A to the memory of CPU-A.

[0071] The copy function of accelerator A is a library function of accelerator A (such as cudaMemcpy of GPU), which supports memory copying of data between accelerators and between accelerator and external device.

[0072] The first activation value is calculated by accelerator A, so its data type is a data type adapted by accelerator A.

[0073] 202: CPU-A sends the target activation value to NIC-A, so that NIC-A sends the target activation value to NIC-B.

[0074] The target activation value is obtained based on the first activation value.

[0075] Exemplarily, the target activation value may be the first activation value.

[0076] Exemplarily, the target activation value is obtained by CPU-A performing precision conversion on the first activation value, and the target activation value is adapted to accelerator B, that is, CPU-A directly converts the data type of the first activation value into a data type adapted to accelerator B.

[0077] Exemplarily, the target activation value is obtained by CPU-A performing precision conversion on the first activation value, and the target activation value is used to generate the second activation value, and the second activation value is adapted to the second accelerator, that is, CPU-A first converts the first activation value into a target activation value with a unified and standardized data type, and then CPU-B converts the target activation value into a second activation value adapted to accelerator B.

[0078] CPU-A sends the target activation value to NIC-A by calling the network send function of the operating system.

[0079] For example, after creating the sockets of nodes A and B (i.e., CPU-A and NIC-A) on Linux, CPU-A can call the send function to send data and send the target activation value to NIC-A:

[0080] After receiving the target activation value, NIC-A sends the target activation value to NIC-B through the network connection pre-established between NIC-A and NIC-B.

[0081] 203: CPU-B receives the target activation value through NIC-B.

[0082] CPU-B receives the target activation value sent by NIC-B by calling the network receiving function (recv function) of the operating system. This process is opposite to the process of step 202, and the principle is similar, so it will not be repeated here.

[0083] 204: CPU-B sends the second activation value to accelerator B.

[0084] CPU-B calls the copy function of accelerator B to copy the second activation value to the memory of accelerator B, and calls the layerN+1 interface running on accelerator B to pass in the second activation value, thereby driving layerN+1 to perform subsequent calculations.

[0085] Exemplarily, although accelerator A and accelerator B are heterogeneous accelerators, if accelerator B can process the first activation value, the target activation value may be the first activation value, and the second activation value may also be the first activation value.

[0086] Exemplarily, if the target activation value is obtained by CPU-A performing precision conversion on the first activation value, and the target activation value is adapted to accelerator B, then the second activation value is the target activation value.

[0087] Exemplarily, if the target activation value is obtained by CPU-A performing precision conversion on the first activation value, and the target activation value is not suitable for accelerator B and is only used as an intermediate value, CPU-B needs to perform precision conversion on the target activation value to obtain a second activation value suitable for accelerator B.

[0088] In this embodiment, when the accelerator A is the first accelerator and the accelerator B is the second accelerator, the first activation value may be a first value, the target activation value may be a target value, and the second activation value may be a second value.

[0089] In the back propagation phase of the neural network model, the data processing method of the gradient value is the same as the data processing method of the activation value in the forward calculation phase. Specifically, accelerator B runs layerN+1 to perform calculations and obtains the first gradient value. CPU-B obtains the first gradient value and sends the target gradient value to NIC-A through NIC-B. The target gradient value is obtained based on the first gradient value. CPU-A receives the target gradient value through NIC-A and sends the second gradient value to accelerator A. The second gradient value is obtained based on the target gradient value. CPU-A calls the layerN interface on accelerator A and passes in the second gradient value, thereby driving layerN to perform subsequent calculations. Among them, the first gradient value can be the fourth value, and the target gradient value can be the third value.

[0090] In the scenario where accelerator A and accelerator B belong to the same server, if the CPU includes a switching unit, it provides a data movement channel between the accelerator and the CPU. In this scenario, communication between heterogeneous accelerators can be achieved by one CPU or two CPUs.

[0091] Taking a CPU as an example, in the forward calculation phase, after obtaining the first activation value, the CPU sends the target activation value to accelerator B without the need for a network card for data transmission. In the reverse propagation phase, after obtaining the first gradient value, the CPU sends the target gradient value to accelerator A, also without the need for a network card for data transmission.

[0092] In the scenario implemented by two CPUs, the communication between heterogeneous accelerators is Figure 2 The communication methods shown are similar, except that CPU-A and CPU-B transmit data through the bus between the CPUs, without the need for a network card. However, it is understandable that in an accelerator cluster, all accelerators used to run a neural network model generally cannot belong to the same server, and heterogeneous accelerators belonging to different servers are essential application scenarios.

[0093] In the embodiment of the present application, a first data processing unit and a second data processing unit are introduced. After obtaining a first value calculated by a first accelerator, the first data processing unit obtains a target value based on the first value, and sends the target value to the second accelerator through the second data processing unit, thereby realizing communication between the first accelerator and the second accelerator, that is, realizing communication between heterogeneous accelerators, laying a foundation for effectively utilizing the computing power of multiple accelerators, so that heterogeneous accelerators can run a complete model.

[0094] In the heterogeneous accelerator cluster, the embodiment of the present application utilizes the CPU connected to the accelerator to cooperate with the NIC to realize data transmission and data reception, thereby realizing the transmission of model parameters between various accelerators, and no parameter server node is required to synchronize the model parameters on each accelerator, thereby realizing the operation of the neural network model in a pipeline parallel manner on the heterogeneous accelerator cluster without relying on the parameter server node. This improvement not only simplifies the system architecture, but also enables the operation of neural network models that exceed the memory capacity of a single accelerator.

[0095] In another possible implementation, the data processing unit is a DPU, which implements the data processing method.

[0096] like Figure 3 In the system architecture diagram shown, accelerator A and accelerator B are heterogeneous accelerators, and accelerator A and accelerator B belong to different servers. Accelerator A runs neural network layer N (layerN) of the neural network model, and accelerator B runs neural network layer N+1 (layerN+1) of the neural network model. Accelerator A is connected to network card A, which includes DPU-A, and accelerator B is connected to network card B, which includes DPU-B.

[0097] In some embodiments, the system architecture may include a switching chip, which may be a PCIE switching chip. Taking DPU-A as an example, the switching chip Switch-A may be used to connect accelerator A and DPU-A to provide a data movement channel between DPU-A and accelerator A.

[0098] In some embodiments, the system architecture may include CPU-A and CPU-B, where CPU-A and CPU-B include a switching unit, CPU-A provides a data movement channel between DPU-A and accelerator A, and CPU-B provides a data movement channel between DPU-B and accelerator B.

[0099] In the training process of the neural network model, taking the forward calculation stage as an example, the data processing method includes the following steps 301-303:

[0100] 301: Accelerator A runs layerN to perform calculations and obtains a first activation value, and DPU-A obtains the first activation value.

[0101] After accelerator A runs layerN to perform calculations, it obtains the first activation value and stores it in the memory of accelerator A, and sends a call request to DPU-A through the interface call between the neural network layers in the neural network. In response to the call request of accelerator A, DPU-A calls the copy function of accelerator A and copies the first activation value in the memory of accelerator A to the memory of DPU-A.

[0102] The first activation value is calculated by accelerator A, and its data type is a data type adapted by accelerator A.

[0103] 302: DPU-A sends the target activation value to DPU-B.

[0104] Since the DPU has a network card function, the target activation value can be directly sent to the DPU-B through the network connection pre-established between the DPU-A and the DPU-B.

[0105] Similar to step 202 in the above embodiment, the target activation value may be the first activation value, and the target activation value may also be obtained by DPU-A performing precision conversion on the first activation value. The target activation value may be directly adapted to accelerator B, that is, DPU-A directly converts the data type of the first activation value to a data type adapted to accelerator B. DPU-A may also first convert the first activation value to a target activation value having a unified and standardized data type, and then DPU-B converts the target activation value to a second activation value adapted to accelerator B.

[0106] 303: DPU-B sends the second activation value to accelerator B.

[0107] DPU-B calls the copy function of accelerator B to copy the second activation value to the memory of accelerator B, and calls the layerN+1 interface running on accelerator B to pass in the second activation value, thereby driving layerN+1 to perform subsequent calculations.

[0108] Similar to step 204 in the above embodiment, if accelerator B can process the first activation value, the target activation value can be the first activation value, and the second activation value is also the first activation value. If the target activation value is obtained by DPU-A performing precision conversion on the first value, and the target activation value is adapted to accelerator B, the second activation value is the target activation value. If the target activation value is obtained by DPU-A performing precision conversion on the first value, and the target activation value is not adapted to accelerator B and is only used as an intermediate value, DPU-B needs to perform precision conversion on the target activation value to obtain the second activation value adapted to accelerator B.

[0109] Compared with the above-mentioned CPU-based embodiment, this embodiment optimizes the data transfer process. Since the data processing unit is the DPU in the network card instead of the CPU, the data transfer does not need to pass through the CPU, which significantly reduces the number of data transfers. In the data transmission process of the CPU-based embodiment, the data needs to be transferred from the accelerator A memory to the CPU-A memory first, and then from the CPU-A memory to the NIC-A. This process involves two data movements. This embodiment introduces the processing chip DPU of the network card. During the data transmission process, the data only needs to be directly transferred from the accelerator A memory to the DPU-A memory, which effectively reduces the number of data transfers between different device memories.

[0110] Since the embodiment reduces the number of times data is transferred between different device memories, it not only reduces the delay in data transmission, but also reduces the computing load of the entire system. In addition, since unnecessary data transfer between different device memories is reduced, faster data processing speed is allowed, and the computing power of the accelerator can be used more effectively, this optimization improves the efficiency and performance of the overall system. In summary, the embodiment realizes a more efficient and smooth data processing process, thereby improving the training and operation efficiency of neural network models in heterogeneous computing environments.

[0111] It should be noted that the precision algorithms of different types of accelerators are inconsistent. An accelerator generally cannot directly process the data calculated by an accelerator that is inconsistent with its precision algorithm. Therefore, while establishing communication between heterogeneous accelerators, it is also necessary to solve the problem of inconsistent precision between heterogeneous accelerators.

[0112] Among them, Figure 2Taking the embodiment shown as an example, one precision conversion method is: CPU-A performs precision conversion on the first activation value to obtain the target activation value adapted by accelerator B, that is, CPU-A directly converts the data type of the first activation value into the data type adapted by accelerator B. If the accelerator cluster used to run the neural network model includes multiple types of accelerators, it is necessary to pre-configure the data type adapted by the upstream accelerator and the data type adapted by the downstream accelerator in the data processing unit connected to each accelerator in communication.

[0113] Another precision conversion method is: CPU-A performs precision conversion on the first activation value to obtain a target activation value. The target activation value has a unified and standardized data type, but the target activation value is not suitable for accelerator B. It only serves as an intermediate value. CPU-B needs to perform precision conversion on the target activation value to obtain a second activation value adapted to accelerator B. Although this precision conversion method requires two precision conversions during the data transmission process of heterogeneous accelerators, since the target activation value has a unified and standardized data type, it is compatible between different accelerators, that is, the activation value calculated by itself can be converted into the target activation value, and the target activation value can also be converted into the activation value adapted to itself. Therefore, the data type of precision conversion can be uniformly configured in the data processing unit connected to each accelerator communication, the configuration is simple, and the hardware resources of the accelerator can be called for precision conversion.

[0114] The embodiment of the present application does not require the accelerator to convert the precision, but the precision conversion is performed by the data processing unit to avoid increasing the calculation burden of the accelerator. In addition, since not all accelerators have the precision conversion capability, some idle accelerators cannot participate in the model calculation. The embodiment of the present application solves the problem of inconsistent precision between heterogeneous accelerators by performing precision conversion by the data processing unit, so that all idle accelerators can jointly complete the model calculation task, thereby improving resource utilization.

[0115] The second precision conversion method is illustrated below by two embodiments.

[0116] In a possible implementation, the data processing unit is a CPU.

[0117] like Figure 4 In the schematic diagram of the system architecture shown, accelerator A and accelerator B are heterogeneous accelerators, and accelerator A and accelerator B belong to different servers. Accelerator A runs neural network layer N (layerN) of the neural network model, and accelerator B runs neural network layer N+1 (layerN+1) of the neural network model. In this embodiment, Switch-A connects accelerator A, CPU-A, and NIC-A, and Switch-B connects accelerator B, CPU-B, and NIC-B.

[0118] In the training process of the neural network model, taking the forward calculation stage as an example, the data processing method includes the following steps 401-405:

[0119] 401. Accelerator A runs layerN to perform calculations and obtains a first activation value, and CPU-A obtains the first activation value through Switch-A.

[0120] After accelerator A runs layerN to perform calculations, it obtains the first activation value and stores it in the memory of accelerator A, and sends a call request to CPU-A through the interface call between the neural network layers in the neural network.

[0121] In response to the call request of accelerator A, CPU-A calls the copy function of accelerator A, and copies the first activation value in the memory of accelerator A to the memory of CPU-A through Switch-A.

[0122] The first activation value is calculated by accelerator A, so its data type is a data type adapted by accelerator A.

[0123] 402. CPU-A performs precision conversion on the first activation value to obtain a second activation value.

[0124] Exemplarily, CPUA performs precision conversion on the first activation value according to a preconfigured data type of the second activation value to obtain a second activation value, and the second activation value is stored in the CPU-A memory, and the data type of the second activation value is a unified and standardized data type.

[0125] 403.CPU-A sends the second activation value to NIC-A through Switch-A.

[0126] The process of CPU-A sending the second activation value to NIC-A through Switch-A is completed by calling the network sending function of the operating system. Figure 2 The same is true for step 202 in the illustrated embodiment, which will not be described in detail here.

[0127] After receiving the second activation value, NIC-A sends the second activation value to NIC-B through the network connection pre-established between NIC-A and NIC-B.

[0128] 404.CPU-B receives the second activation value through Switch-B and NIC-B.

[0129] After NIC-B receives the second activation value, CPU-B obtains the second activation value through the data movement channel between NIC-B and CPU-B provided by Switch-B, which is also accomplished by calling the network receiving function of the operating system.

[0130] 405. CPU-B performs precision conversion on the second activation value to obtain a third activation value, and sends the third activation value to accelerator B.

[0131] Among them, the third activation value adapts accelerator B.

[0132] Exemplarily, the data type supported by accelerator A is bfloat16, the data type supported by accelerator B is float16, and the data type of the pre-configured second activation value is float32. The data precision conversion process for sending data from accelerator A to accelerator B is: bfloat16→float32→float16.

[0133] After obtaining the third activation value, CPU-B calls the copy function of accelerator B, copies the third activation value from the CPU-B memory to the accelerator B memory through Switch-B, and calls the layerN+1 interface on accelerator B to pass in the third activation value, thereby driving layerN+1 to perform subsequent calculations.

[0134] In the back propagation phase of the neural network model, the data processing method of the gradient value is the same as the data processing method of the activation value in the forward calculation phase. Specifically, accelerator B runs layerN+1 to perform calculations and obtains the first gradient value. CPU-B obtains the first gradient value through Switch-B, and performs precision conversion on the first gradient value to obtain the second gradient value. CPU-B sends the second gradient value to NIC-B through Switch-B, and then sends the second gradient value to NIC-A through the network connection between NIC-A and NIC-B. CPU-A receives the second gradient value through NIC-A and Switch-A, performs precision conversion on it, obtains the third gradient value, and sends the third gradient value to accelerator A, calls the layerN interface on accelerator A, and passes in the third gradient value, thereby driving layerN to perform subsequent calculations.

[0135] In the embodiment of the present application, the CPU is used to perform precision conversion on the data to be transmitted between heterogeneous accelerators, and the NIC is used to realize data transmission and data reception, so as to realize the operation of the neural network model in a pipeline parallel manner on the heterogeneous accelerator cluster without relying on the parameter server node. This improvement not only simplifies the system architecture, but also enables the operation of neural network models that exceed the memory capacity of a single accelerator.

[0136] In addition, the embodiments of the present application solve the communication problems and inconsistent precision problems between heterogeneous accelerators, realize efficient data transmission and accurate precision matching between heterogeneous accelerators, and ensure that activation values ​​and gradient values ​​can be effectively exchanged even between heterogeneous accelerators without direct communication capabilities. Such improvements enable the smooth implementation of pipeline parallelism on heterogeneous accelerators, greatly improving the utilization efficiency of computing resources and the training efficiency of models.

[0137] In another possible implementation, the data processing unit is a DPU.

[0138] like Figure 5 In the system architecture diagram shown, accelerator A and accelerator B are heterogeneous accelerators, and accelerator A and accelerator B belong to different servers. Accelerator A runs neural network layer N (layerN) of the neural network model, and accelerator B runs neural network layer N+1 (layerN+1) of the neural network model. Switch-A connects accelerator A and DPU-A to provide a data movement channel between accelerator A and DPU-A. Switch-B connects accelerator B and DPU-B to provide a data movement channel between accelerator B and DPU-B.

[0139] In the training process of the neural network model, taking the forward calculation stage as an example, the data processing method includes the following steps 501-503:

[0140] 501. Accelerator A runs layerN to perform calculations and obtains a first activation value, and DPU-A obtains the first activation value through Switch-A.

[0141] After accelerator A runs layerN to perform calculations, it obtains the first activation value and stores it in the memory of accelerator A. It also sends a call request to DPU-A through the interface call between the neural network layers in the neural network. In response to the call request of accelerator A, DPU-A calls the copy function of accelerator A and copies the first activation value in the memory of accelerator A to the memory of DPU-A through Switch-A.

[0142] The first activation value is calculated by accelerator A, and its data type is a data type adapted by accelerator A.

[0143] 502. DPU-A performs precision conversion on the first activation value to obtain a second activation value, and sends the second activation value to DPU-B through the network between DPU-A and DPU-B.

[0144] Exemplarily, DPU-A performs precision conversion on the first activation value according to the data type of the pre-configured second activation value to obtain the second activation value. The data type of the second activation value is a unified and standardized data type to ensure that the data types are compatible between different accelerators. DPU-A can call the hardware resources of accelerator A to convert the first activation value into the second activation value, and subsequently DPU-B can call the hardware resources of accelerator B to convert the second activation value into a third activation value adapted to accelerator B.

[0145] The DPU, as a processing chip of the network card, can directly send the second activation value to DPU-B through the pre-established network connection between DPU-A and DPU-B.

[0146] 503. DPU-B receives the second activation value, stores the second activation value in the DPU-B memory, performs precision conversion on the second activation value to obtain a third activation value, and sends the third activation value to accelerator B.

[0147] Among them, the third activation value adapts accelerator B.

[0148] After obtaining the third activation value, DPU-B calls the copy function of accelerator B, copies the third activation value from the DPU-B memory to the accelerator B memory through Switch-B, and calls the layerN+1 interface on accelerator B to pass in the third activation value, thereby driving layerN+1 to perform subsequent calculations.

[0149] Compared with the above-mentioned CPU-based embodiment, this embodiment optimizes the data transfer process and reduces the number of times data is transferred between different device memories, which not only reduces the delay in data transmission, but also reduces the computing load of the entire system. In addition, since unnecessary data transfer between different device memories is reduced, faster data processing speed is allowed, and the computing power of the accelerator can be used more effectively, this optimization improves the efficiency and performance of the overall system. In summary, the embodiment of the present application realizes a more efficient and smooth data processing process, thereby improving the training and operation efficiency of neural network models in heterogeneous computing environments.

[0150] It should also be noted that, in other embodiments, it can be determined in advance whether the two accelerators that need to perform data transmission are heterogeneous accelerators. If they are not heterogeneous accelerators, there is no need to convert the accuracy.

[0151] The following is a detailed introduction to the software framework running on the data processing unit.

[0152] See also Figure 6 , Figure 6 The accelerator cluster architecture schematic diagram provided in the embodiment of the present application is shown. The accelerator cluster includes multiple accelerators, Figure 6In the example, accelerator 1 (accelerator_1), accelerator 2 (accelerator_2) and accelerator 3 (accelerator_3) are used for explanation. The three accelerators are of different types, that is, heterogeneous accelerators. Each accelerator is connected to a data processing unit for communication, and the data processing unit serves as a transfer device between heterogeneous accelerators.

[0153] At least one of a multi-device transport agent upstream layer (MDTA-UP) and a multi-device transport agent downstream layer (MDTA-DOWN) runs on the data processing unit.

[0154] Taking running a neural network model on an accelerator cluster as an example, accelerator 1 (accelerator_1) corresponds to the first node of the neural network model, and runs neural network layer 1 (layer1) and neural network layer 2 (layer2) in the neural network model. Accelerator 2 (accelerator_2) corresponds to the second node in the neural network model, and runs neural network layer 3 (layer3) and neural network layer 4 (layer4) in the neural network model. Accelerator 3 (accelerator_3) corresponds to the last node of the neural network model, and runs neural network layer 5 (layer5). Only MDTA-UP1 runs on data processing unit 1 connected to accelerator 1 (accelerator_1), MDTA-UP2 and MDTA-DOWN2 run on data processing unit 2 connected to accelerator 2 (accelerator_2), and only MDTA-DOWN3 runs on data processing unit 3 connected to accelerator 3 (accelerator_3). Together, they form a neural network computing environment that can support pipeline parallelism on heterogeneous accelerators. In the forward computing phase, the accelerator passes the calculated activation value to the next accelerator. In the back-propagation phase, the accelerator passes the calculated gradient value back to the previous accelerator, and each accelerator can also process other computing tasks in the neural network training task in parallel.

[0155] like Figure 7 As shown, in the forward calculation stage of the neural network model, the following steps 701-710 are included:

[0156] 701: MDTA-UP1 obtains the first activation value in response to the calculation of the neural network layer layer2 running on the accelerator 1;

[0157] After accelerator 1 runs layer2 to calculate the first activation value, it stores it in the memory of accelerator 1. Accelerator 1 sends a call request to MDTA-UP1 running on data processing unit 1 through the interface call between the neural network layers in the neural network. MDTA-UP1 responds to the call request of accelerator 1, calls the copy function of accelerator 1, and copies the first activation value from the memory of accelerator 1 to the memory of data processing unit 1.

[0158] 702: MDTA-UP1 performs precision conversion on the first activation value to obtain a second activation value;

[0159] MDTA-UP1 performs precision conversion on the first activation value according to the pre-configured data type of the second activation value to obtain the second activation value, and the second activation value is stored in the memory of the data processing unit 1.

[0160] 703: MDTA-UP1 sends the second activation value to MDTA-DOWN2;

[0161] If data processing unit 1 and data processing unit 2 are both CPUs, data processing unit 1 is connected to network card 1, data processing unit 2 is connected to network card 2, and accelerator 1 and accelerator 2 belong to different servers, then MDTA-UP1 first sends the second activation value to network card 1, and then sends the second activation value to network card 2 through the pre-established network between network card 1 and network card 2, and network card 2 then passes the second activation value to MDTA-DOWN2 running on data processing unit 2.

[0162] If data processing unit 1 is a DPU, and accelerator 1 and accelerator 2 belong to different servers, MDTA-UP1 sends the second activation value to MDTA-DOWN2 running on data processing unit 2 through the network between data processing unit 1 and data processing unit 2.

[0163] 704: MDTA-DOWN2 moves the second activation value to the memory of the data processing unit 2, performs precision conversion on the second activation value, and obtains a third activation value;

[0164] The third activation value after precision conversion is adapted to the accelerator 2 , and the third activation value is stored in the memory of the data processing unit 2 .

[0165] 705: MDTA-DOWN2 sends the third activation value to accelerator 2;

[0166] MDTA-DOWN2 calls the copy function of accelerator 2 to copy the third activation value to the memory of accelerator 2, and calls the layer3 interface to pass in the third activation value, thereby driving layer3 to perform subsequent forward calculations.

[0167] Exemplarily, the data type supported by accelerator 1 is bfloat16, the data type supported by accelerator 2 is float16, and the data type of the pre-configured second activation value is float32. The data precision conversion process of sending data from accelerator 1 to accelerator 2 is: bfloat16→float32→float16.

[0168] The forward calculation process between layer3 and layer4 is the internal data processing process in the neural network model and will not be described in detail.

[0169] 706: MDTA-UP2 obtains the fourth activation value in response to layer4 calculating the fourth activation value;

[0170] After accelerator 2 runs Layer4 to calculate the fourth activation value, it stores it in the memory of accelerator 2. Accelerator 2 sends a call request to MDTA-UP2 running on data processing unit 2 through the interface call between the neural network layers in the neural network. MDTA-UP2 responds to the call request of accelerator 2, calls the copy function of accelerator 2, and copies the fourth activation value from the memory of accelerator 2 to the memory of data processing unit 2.

[0171] 707: MDTA-UP2 performs precision conversion on the fourth activation value to obtain a fifth activation value;

[0172] MDTA-UP2 performs precision conversion on the fourth activation value according to the pre-configured data type of the fifth activation value to obtain the fifth activation value, and the fifth activation value is stored in the memory of the data processing unit 2.

[0173] 708: MDTA-UP2 sends the fifth activation value to MDTA-DOWN3;

[0174] If data processing unit 2 and data processing unit 3 are both CPUs, data processing unit 2 is connected to network card 2, data processing unit 3 is connected to network card 3, and accelerator 2 and accelerator 3 belong to different servers, then MDTA-UP2 first sends the fifth activation value to network card 2, and then sends the fifth activation value to network card 3 through the network pre-established between network card 2 and network card 3, and network card 3 then passes the fifth activation value to MDTA-DOWN3 running on data processing unit 3.

[0175] If data processing unit 2 is a DPU, and accelerator 2 and accelerator 3 belong to different servers, MDTA-UP2 sends the fifth activation value to MDTA-DOWN3 running on data processing unit 3 through the network between data processing unit 2 and data processing unit 3.

[0176] 709: MDTA-DOWN3 moves the fifth activation value to the memory of the data processing unit 3, performs precision conversion on the fifth activation value, and obtains a sixth activation value;

[0177] The sixth activation value after precision conversion is adapted to the accelerator 3 , and the sixth activation value is stored in the memory of the data processing unit 3 .

[0178] 710: MDTA-DOWN3 sends the sixth activation value to accelerator 3;

[0179] MDTA-DOWN3 calls the copy function of accelerator 3 to copy the sixth activation value to the memory of accelerator 3, and calls the layer5 interface to pass in the sixth activation value, thereby driving layer5 to perform subsequent forward calculations.

[0180] Exemplarily, the data type supported by accelerator 2 is float16, the data type supported by accelerator 3 is int8, and the data type of the pre-configured fifth activation value is double, then the data precision conversion process of sending data from accelerator 2 to accelerator 3 is: float16→double→int8.

[0181] In the back propagation phase of the neural network model, the following steps 711-720 are included:

[0182] 711: MDTA-DOWN3 obtains the first gradient value in response to the first gradient value calculated by the neural network layer layer5 running on the accelerator 3;

[0183] After accelerator 3 runs Layer5 to calculate the first gradient value, it stores it in the memory of accelerator 3. Accelerator 3 sends a call request to MDTA-DOWN3 running on data processing unit 3 through the interface call between the neural network layers in the neural network. MDTA-DOWN3 responds to the call request of accelerator 3, calls the copy function of accelerator 3, and copies the first gradient value from the memory of accelerator 3 to the memory of data processing unit 3.

[0184] 712: MDTA-DOWN3 performs precision conversion on the first gradient value to obtain a second gradient value;

[0185] MDTA-DOWN3 performs precision conversion on the first gradient value according to the pre-configured data type of the second gradient value to obtain the second gradient value, and the second gradient value is stored in the memory of the data processing unit 3.

[0186] The data types of the activation values ​​and gradient values ​​calculated by the same accelerator are the same. Similar to the activation values, the data types of the gradient values ​​calculated by different accelerators are different, and the gradient values ​​also need to be converted in terms of accuracy.

[0187] 713: MDTA-DOWN3 sends the second gradient value to MDTA-UP2;

[0188] If data processing unit 2 and data processing unit 3 are both CPUs, data processing unit 2 is connected to network card 2, data processing unit 3 is connected to network card 3, and accelerator 3 and accelerator 2 belong to different servers, then MDTA-DOWN3 first sends the second gradient value to network card 3, and then sends the second gradient value to network card 2 through the network between network card 3 and network card 2, and network card 2 then passes the second gradient value to MDTA-UP2 running on data processing unit 2.

[0189] If the data processing unit 3 is a DPU, and the accelerator 3 and the accelerator 2 belong to different servers, MDTA-DOWN3 sends the second gradient value to MDTA-UP2 running on the data processing unit 2 through the network between the data processing unit 3 and the data processing unit 2.

[0190] 714: MDTA-UP2 moves the second gradient value to the memory of the data processing unit 2, performs precision conversion on the second gradient value, and obtains a third gradient value;

[0191] The third gradient value after precision conversion is adapted to the accelerator 2 , and the third gradient value is stored in the memory of the data processing unit 2 .

[0192] 715: MDTA-UP2 sends the third gradient value to accelerator 2;

[0193] MDTA-UP2 calls the copy function of accelerator 2 to copy the third gradient value to the memory of accelerator 2, and calls the layer4 interface to pass in the third gradient value, thereby driving layer4 to perform subsequent backward propagation.

[0194] Since the data types of the activation value and gradient value calculated by the same accelerator are the same, the data type supported by accelerator 3 is int8, and the data type supported by accelerator 2 is float16. As an example, the data type of the second gradient value is pre-configured to be double. The data precision conversion process of sending data from accelerator 3 to accelerator 2 is: int8→double→float16.

[0195] The back propagation process between Layer4 and Layer3 is the internal data processing process in the neural network model and will not be described in detail.

[0196] 716: MDTA-DOWN2 obtains the fourth gradient value in response to layer3 calculating the fourth gradient value;

[0197] After accelerator 2 runs Layer3 to calculate the fourth gradient value, it stores it in the memory of accelerator 2. Accelerator 2 sends a call request to MDTA-DOWN2 running on data processing unit 2 through the interface call between the neural network layers in the neural network. MDTA-DOWN2 responds to the call request of accelerator 2, calls the copy function of accelerator 2, and copies the fourth gradient value from the memory of accelerator 2 to the memory of data processing unit 2.

[0198] 717: MDTA-DOWN2 performs precision conversion on the fourth gradient value to obtain a fifth gradient value;

[0199] MDTA-DOWN2 performs precision conversion on the fourth gradient value according to a preconfigured data type to obtain a fifth gradient value, and the fifth gradient value is stored in the memory of the data processing unit 2.

[0200] 718: MDTA-DOWN2 sends the fifth gradient value to MDTA-UP1;

[0201] If data processing unit 1 and data processing unit 2 are both CPUs, data processing unit 1 is connected to network card 1, data processing unit 2 is connected to network card 2, and accelerator 2 and accelerator 1 belong to different servers, then MDTA-DOWN2 first sends the fifth gradient value to network card 2, and then sends the fifth gradient value to network card 1 through the network pre-established between network card 2 and network card 1, and network card 1 then passes the fifth gradient value to MDTA-UP1 running on data processing unit 1.

[0202] If data processing unit 2 is a DPU, and accelerator 2 and accelerator 1 belong to different servers, MDTA-DOWN2 sends the fifth gradient value to MDTA-UP1 running on data processing unit 1 through the network between data processing unit 2 and data processing unit 1.

[0203] 719: MDTA-UP1 moves the fifth gradient value to the memory of the data processing unit 1, performs precision conversion on the fifth gradient value, and obtains a sixth gradient value;

[0204] The sixth gradient value after precision conversion is adapted to the accelerator 1 , and the sixth gradient value is stored in the memory of the data processing unit 1 .

[0205] 720: MDTA-UP1 sends the sixth gradient value to accelerator 1;

[0206] MDTA-UP1 calls the copy function of accelerator 1 to copy the sixth gradient value to the memory of accelerator 1, and calls the layer2 interface to pass in the sixth gradient value, thereby driving layer2 to perform subsequent backward propagation.

[0207] Assuming that the data type supported by accelerator 2 is float16, the data type supported by accelerator 1 is bfloat16, and the data type of the fifth gradient value is pre-configured as float32, the data precision conversion process of sending data from accelerator 2 to accelerator 1 is: float16→float32→bfloat16.

[0208] The embodiment of the present application inserts MDTA-UP and MDTA-DOWN between neural network layers involving cross-accelerators to achieve efficient data transmission and data precision conversion, effectively solving the communication and precision consistency problems faced when performing pipeline parallel training in a heterogeneous accelerator environment.

[0209] In addition, in the embodiment of the present application, MDTA-UP and MDTA-DOWN constitute a multi-device transmission proxy layer, and the multi-device transmission proxy layer is implemented in the form of a neural network layer, that is, it can respond to the call of the previous neural network layer, perform precision conversion on the data calculated by the previous neural network layer, and pass the precision-converted data to the next neural network layer by calling the interface of the next neural network layer. Therefore, the multi-device transmission proxy layer can be directly inserted into the neural network model in a non-invasive manner without the need to modify the tasks of the neural network operation framework. This seamless integration method greatly improves the applicability and flexibility of the embodiment of the present application, and allows efficient pipeline parallel training using a collection of heterogeneous accelerators without changing the existing neural network operation framework.

[0210] It should also be noted that before executing the above data processing method, the MDTA-UP and / or MDTA-DOWN running on the data processing unit needs to be initialized.

[0211] The initialization process includes: responding to the initialization instruction, reading the configuration file, obtaining the configuration information, and establishing a network connection between the local end and the remote end according to the configuration information. Figure 6 As shown, taking data processing unit 2 as the local end as an example, the opposite end of MDTA-UP2 is MDTA-DOWN3, and the opposite end of MDTA-DOWN2 is MDTA-UP1.

[0212] Among them, if only MDTA-UP is run on the data processing unit, the configuration file only includes the first configuration file, and the first configuration file is used to initialize MDTA-UP; if MDTA-UP and MDTA-DOWN are run on the data processing unit, the configuration file includes the first configuration file and the second configuration file, the first configuration file is used to initialize MDTA-UP, and the second configuration file is used to initialize MDTA-DOWN; if only MDTA-DOWN is run on the data processing unit, the configuration file only includes the second configuration file, and the second configuration file is used to initialize MDTA-DOWN.

[0213] If the configuration file includes a first configuration file, the first configuration file is read to obtain first configuration information, the first configuration information including: the local network receiving monitoring IP and port and the downstream first peer network receiving IP and port. The first peer network receiving IP and port are the peer network receiving IP and port of MDTA-UP. Figure 6 As shown, the first peer network receiving IP and port of MDTA-UP2 are the network receiving IP and port of the downstream MDTA-DOWN3.

[0214] Then, MDTA-UP is initialized, and MDTA-UP calls the preset network monitoring interface (if the data processing unit is a CPU, the preset network monitoring interface is the network monitoring interface of the CPU operating system; if the data processing unit is a DPU, the preset network monitoring interface is the network monitoring interface of the DPU), monitors the local network receiving monitoring IP and port, completes the initialization of the first local network receiving function, and waits for the network sending layer of the first peer end to connect. The first local network receiving function is used to receive data sent by the first peer end, such as Figure 6 As shown, the first peer of MDTA-UP2 is MDTA-DOWN3. After completing the initialization of the first local network receiving function, MDTA-UP2 waits for the network sending layer connection of MDTA-DOWN3, and then MDTA-UP2 can receive the data sent by MDTA-DOWN3. After completing the initialization of the first local network receiving function, MDTA-UP calls the preset network connection interface (if the data processing unit is a CPU, the preset network connection interface is the network connection interface of the CPU operating system; if the data processing unit is a DPU, the preset network connection interface is the network connection interface of the DPU), monitors the first peer network receiving IP and port, and enables MDTA-UP to establish a network connection with the first peer, so as to send data to the first peer, as shown in FIG. Figure 6 As shown, the first counterpart of MDTA-UP2 is MDTA-DOWN3, then MDTA-UP2 is connected to MDTA-DOWN3, and subsequently MDTA-UP2 can send data to MDTA-DOWN3.

[0215] If the configuration file includes a second configuration file, the second configuration file is read to obtain second configuration information, which includes: the local network receiving monitoring IP and port and the upstream second peer network receiving IP and port. The second peer network receiving IP and port are the peer network receiving IP and port of MDTA-DOWN. Figure 6 As shown, the peer network receiving IP and port of MDTA-DOWN2 are the upstream MDTA-UP1 network receiving IP and port.

[0216] Then, MDTA-DOWN is initialized, and the preset network monitoring interface is called (if the data processing unit is a CPU, the preset network monitoring interface is the network monitoring interface of the CPU operating system; if the data processing unit is a DPU, the preset network monitoring interface is the network monitoring interface of the DPU), the local network receives the monitoring IP and port, and the second local network receiving function is initialized, and the network sending layer of the second peer is waited for to be connected. The second local network receiving function is used to receive the data sent by the second peer, such as Figure 6 As shown, the second peer of MDTA-DOWN2 is MDTA-UP1. After completing the initialization of the second local network receiving function, MDTA-DOWN2 waits for the network sending layer connection of MDTA-UP1, and then MDTA-DOWN2 can receive the data sent by MDTA-UP1. After completing the initialization of the second local network receiving function, MDTA-DOWN2 calls the preset network connection interface (if the data processing unit is a CPU, the preset network connection interface is the network connection interface of the CPU operating system; if the data processing unit is a DPU, the preset network connection interface is the network connection interface of the DPU), monitors the network receiving IP and port of the second peer, so that MDTA-DOWN establishes a network connection with the second peer, which is used to send data to the second peer, such as Figure 6 As shown, the second opposite end of MDTA-DOWN2 is MDTA-UP1, so MDTA-DOWN2 is connected to MDTA-UP1, and subsequently MDTA-DOWN2 can send data to MDTA-UP1.

[0217] Furthermore, the first configuration file information may also include an identifier of the accelerator. During the accelerator initialization process, the data processing unit calls a copy function corresponding to the identifier of the accelerator according to the identifier of the accelerator, and copies the weights of the neural network layer run by the accelerator to the accelerator, in preparation for the subsequent accelerator to run the neural network.

[0218] If the data processing unit is a CPU, initializing MDTA-UP and / or MDTA-DOWN running on the CPU includes:

[0219] If MDTA-UP is running on the CPU, the initialization of MDTA-UP includes: the network receiving module of MDTA-UP calls the network monitoring interface (bind) of the operating system, monitors the local network receiving monitoring IP and port to complete the local network receiving module initialization, and waits for the downstream MDTA-DOWN network sending layer to connect. The network sending module of MDTA-UP calls the network connection interface (connect) of the operating system, monitors the peer network receiving IP and port, and connects to the downstream MDTA-DOWN network receiving module.

[0220] If MDTA-DOWN is running on the CPU, the network receiving module of MDTA-DOWN calls the network monitoring interface (bind) of the operating system, monitors the local network receiving monitoring IP and port to complete the local network receiving module initialization, and waits for the upstream MDTA-UP network sending module to connect. The network sending module of MDTA-DOWN calls the network connection interface (connect) of the operating system, monitors the peer network receiving IP and port, and connects to the upstream MDTA-UP network receiving module.

[0221] If the data processing unit is a DPU, initializing MDTA-UP and / or MDTA-DOWN running on the DPU includes:

[0222] If MDTA-UP is running on the DPU, the initialization of MDTA-UP includes: the network receiving module of MDTA-UP calls the network monitoring interface of DPU (rte_eth_dev_configure, rte_eth_rx_queue_setup), monitors the local network receiving monitoring IP and port to complete the local network receiving module initialization, and waits for the network sending layer of the downstream MDTA-DOWN to connect. The network sending module of MDTA-UP calls the DPU network connection interface (rte_eth_dev_configure, rte_eth_tx_queue_setup), monitors the peer network receiving IP and port, and connects to the downstream MDTA-DOWN network receiving module.

[0223] If MDTA-DOWN is running on the DPU, the initialization of MDTA-DOWN includes: the network receiving module of MDTA-DOWN calls the network monitoring interface of DPU (rte_eth_dev_configure, rte_eth_rx_queue_setup, monitors the local network receiving monitoring IP and port to complete the local network receiving module initialization, and waits for the network sending module of the upstream MDTA-UP to connect. The network sending module of MDTA-DOWN calls the DPU network connection interface (rte_eth_dev_configure, rte_eth_tx_queue_setup), monitors the peer network receiving IP and port, and connects to the upstream MDTA-UP network receiving module.

[0224] Figure 8 The software structure diagram of the data processing unit is shown. The above-mentioned MDTA-UP and MDTA-DOWN respectively include an accelerator data handling module, a precision conversion module, a network sending module and a network receiving module.

[0225] The accelerator data handling module is used to move data between the accelerator and the data processing unit memory.

[0226] Exemplarily, the accelerator data handling module copies data from the accelerator's memory to the data processing unit memory (such as CPU memory or DPU memory).

[0227] Exemplarily, the accelerator data handling module copies data from the data processing unit memory to the accelerator memory.

[0228] The accelerator data handling module is used to realize flexible data migration between different computing devices, providing a basis for data sharing between heterogeneous accelerators.

[0229] The precision conversion module is used to convert data from different data sources into a unified data type, and then convert the data of the unified data type into a data type adapted by the target accelerator.

[0230] like Figure 8 As shown, taking the application of this embodiment to the distributed training scenario of the neural network model based on pipeline parallelism as an example, in the forward calculation stage of the neural network model, the precision conversion module in MDTA-UP converts the activation value calculated by the neural network layer N (layerN) running on accelerator 1 into a unified data type, and after MDTA-DOWN receives the activation value of the unified data type, the precision conversion module in MDTA-DOWN converts the activation value of the unified data type into the activation value of the data type adapted by accelerator 2. The precision conversion module ensures that the data transmitted between different accelerators has a unified and compatible data type.

[0231] The network sending module is used to transmit data from the data processing unit at the sending end to the network receiving module at the receiving end through the network, ensuring that the data can be transmitted efficiently and securely on the network, and providing support for remote communication of data in a distributed computing environment. The network receiving module is used to receive data sent from the network sending module at the opposite end at the receiving end, and store it in the memory of the data processing unit. The precision conversion module performs precision conversion on the data in the memory of the data processing unit to obtain data adapted by the accelerator at the receiving end. The accelerator data handling module copies the precision-converted data to the memory of the accelerator at the receiving end, and the accelerator at the receiving end can access and use the data to complete subsequent computing tasks.

[0232] like Figure 8 As shown, in the forward calculation stage of the neural network model, the network sending module in MDTA-UP transmits the activation value after precision conversion to the network receiving module in MDTA-DOWN through the network; in the backward propagation stage of the neural network model, the network sending module in MDTA-DOWN transmits the gradient after precision conversion to the network receiving module in MDTA-UP through the network.

[0233] The accelerator data handling module, precision conversion module, network sending module and network receiving module together constitute the core module of the data processing unit in this embodiment. Through mutual cooperation, an efficient data transmission and processing framework is realized. In the pipeline parallel scheme, it can ensure the effective handling, conversion and accurate transmission of data between heterogeneous accelerators.

[0234] The embodiment of the present application also provides a server, including a first accelerator and a first processing unit;

[0235] The first accelerator is connected to the first data processing unit, the first data processing unit is connected to the second data processing unit, the second data processing unit is connected to the second accelerator, the first accelerator and the second accelerator are of different types, and the first accelerator and the second accelerator are respectively used to run the Nth layer and the N+1th layer of the same model, where N is a positive integer;

[0236] The first data processing unit is used for obtaining a first value in response to the first accelerator running the Nth layer calculation, acquiring the first value, and sending a target value to the second accelerator through the second data processing unit, the target value being obtained based on the first value.

[0237] The first accelerator and the second accelerator may belong to the same server or different servers, which is not specifically limited in this embodiment.

[0238] In a possible implementation, the server further includes a first network card, the first data processing unit and the second data processing unit are CPUs, the first data processing unit is connected to the first network card, the first network card is communicatively connected to the second network card, and the second network card is connected to the second data processing unit;

[0239] Or the first data processing unit and the second data processing unit are DPUs.

[0240] Among them, the first data processing unit and the second data processing unit are CPUs, the first data processing unit and the second data processing unit are DPUs, and the specific data interaction methods in the two scenarios have been introduced in detail in the above embodiments and will not be repeated here.

[0241] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on a data processing unit in an accelerator cluster, the data processing unit implements the data processing method described in any embodiment of the present application.

[0242] An embodiment of the present application also provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by a data processing unit in an accelerator cluster, the data processing unit can implement the data processing method described in any embodiment of the present application.

[0243] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0244] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0245] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0246] The above-mentioned embodiments may be combined arbitrarily. For the above description of the disclosed embodiments, the features recorded in each embodiment in this specification may be replaced or combined with each other, so that professional and technical personnel in this field can implement or use the embodiments of the present application.

[0247] The above description of the disclosed embodiments enables professionals and technicians in the field to implement or use the embodiments of the present application. Various modifications to these embodiments will be apparent to professionals and technicians in the field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the embodiments of the present application. Therefore, the embodiments of the present application will not be limited to these embodiments shown herein, but will conform to the widest range consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, characterized in that: Applied to an accelerator cluster, the accelerator cluster includes a first accelerator, a first data processing unit, a second data processing unit, and a second accelerator, the first accelerator is connected to the first data processing unit, the second accelerator is connected to the second data processing unit, the first data processing unit is connected to the second data processing unit, the first accelerator and the second accelerator are of different types, the first accelerator and the second accelerator are respectively used to run the Nth layer and the N+1th layer of the same model, wherein N is a positive integer, and the method includes: The first data processing unit obtains the first value in response to the first accelerator running the Nth layer calculation to obtain the first value; The first data processing unit sends a target value to the second accelerator through the second data processing unit, where the target value is obtained based on the first value.

2. The data processing method according to claim 1, characterized in that: After the first data processing unit obtains a first value in response to the first accelerator executing the Nth layer calculation, the method further includes: The first data processing unit performs precision conversion on the first value to obtain the target value, and the precision of the target value is adapted to the precision of the second accelerator.

3. The data processing method according to claim 1, characterized in that: After the first data processing unit obtains a first value in response to the first accelerator executing the Nth layer calculation, the method further includes: The first data processing unit performs precision conversion on the first value to obtain the target value, and the target value is used to generate a second value, and the precision of the second value is adapted to the precision of the second accelerator.

4. The data processing method according to claim 1, characterized in that: The data processing method further includes: The first data processing unit receives a third value through the second data processing unit, where the third value is obtained based on a fourth value, where the fourth value is obtained by the second accelerator running the N+1th layer calculation; The first data processing unit sends the third value to the first accelerator.

5. The data processing method according to claim 1, characterized in that: Before the first data processing unit executes the Nth layer calculation in response to the first accelerator to obtain the first value, the data processing method further includes: The first data processing unit obtains first configuration information by reading a first configuration file in response to the initialization instruction, wherein the first configuration information includes: a local network receiving listening IP and port, and a network receiving IP and port of the second data processing unit; The first data processing unit calls a preset network monitoring interface, monitors the local network receiving monitoring IP and port to complete the local network receiving function initialization, and waits for the network sending layer of the second data processing unit to connect; The first data processing unit calls a preset network connection interface, monitors the network receiving IP and port of the second data processing unit, and establishes a network connection with the second data processing unit.

6. The data processing method according to any one of claims 1 to 5, characterized in that: The first data processing unit is a CPU or a DPU.

7. A data processing unit, characterized in that: The data processing unit is the first data processing unit in the accelerator cluster, the accelerator cluster further includes a first accelerator, a second data processing unit, and a second accelerator, the first accelerator is connected to the first data processing unit, the second accelerator is connected to the second data processing unit, the first data processing unit is connected to the second data processing unit, the first accelerator and the second accelerator are of different types, and the first accelerator and the second accelerator are respectively used to run the Nth layer and the N+1th layer of the same model, where N is a positive integer; The data processing unit is used for obtaining a first value in response to the first accelerator running the Nth layer calculation, acquiring the first value, and sending a target value to the second accelerator through the second data processing unit, the target value being obtained based on the first value.

8. A server, characterized in that: including a first accelerator and a first processing unit; The first accelerator is connected to the first data processing unit, the first data processing unit is connected to the second data processing unit, the second data processing unit is connected to the second accelerator, the first accelerator and the second accelerator are of different types, and the first accelerator and the second accelerator are respectively used to run the Nth layer and the N+1th layer of the same model, where N is a positive integer; The first data processing unit is used for obtaining a first value in response to the first accelerator running the Nth layer calculation, acquiring the first value, and sending a target value to the second accelerator through the second data processing unit, the target value being obtained based on the first value.

9. The server according to claim 8, characterized in that: The server further includes a first network card, the first data processing unit and the second data processing unit are both CPUs, the first data processing unit is connected to the first network card, the first network card is communicatively connected to the second network card, and the second network card is connected to the second data processing unit; Or the first data processing unit and the second data processing unit are both DPUs.

10. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on a data processing unit in an accelerator cluster, enable the data processing unit to implement the data processing method according to any one of claims 1 to 6.

Images