A computing system, model training method, device and equipment, medium and product

By setting up a CXL interface between the computing device and the network card, efficient transmission of gradient data is achieved, which solves the problem of low gradient data transmission efficiency in distributed training and improves the training efficiency of artificial intelligence models.

CN118709745BActive Publication Date: 2024-11-26INSPUR (BEIJING) ELECTRONICS INFORMATION IND CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411162431.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-26
Estimated Expiration
2044-08-23

AI Technical Summary

Technical Problem

When training artificial intelligence models in distributed manner, the transmission efficiency of gradient data is low, resulting in inefficient model training.

Method used

Multiple computing nodes are used for data parallel training. By setting up a hardware interface that complies with the CXL protocol between the computing device and the network card, the efficient transmission of gradient data is achieved, and the cache consistency between the network card and the computing device is maintained through the device consistency maintenance engine. sex.

Benefits of technology

The gradient data transmission efficiency between computing devices and network cards is improved, thereby improving the training efficiency of artificial intelligence models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118709745B_ABST
    Figure CN118709745B_ABST
Patent Text Reader

Abstract

The present application discloses a computing system, a model training method, an apparatus and equipment, a medium and a product, and relates to the field of computer technology. The artificial intelligence computing system includes multiple computing nodes, and the multiple computing nodes include a host, a computing device and a network card; the computing device includes a first hardware interface, a first device consistency maintenance engine, and a first computing core, and the computing device is connected to the host through the first hardware interface; the network card includes a second hardware interface and a second device consistency maintenance engine, and the network card is connected to the host through the second hardware interface; the computing device transmits gradient data between the network card memory of the network card through a cache refresh operation, and the network card is used to synchronize gradient data with other computing nodes. The present application improves the transmission efficiency of the gradient data between the computing device and the network card, thereby improving the training efficiency of the artificial intelligence model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a computing system, a model training method, an apparatus and equipment, a medium and a product. Background Art

[0002] In related technologies, the network card and GPU are connected using a PCIe (Peripheral Component Interconnect Express) bus, and communication between the two is carried out using DMA (Direct Memory Access). The lengthy data communication path results in low transmission efficiency of gradient data during distributed training of artificial intelligence models.

[0003] Therefore, how to improve the transmission efficiency of gradient data during distributed training of artificial intelligence models, and thus improve the training efficiency of artificial intelligence models, is a technical problem that technical personnel in this field need to solve. Summary of the invention

[0004] The purpose of this application is to provide a computing system, a model training method, an apparatus and equipment, a medium and a product, which improve the transmission efficiency of gradient data during distributed training of artificial intelligence models, thereby improving the training efficiency of artificial intelligence models.

[0005] To achieve the above-mentioned purpose, the present application provides a computing system, including a plurality of computing nodes, wherein the plurality of computing nodes train an artificial intelligence model in a data parallel manner;

[0006] The computing node includes a host, a computing device and a network card; the computing device includes a first hardware interface, a first device consistency maintenance engine, and a first computing core. The computing device is connected to the host through the first hardware interface. The first device consistency maintenance engine is used to maintain a consistency state with the host. The first computing core is used to train an artificial intelligence model; the network card includes a second hardware interface and a second device consistency maintenance engine. The network card is connected to the host through the second hardware interface. The second device consistency maintenance engine is used to maintain a consistency state with the host.

[0007] The computing device transmits gradient data between the network card memory of the network card through a cache refresh operation, and the network card is used to synchronize gradient data with other computing nodes.

[0008] Wherein, the computing device is a graphics processing unit.

[0009] The computing device further includes a computing device memory connected to the first computing core and the first device consistency maintenance engine via a memory bus, and the computing device memory is used to store training data and model parameters.

[0010] The network card further includes a storage controller, which is used to access a network card memory corresponding to the network card, and the network card memory is used to cache gradient data.

[0011] Wherein, the host includes a host agent;

[0012] The process of transmitting gradient data between the computing device and the network card memory of the network card through the cache refresh operation is as follows: the computing device sends a first data request for transmitting gradient data to the host agent; the host agent modifies the request type of the first data request to obtain a second data request, and sends the second data request to the network card; the second device consistency maintenance engine in the network card obtains gradient data through the storage controller, encapsulates a first data response based on the gradient data, and sends the first data response to the host agent; the host agent modifies the response type of the first data response to obtain a second data response, and sends the second data response to the computing device; the computing device obtains gradient data based on the second data response.

[0013] Wherein, the network card further includes a second computing core, a network perception unit, and a network function unit;

[0014] The network function unit is used to implement a network processing function;

[0015] The network perception unit is used to obtain the current network status from the network function unit;

[0016] The second computing core is used to determine a compression scale based on the current network state, and compress the gradient data to be transmitted according to the compression scale, so that the network card transmits the compressed gradient data to other computing nodes.

[0017] Among them, the second computing core is also used to: read the gradient data calculated by all computing nodes cached in the network card memory through the second device consistency maintenance engine, and average the gradient data calculated by all computing nodes to obtain average gradient data, and cache the average gradient data in the network card memory.

[0018] Among them, the first computing core reads the average gradient data in the network card memory through the first device consistency maintenance engine and the second device consistency maintenance engine, and the first computing core reads the model parameters in the computing device memory and updates the model parameters according to the average gradient data.

[0019] Wherein, the first computing core directly accesses the computing device memory.

[0020] The second device consistency maintenance engine is further used for: performing conflict processing when the first computing core and the second computing core access the network card memory at the same time.

[0021] The host connects the first hardware interface in the computing device and the second hardware interface in the network card through a computing fast link switch.

[0022] Among them, the artificial intelligence model includes multiple levels. After the first computing core calculates and obtains the gradient data of a preset number of levels, the gradient data of the preset number of levels is transmitted to the network card through a cache refresh operation, so that the network card can synchronize the gradient data of the preset number of levels with other computing nodes.

[0023] To achieve the above objectives, the present application provides an artificial intelligence model training method, which is applied to a computing device in the above computing system, and the method includes:

[0024] Reading training data in the memory of the computing device, and forward-propagating the training artificial intelligence model based on the training data;

[0025] Back-propagating the calculated gradient data based on the result output by the artificial intelligence model, transmitting the gradient data to the network card through a cache refresh operation, so that the network card synchronizes the gradient data with other computing nodes to obtain average gradient data, and caches the average gradient data in the network card memory;

[0026] The model parameters in the computing device memory and the average gradient data in the network card memory are read, and the model parameters are updated based on the average gradient data.

[0027] Wherein, reading the average gradient data in the network card memory includes:

[0028] The average gradient data in the memory of the network card is read by a first device consistency maintenance engine in the computing device and a second device consistency maintenance engine in the network card.

[0029] The method of reading the average gradient data in the memory of the network card by using the first device consistency maintenance engine in the computing device and the second device consistency maintenance engine in the network card includes:

[0030] Sending a first data request for transmitting average gradient data to a host agent in a host through a first device consistency maintenance engine in the computing device, so that the host agent modifies a request type of the first data request to obtain a second data request, and sends the second data request to the network card;

[0031] Receive a second data response sent by the host agent; wherein the second device consistency maintenance engine in the network card obtains average gradient data through the storage controller, encapsulates a first data response based on the average gradient data, and sends the first data response to the host agent, and the host agent modifies the response type of the first data response to obtain the second data response;

[0032] Average gradient data is acquired based on the second data response.

[0033] The artificial intelligence model includes multiple levels, and gradient data is calculated by back propagation based on the result output by the artificial intelligence model, and the gradient data is transmitted to the network card through a cache refresh operation, so that the network card synchronizes the gradient data with other computing nodes to obtain average gradient data, and the average gradient data is cached in the network card memory, including:

[0034] After each calculation to obtain a preset number of levels of gradient data, the preset number of levels of gradient data are transmitted to the network card through a cache refresh operation, so that the network card and other computing nodes synchronize the preset number of levels of gradient data to obtain the preset number of levels of average gradient data, and cache the preset number of levels of average gradient data to the network card memory.

[0035] To achieve the above objectives, the present application provides an artificial intelligence model training device, which is applied to a computing device in the above computing system, and the device includes:

[0036] A forward propagation computing module, used to read the training data in the memory of the computing device and forward propagate the training artificial intelligence model based on the training data;

[0037] A back propagation calculation module, used for back propagating and calculating gradient data based on the result output by the artificial intelligence model, transmitting the gradient data to the network card through a cache refresh operation, so that the network card synchronizes the gradient data with other computing nodes to obtain average gradient data, and caches the average gradient data in the network card memory;

[0038] An updating module is used to read the model parameters in the memory of the computing device and the average gradient data in the memory of the network card, and update the model parameters based on the average gradient data.

[0039] To achieve the above objectives, the present application provides an electronic device, including:

[0040] Memory for storing computer programs;

[0041] A processor is used to implement the steps of the above-mentioned artificial intelligence model training method when executing the computer program.

[0042] To achieve the above-mentioned objectives, the present application provides a non-volatile storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned artificial intelligence model training method are implemented.

[0043] To achieve the above objectives, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned artificial intelligence model training method.

[0044] It can be seen from the above scheme that a computing system provided by the present application includes multiple computing nodes, and the multiple computing nodes use data parallelism to train artificial intelligence models; the computing nodes include a host, a computing device and a network card; the computing device includes a first hardware interface, a first device consistency maintenance engine, and a first computing core. The computing device is connected to the host through the first hardware interface, and the first device consistency maintenance engine is used to maintain a consistency state with the host, and the first computing core is used to train the artificial intelligence model; the network card includes a second hardware interface and a second device consistency maintenance engine. The network card is connected to the host through the second hardware interface, and the second device consistency maintenance engine is used to maintain a consistency state with the host; the computing device transmits gradient data between the network card memory of the network card through a cache refresh operation, and the network card is used to synchronize gradient data with other computing nodes.

[0045] The computing system provided by the present application sets a hardware interface compliant with CXL (Compute Express Link) in the computing device and the network card in each computing node. By realizing the transmission of gradient data between the computing device and the network card, the transmission efficiency of the gradient data between the computing device and the network card is improved, thereby improving the training efficiency of the artificial intelligence model. Furthermore, the computing device and the network card respectively maintain the consistency state with the host through the device consistency maintenance engine, thereby realizing cache consistency data communication between the network card and the computing device. The present application also discloses an artificial intelligence model training device, an electronic device, a non-volatile storage medium and a computer program product, which can also achieve the above-mentioned technical effects.

[0046] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. The drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific implementation methods, they are used to explain the present disclosure, but do not constitute a limitation to the present disclosure. In the drawings:

[0048] Figure 1 is a structural diagram of a computing node of a computing system according to an exemplary embodiment;

[0049] Figure 2 is a structural diagram of a computing device according to an exemplary embodiment;

[0050] Figure 3 is a structural diagram of a network card according to an exemplary embodiment;

[0051] Figure 4 A schematic diagram of a network card and a computing device accessing a network card memory according to an exemplary embodiment;

[0052] Figure 5 The present invention is a flowchart of an artificial intelligence model training method according to an exemplary embodiment;

[0053] Figure 6 is a structural diagram of an artificial intelligence model training device according to an exemplary embodiment;

[0054] Figure 7 The figure is a structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In addition, in the embodiments of the present application, "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0056] As the scale of artificial intelligence models continues to increase, the huge number of parameters poses a great challenge to the effective training of the models. A single GPU (Graphics Processing Unit) or even a single server node cannot store all the parameters of the model, so more machines are needed for distributed parallel training. Data parallelism usually stores a complete model on each machine, and the training data is divided into multiple copies according to the number of machines. After each computing node completes the training of its own part of the data, it will synchronize the model gradient data through network communication before conducting the next round of training. The network communication performance between computing nodes determines the performance of model training. The emergence of smart network card technology reduces the burden on the host by offloading the protocol processing in network communication to the network card, while improving the efficiency of network communication.

[0057] In related technologies, the network card and GPU are connected using a PCIe bus, and communication between the two is carried out using DMA. The lengthy data communication path results in low transmission efficiency of gradient data during distributed training of artificial intelligence models.

[0058] Therefore, hardware interfaces that comply with CXL are set in the computing devices and network cards in each computing node, and gradient data transmission is achieved between the computing devices and the network cards through CXL, which improves the transmission efficiency of gradient data between the computing devices and the network cards, thereby improving the training efficiency of the artificial intelligence model. Furthermore, the computing devices and network cards respectively maintain the consistency state with the host through the device consistency maintenance engine, realizing cache-consistent data communication between the network card and the computing devices.

[0059] The embodiment of the present application discloses a computing system, including a plurality of computing nodes, wherein the plurality of computing nodes train an artificial intelligence model in a data parallel manner;

[0060] The computing node includes a host, a computing device and a network card; the computing device includes a first hardware interface, a first device consistency maintenance engine, and a first computing core. The computing device is connected to the host through the first hardware interface. The first device consistency maintenance engine is used to maintain a consistency state with the host. The first computing core is used to train an artificial intelligence model; the network card includes a second hardware interface and a second device consistency maintenance engine. The network card is connected to the host through the second hardware interface. The second device consistency maintenance engine is used to maintain a consistency state with the host.

[0061] The computing device transmits gradient data between the network card memory of the network card through a cache refresh operation, and the network card is used to synchronize gradient data with other computing nodes.

[0062] In this embodiment, the artificial intelligence computing system includes multiple computing nodes, each computing node uses data parallelism to train the artificial intelligence model. During the training of the artificial intelligence model, the gradient data needs to be synchronized between the computing nodes. Figure 1 As shown, each computing node includes a host, and a computing device and a network card connected to the host. As a preferred embodiment, the host is connected to the first hardware interface in the computing device and the second hardware interface in the network card through a switch. That is, the host connects the computing device and the network card through a computing fast link switch (CXL Switch). The computing device can be a GPU, which is used to train an artificial intelligence model based on the training data of the computing node to which it belongs, calculate gradient data, and transmit the calculated gradient data to the network card. The network card is responsible for synchronizing the gradient data with other computing nodes, and then transmitting the synchronized gradient data to the computing device. The computing device updates the model parameters based on the received gradient data. In distributed model training, gradient data refers to the partial derivatives of the loss function calculated by each node with respect to the model parameters during the model parameter update process independently performed on multiple computing nodes. These gradient data will then be averaged to obtain average gradient data to guide the parameter update of the entire model, ensuring that the model copies on all nodes can work together and converge quickly to the optimal solution.

[0063] Computing equipment such as Figure 2 As shown, it includes a first hardware interface, a first device consistency maintenance engine (Device Coherence Engine, DCOH), and a first computing core. The computing device is connected to the host through the first hardware interface. The first hardware interface is an interface that complies with the CXL protocol. The first hardware interface is responsible for parsing the transaction layer messages of the three CXL sub-protocols CXL.io, CXL.cache, and CXL.mem. The first device consistency maintenance engine maintains consistency with the host through the CXL.cache protocol. The device consistency maintenance engine is an advanced system that ensures the stability and data consistency of distributed devices in a high-availability environment through key functions such as real-time monitoring, data synchronization, fault detection and transfer, configuration management, optimization algorithms, redundancy mechanisms, automated repair, log auditing, and user interfaces, thereby improving the performance and reliability of the overall system. The first computing core is responsible for training the artificial intelligence model based on the training data of the computing node to which it belongs, calculating the gradient data, and updating the model parameters based on the synchronized gradient data.

[0064] As a feasible implementation, the computing device further includes a computing device memory connected to the first computing core and the first device consistency maintenance engine via a memory bus, and the computing device memory is used to store training data and model parameters.

[0065] In a specific implementation, the first computing core is connected to the computing device memory via a memory bus, which includes multiple memory blocks. The first device consistency maintenance engine is connected to the computing device memory via the memory bus. The first device consistency maintenance engine needs to maintain the consistency state of the data on the computing device memory. At the same time, the first device consistency maintenance engine is connected to the first computing core to ensure the consistency state of the cached data in the first computing core.

[0066] Network card such as Figure 3 As shown, it includes a second hardware interface and a second device consistency maintenance engine. The network card is connected to the host through the second hardware interface. The second hardware interface is an interface that complies with the CXL protocol. The second hardware interface is responsible for parsing the transaction layer messages of the three CXL sub-protocols CXL.io, CXL.cache, and CXL.mem. The second device consistency maintenance engine maintains consistency with the host through the CXL.cache protocol.

[0067] As a feasible implementation, the network card also includes a storage controller, and the storage controller is used to access the network card memory corresponding to the network card, and the network card memory is used to cache gradient data. In a specific implementation, the network card includes a portion of on-chip memory space, namely the network card memory, which is accessed through the storage controller in the network card, and the network card memory is used to cache gradient data. The network card physical memory and the host physical memory are uniformly managed by the host. The device-side memory is called HDM (Host-Device Memory). The consistency of memory access between the host and the network card is maintained through CXL.cache. At this time, the management mode of the network card memory is HDM-D (Host-Device Memory with Direct Cache) mode.

[0068] As a feasible implementation mode, the network card further includes a second computing core, a network perception unit, and a network function unit; the network function unit is used to implement a network processing function; the network perception unit is used to obtain a current network state from the network function unit; the second computing core is used to determine a compression scale based on the current network state, and compress the gradient data to be transmitted according to the compression scale, so that the network card transmits the compressed gradient data to other computing nodes.

[0069] In a specific implementation, the network card also includes a second computing core, a network perception unit, and a network function unit. The network function unit is used to implement traditional network processing functions, such as parsing RDMA (Remote Direct Memory Access), TCP (Transmission Control Protocol), UDP messages, traffic management, data packet modification, etc. The network perception unit is used for the network function unit to collect the current network status, and the current network status can be cached in the network card memory. The second computing core is responsible for adjusting the compression scale of the gradient data to be transmitted according to the current network status. When the network environment is congested, a larger compression scale can be set, and when the network environment is good, a smaller compression scale can be set, thereby improving the transmission efficiency of the gradient data. For example, when the network environment is congested, it can be compressed to 8bit transmission, and when the network environment is good, it can be transmitted in the original 32bit format. The compression of the gradient data is the responsibility of the network card, and does not need to pass through the host, which reduces the burden on the host.

[0070] The CXL interface defines six channels of the CXL.cache protocol, including D2H Req (device-host request channel), D2H Resq (device-host response channel), D2H Data (device-host data channel), H2D Req (host-device request channel), H2D Resq (host-device response channel), and H2D Data (host-device data channel). The H in D2H or H2D represents the host host, and D represents the device device. The host of this embodiment can be a CPU (Central Processing Unit). In order to achieve consistent data communication between the network card and the computing device, such as the computing device accessing the memory on the network card in a consistent manner, it is necessary to forward messages through the host proxy module on the host side.

[0071] As a feasible implementation mode, the host includes a host agent; the process of transmitting gradient data between the computing device and the network card memory of the network card through the cache refresh operation is: the computing device sends a first data request for transmitting gradient data to the host agent; the host agent modifies the request type of the first data request to obtain a second data request, and sends the second data request to the network card; the second device consistency maintenance engine in the network card obtains gradient data through the storage controller, encapsulates a first data response based on the gradient data, and sends the first data response to the host agent; the host agent modifies the response type of the first data response to obtain a second data response, and sends the second data response to the computing device; the computing device obtains gradient data based on the second data response.

[0072] In a specific implementation, the computing device first requests the host for data on a physical address on the network card through the D2H Req channel. The host modifies the header of the D2H message. Since the system physical address is uniformly addressed, the address part does not need to be changed. The host modifies it into a message of the H2D Req type and forwards it to the network card. The second hardware interface on the network card parses the message and then forwards it to the second device consistency maintenance engine. After the second device consistency maintenance engine obtains the data by communicating with the storage controller, it encapsulates it into D2H Resp and D2H Data messages and returns them to the host. The host modifies the header to obtain H2D Resp and H2D Data messages and returns them to the computing device.

[0073] In the process of transmitting gradient data between the computing device and the network card, the computing device and the network card can communicate through the CXL.cache protocol, and the gradient is automatically transferred to the network card memory through the cache flush operation. At this time, the network card memory is equivalent to the secondary storage of the computing device cache. In this way, the lengthy data transmission process of gradient data from computing device cache → computing device memory → network card memory is avoided, thereby improving the transmission efficiency of gradient data.

[0074] As a feasible implementation, the second computing core is also used to: read the gradient data calculated by all computing nodes cached in the network card memory through the second device consistency maintenance engine, and average the gradient data calculated by all computing nodes to obtain average gradient data, and cache the average gradient data in the network card memory. In a specific implementation, the network card will average the gradient data after receiving the gradient data from other computing nodes, and then cache the calculated average gradient data in the network card memory. Compared with related technologies, the gradient information that needs to be synchronized between machines is stored in the network card memory, and the network card memory is used to indirectly expand the size of the memory available to the computing device.

[0075] As a feasible implementation, the first computing core reads the average gradient data in the network card memory through the first device consistency maintenance engine and the second device consistency maintenance engine, the first computing core reads the model parameters in the computing device memory, and updates the model parameters according to the average gradient data. The first computing core directly accesses the computing device memory.

[0076] In the specific implementation, Figure 4As shown, the second computing core in the network card needs to be forwarded through the second device consistency maintenance engine when accessing its own network card memory, and the first computing core in the computing device needs to be forwarded through the first device consistency maintenance engine and the second device consistency maintenance engine when accessing the network card memory. The first computing core in the computing device can directly access its own computing device memory without being forwarded by the first device consistency maintenance engine.

[0077] As a feasible implementation manner, the second device consistency maintenance engine is further used to: perform conflict processing when the first computing core and the second computing core access the network card memory at the same time.

[0078] In a specific implementation, when the computing device and the network card access the network card memory at the same time, the second device consistency maintenance engine in the network card manages the conflict situation where the network card memory is accessed by both the network card and the computing device at the same time. Through the CXL.cache communication between the second device consistency maintenance engine and the first device consistency maintenance engine, if a memory page of the network card memory is accessed by the first computing core in the computing device, then the second computing core in the network card will not be able to access the memory page.

[0079] As a feasible implementation mode, the artificial intelligence model includes multiple levels. After the first computing core calculates and obtains the gradient data of a preset number of levels, the gradient data of the preset number of levels is transmitted to the network card through a cache refresh operation, so that the network card synchronizes the gradient data of the preset number of levels with other computing nodes.

[0080] It is understandable that during the training process of the artificial intelligence model, after the computing device forward calculates the loss value, it will calculate the gradient data of each level from back to front according to the model network level. When the model network depth is large, it is not necessary to calculate the gradient at one time and then synchronize the gradient data of all levels through the network. After calculating a preset number of levels, it can be synchronized in batches through the network card with other computing nodes to improve the synchronization efficiency of the gradient data.

[0081] The training process of the computing devices and network cards in each computing node participating in the artificial intelligence model is as follows: In the initialization phase, the network card will store the corresponding gradients according to the scale of the model and set all the gradient values ​​to 0. After the computing device completes the forward propagation, each time the computing device calculates a layer of gradients during the backward propagation, it can automatically synchronize to the network card memory through the cache flush operation. The network card memory runs the gradient compression strategy according to the network conditions, and then synchronizes with other machines in real time. At the same time, the gradient information of other machines will also be received in real time on the current machine. At this time, taking the synchronization operator of Ring AllReduce as an example, the network card will process the message in real time after receiving it. After the gradient synchronization is completed, the network card averages all the collected gradients, and the computing device simultaneously reads the gradient information in the network card memory and adds it to the model parameters in the computing device memory to complete the update operation of the model parameters, and start the model training of the next batch.

[0082] The computing system provided in the embodiment of the present application sets a hardware interface compliant with CXL in the computing device and the network card in each computing node, and realizes the transmission of gradient data between the computing device and the network card through CXL, thereby improving the transmission efficiency of the gradient data between the computing device and the network card, and thus improving the training efficiency of the artificial intelligence model. Furthermore, the computing device and the network card respectively maintain the consistency state with the host through the device consistency maintenance engine, thereby realizing the data communication of cache consistency between the network card and the computing device.

[0083] The present application embodiment discloses an artificial intelligence model training method, see Figure 5 , according to a flowchart of an artificial intelligence model training method shown in an exemplary embodiment, as shown in Figure 5 As shown, including:

[0084] S101: Reading training data in the memory of a computing device, and forward-propagating training of an artificial intelligence model based on the training data;

[0085] The executor of this embodiment is a computing device in a computing node, such as a GPU. The computing device includes a first hardware interface, a first device consistency maintenance engine, and a first computing core. The computing device is connected to the host through a first hardware interface. The first hardware interface is an interface that complies with the CXL protocol. The first hardware interface is responsible for parsing the transaction layer messages of the three CXL sub-protocols CXL.io, CXL.cache, and CXL.mem. The first device consistency maintenance engine maintains consistency with the host through the CXL.cache protocol. The first computing core is responsible for training the artificial intelligence model based on the training data of the computing node to which it belongs, calculating the gradient data, and updating the model parameters based on the synchronized gradient data.

[0086] In a specific implementation, the computing device reads the training data from the computing device memory and uses the training data to train the artificial intelligence model through a forward propagation process. Forward propagation is a basic step in a neural network, in which input data is passed through each layer of the model to generate an output result.

[0087] As a feasible implementation, the computing device also includes a computing device memory connected to the first computing core and the first device consistency maintenance engine via a memory bus, and the computing device memory is used to store training data and model parameters. In a specific implementation, the first computing core is connected to the computing device memory via a memory bus, which includes a plurality of memory blocks, and the first device consistency maintenance engine is connected to the computing device memory via a memory bus. The first device consistency maintenance engine needs to maintain the consistency state of the data on the computing device memory, and at the same time, the first device consistency maintenance engine is connected to the first computing core to ensure the consistency state of the cached data in the first computing core.

[0088] S102: Back propagating gradient data based on the result output by the artificial intelligence model, transmitting the gradient data to the network card through a cache refresh operation, so that the network card synchronizes the gradient data with other computing nodes to obtain average gradient data, and caches the average gradient data in the network card memory;

[0089] In the specific implementation, the computing device performs back propagation calculations based on the output results of the artificial intelligence model. Back propagation is a key step in training a neural network and is used to calculate the gradient of the loss function relative to the model parameters. The calculated gradient data is transmitted to the network card through the cache refresh operation of the CXL (Compute Express Link) protocol. The CXL protocol allows high-speed, low-latency data transmission, thereby improving the efficiency of gradient synchronization. The network card synchronizes gradient data with other computing nodes, then calculates the average gradient data, and caches the calculated average gradient data in the memory of the network card.

[0090] The network card includes a second hardware interface and a second device consistency maintenance engine. The network card is connected to the host through the second hardware interface. The second hardware interface is an interface that complies with the CXL protocol. The second hardware interface is responsible for parsing transaction layer messages of the three CXL sub-protocols CXL.io, CXL.cache, and CXL.mem. The second device consistency maintenance engine maintains consistency with the host through the CXL.cache protocol.

[0091] As a feasible implementation, the network card also includes a storage controller, and the storage controller is used to access the network card memory corresponding to the network card, and the network card memory is used to cache gradient data. In a specific implementation, the network card includes a portion of on-chip memory space, namely the network card memory, which is accessed through the storage controller in the network card, and the network card memory is used to cache gradient data. The network card physical memory and the host physical memory are uniformly managed by the host. The device-side memory is called HDM. The consistency of memory access between the host and the network card is maintained through CXL.cache. At this time, the management mode of the network card memory is HDM-D mode.

[0092] As a feasible implementation mode, the network card further includes a second computing core, a network perception unit, and a network function unit; the network function unit is used to implement a network processing function; the network perception unit is used to obtain a current network state from the network function unit; the second computing core is used to determine a compression scale based on the current network state, and compress the gradient data to be transmitted according to the compression scale, so that the network card transmits the compressed gradient data to other computing nodes.

[0093] In a specific implementation, the network card also includes a second computing core, a network perception unit, and a network function unit. The network function unit is used to implement traditional network processing functions, such as parsing RDMA, TCP, UDP messages, traffic management, data packet modification, etc. The network perception unit is used for the network function unit to collect the current network status, and the current network status can be cached in the network card memory. The second computing core is responsible for adjusting the compression scale of the gradient data to be transmitted according to the current network status. When the network environment is congested, a larger compression scale can be set, and when the network environment is good, a smaller compression scale can be set. For example, when the network environment is congested, it can be compressed to 8bit transmission, and when the network environment is good, it can be transmitted in the original 32bit format. The compression of the gradient data is the responsibility of the network card, and does not need to pass through the host, which reduces the burden on the host.

[0094] S103: Read the model parameters in the computing device memory and the average gradient data in the network card memory, and update the model parameters based on the average gradient data.

[0095] In a specific implementation, the computing device reads the current model parameters from the computing device memory, reads the average gradient data of each layer calculated previously from the network card memory, and updates the model parameters of each layer based on the average gradient data of each layer to reduce the loss value of the model and improve the prediction accuracy of the model.

[0096] As a feasible implementation, reading the average gradient data in the network card memory includes: reading the average gradient data in the network card memory through a first device consistency maintenance engine in the computing device and a second device consistency maintenance engine in the network card. In a specific implementation, when the first computing core in the computing device accesses the network card memory, it needs to be forwarded through the first device consistency maintenance engine and the second device consistency maintenance engine.

[0097] The CXL interface defines six channels of the CXL.cache protocol, including D2H Req (device-host request channel), D2H Resq (device-host response channel), D2H Data (device-host data channel), H2D Req (host-device request channel), H2D Resq (host-device response channel), and H2D Data (host-device data channel). The H in D2H or H2D represents the host host, and D represents the device device. The host of this embodiment can be a CPU (Central Processing Unit). In order to achieve consistent data communication between the network card and the computing device, such as the computing device accessing the memory on the network card in a consistent manner, it is necessary to forward messages through the host proxy module on the host side.

[0098] As a feasible implementation, the average gradient data in the network card memory is read through the first device consistency maintenance engine in the computing device and the second device consistency maintenance engine in the network card, including: sending a first data request for transmitting the average gradient data to the host agent in the host through the first device consistency maintenance engine in the computing device, so that the host agent modifies the request type of the first data request to obtain a second data request, and sends the second data request to the network card; receiving a second data response sent by the host agent; wherein the second device consistency maintenance engine in the network card obtains the average gradient data through the storage controller, encapsulates a first data response based on the average gradient data, and sends the first data response to the host agent, and the host agent modifies the response type of the first data response to obtain the second data response; and obtains the average gradient data based on the second data response.

[0099] In a specific implementation, the computing device first requests the host for the average gradient data on a physical address on the network card through the D2H Req channel. The host modifies the header of the D2H message. Since the system physical address is uniformly addressed, the address part does not need to be changed. The host modifies it into a message of the H2D Req type and forwards it to the network card. The second hardware interface on the network card parses the message and then forwards it to the second device consistency maintenance engine. After the second device consistency maintenance engine obtains the average gradient data by communicating with the storage controller, it encapsulates it into D2H Resp and D2HData messages and returns them to the host. The host modifies the header to obtain H2D Resp and H2D Data messages and returns them to the computing device. The computing device parses the received message to obtain the average gradient data.

[0100] In the process of transmitting gradient data between the computing device and the network card, the computing device and the network card can communicate through the CXL.cache protocol, and the gradient is automatically transferred to the network card memory through the cache flush operation. At this time, the network card memory is equivalent to the secondary storage of the computing device cache. In this way, the lengthy data transmission process of gradient data from computing device cache → computing device memory → network card memory is avoided, and the transmission efficiency of average gradient data is improved.

[0101] As a feasible implementation mode, the artificial intelligence model includes multiple levels, and gradient data is calculated by back propagation based on the results output by the artificial intelligence model. The gradient data is transmitted to the network card through a cache refresh operation so that the network card synchronizes the gradient data with other computing nodes to obtain average gradient data, and the average gradient data is cached in the network card memory, including: after each calculation to obtain a preset number of levels of gradient data, the gradient data of a preset number of levels is transmitted to the network card through a cache refresh operation so that the network card synchronizes the gradient data of a preset number of levels with other computing nodes to obtain the average gradient data of a preset number of levels, and the average gradient data of the preset number of levels is cached in the network card memory.

[0102] In specific implementations, when the model network depth is large, it is not necessary to calculate the gradients at one time and then synchronize the gradient data of all levels through the network. After calculating a preset number of levels, the data can be synchronized in batches through the network card with other computing nodes to improve the synchronization efficiency of the gradient data.

[0103] It can be seen that in this embodiment, the gradient data is transmitted between the computing device and the network card in each computing node through CXL, which improves the transmission efficiency of the gradient data between the computing device and the network card, and further improves the training efficiency of the artificial intelligence model.

[0104] An artificial intelligence model training device provided in an embodiment of the present application is introduced below. The artificial intelligence model training device described below and the artificial intelligence model training method described above can be referenced to each other.

[0105] See also Figure 6 , according to a structural diagram of an artificial intelligence model training device shown in an exemplary embodiment, such as Figure 6 As shown, including:

[0106] A forward propagation computing module 100 is used to read the training data in the memory of the computing device and to forward propagate the training artificial intelligence model based on the training data;

[0107] A back propagation calculation module 200 is used to calculate gradient data by back propagation based on the result output by the artificial intelligence model, transmit the gradient data to the network card through a cache refresh operation, so that the network card synchronizes the gradient data with other computing nodes to obtain average gradient data, and cache the average gradient data to the network card memory;

[0108] The updating module 300 is used to read the model parameters in the memory of the computing device and the average gradient data in the memory of the network card, and update the model parameters based on the average gradient data.

[0109] It can be seen that in this embodiment, the gradient data is transmitted between the computing device and the network card in each computing node through CXL, which improves the transmission efficiency of the gradient data between the computing device and the network card, and further improves the training efficiency of the artificial intelligence model.

[0110] Based on the above embodiment, as a preferred implementation, the update module 300 is specifically used to: read the average gradient data in the network card memory through the first device consistency maintenance engine in the computing device and the second device consistency maintenance engine in the network card.

[0111] On the basis of the above embodiments, as a preferred implementation mode, the update module 300 is specifically used for: sending a first data request for transmitting average gradient data to a host agent in a host through a first device consistency maintenance engine in the computing device, so that the host agent modifies the request type of the first data request to obtain a second data request, and sends the second data request to the network card; receiving a second data response sent by the host agent; wherein the second device consistency maintenance engine in the network card obtains the average gradient data through a storage controller, encapsulates a first data response based on the average gradient data, and sends the first data response to the host agent, and the host agent modifies the response type of the first data response to obtain the second data response; and obtains the average gradient data based on the second data response.

[0112] On the basis of the above embodiments, as a preferred implementation mode, the artificial intelligence model includes multiple levels, and the back propagation calculation module 200 is specifically used for: after each calculation to obtain the gradient data of a preset number of levels, the gradient data of the preset number of levels are transmitted to the network card through a cache refresh operation, so that the network card synchronizes the gradient data of the preset number of levels with other computing nodes to obtain the average gradient data of the preset number of levels, and caches the average gradient data of the preset number of levels to the network card memory.

[0113] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0114] Based on the hardware implementation of the above program module, and in order to implement the method of the embodiment of the present application, the embodiment of the present application also provides an electronic device, Figure 7 FIG. 1 is a structural diagram of an electronic device according to an exemplary embodiment. Figure 7 As shown, the electronic equipment includes:

[0115] Communication interface 1, capable of exchanging information with other devices such as network devices;

[0116] The processor 2 is connected to the communication interface 1 to realize information interaction with other devices, and is used to execute the artificial intelligence model training method provided by one or more technical solutions when running the computer program. The computer program is stored in the memory 3.

[0117] Of course, in actual application, the various components in the electronic device are coupled together through the bus system 4. It can be understood that the bus system 4 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 4 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 7 Various buses are labeled as bus system 4.

[0118] The memory 3 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program used to operate on the electronic device.

[0119] It can be understood that the memory 3 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 3 described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memories.

[0120] The method disclosed in the above embodiment of the present application can be applied to the processor 2, or implemented by the processor 2. The processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 2 or the instruction in the form of software. The above processor 2 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 2 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in the memory 3, and the processor 2 reads the program in the memory 3 and completes the steps of the above method in combination with its hardware.

[0121] When the processor 2 executes the program, the corresponding processes in the various methods of the embodiments of the present application are implemented, which will not be repeated here for the sake of brevity.

[0122] In an exemplary embodiment, the embodiment of the present application further provides a non-volatile storage medium storing a computer program, which can be executed by the processor 2 to complete the steps of the aforementioned method.

[0123] In an exemplary embodiment, the present application also provides a computer program product, including a computer program, which is executed by the processor 2 to complete the steps of the aforementioned method.

[0124] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to computer program instructions, and the aforementioned computer program can be stored in a non-volatile storage medium. When the computer program is executed, it executes the steps of the above-mentioned method embodiment. Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product, which is stored in a non-volatile storage medium and includes a number of instructions for an electronic device (which can be a personal computer, a server, a network device, etc.) to execute all or part of the methods described in each embodiment of the present application.

[0125] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A computing system, characterized in that: It includes a plurality of computing nodes, and the plurality of computing nodes train an artificial intelligence model in a data parallel manner; The computing node includes a host, a computing device and a network card; the computing device includes a first hardware interface, a first device consistency maintenance engine, and a first computing core. The computing device is connected to the host through the first hardware interface. The first device consistency maintenance engine is used to maintain a consistency state with the host. The first computing core is used to train an artificial intelligence model; the network card includes a second hardware interface and a second device consistency maintenance engine. The network card is connected to the host through the second hardware interface. The second device consistency maintenance engine is used to maintain a consistency state with the host. The computing device transmits gradient data between the network card memory of the network card through a cache refresh operation, and the network card is used to synchronize gradient data with other computing nodes; The network card further includes a storage controller, the storage controller is used to access the network card memory corresponding to the network card, and the network card memory is used to cache gradient data; The host includes a host agent; The process of transmitting gradient data between the computing device and the network card memory of the network card through the cache refresh operation is as follows: the computing device sends a first data request for transmitting gradient data to the host agent; the host agent modifies the request type of the first data request to obtain a second data request, and sends the second data request to the network card; the second device consistency maintenance engine in the network card obtains gradient data through the storage controller, encapsulates a first data response based on the gradient data, and sends the first data response to the host agent; the host agent modifies the response type of the first data response to obtain a second data response, and sends the second data response to the computing device; the computing device obtains gradient data based on the second data response.

2. The computing system according to claim 1, characterized in that: The computing device is a graphics processing unit.

3. The computing system according to claim 1, characterized in that: The computing device also includes a computing device memory connected to the first computing core and the first device consistency maintenance engine via a memory bus, and the computing device memory is used to store training data and model parameters.

4. The computing system according to claim 1, characterized in that: The network card also includes a second computing core, a network perception unit, and a network function unit; The network function unit is used to implement a network processing function; The network perception unit is used to obtain the current network status from the network function unit; The second computing core is used to determine a compression scale based on the current network state, and compress the gradient data to be transmitted according to the compression scale, so that the network card transmits the compressed gradient data to other computing nodes.

5. The computing system according to claim 4, characterized in that: The second computing core is also used to: read the gradient data calculated by all computing nodes cached in the network card memory through the second device consistency maintenance engine, average the gradient data calculated by all computing nodes to obtain average gradient data, and cache the average gradient data in the network card memory.

6. The computing system according to claim 5, characterized in that: The first computing core reads the average gradient data in the network card memory through the first device consistency maintenance engine and the second device consistency maintenance engine. The first computing core reads the model parameters in the computing device memory and updates the model parameters according to the average gradient data.

7. The computing system according to claim 6, characterized in that: The first computing core directly accesses the computing device memory.

8. The computing system according to claim 6, characterized in that: The second device consistency maintenance engine is further used for: performing conflict processing when the first computing core and the second computing core access the network card memory at the same time.

9. The computing system according to claim 1, characterized in that: The host is connected to the first hardware interface in the computing device and the second hardware interface in the network card through a switch.

10. The computing system according to claim 1, characterized in that: The artificial intelligence model includes multiple levels. After the first computing core calculates and obtains gradient data of a preset number of levels, the gradient data of the preset number of levels is transmitted to the network card through a cache refresh operation, so that the network card can synchronize the gradient data of the preset number of levels with other computing nodes.

11. An artificial intelligence model training method, characterized in that: The method applied to the computing device in the computing system according to any one of claims 1 to 10 comprises: Reading training data in the memory of the computing device, and forward-propagating the training artificial intelligence model based on the training data; Back-propagating the calculated gradient data based on the result output by the artificial intelligence model, transmitting the gradient data to the network card through a cache refresh operation, so that the network card synchronizes the gradient data with other computing nodes to obtain average gradient data, and caches the average gradient data in the network card memory; The model parameters in the computing device memory and the average gradient data in the network card memory are read, and the model parameters are updated based on the average gradient data.

12. The artificial intelligence model training method according to claim 11, characterized in that: Reading the average gradient data in the network card memory includes: The average gradient data in the memory of the network card is read by a first device consistency maintenance engine in the computing device and a second device consistency maintenance engine in the network card.

13. The artificial intelligence model training method according to claim 12, characterized in that: Reading the average gradient data in the memory of the network card through a first device consistency maintenance engine in the computing device and a second device consistency maintenance engine in the network card includes: Sending a first data request for transmitting average gradient data to a host agent in a host through a first device consistency maintenance engine in the computing device, so that the host agent modifies a request type of the first data request to obtain a second data request, and sends the second data request to the network card; Receive a second data response sent by the host agent; wherein the second device consistency maintenance engine in the network card obtains average gradient data through the storage controller, encapsulates a first data response based on the average gradient data, and sends the first data response to the host agent, and the host agent modifies the response type of the first data response to obtain the second data response; Average gradient data is acquired based on the second data response.

14. The artificial intelligence model training method according to claim 11, characterized in that: The artificial intelligence model includes multiple levels, and gradient data is calculated by back propagation based on the result output by the artificial intelligence model, and the gradient data is transmitted to the network card through a cache refresh operation, so that the network card synchronizes the gradient data with other computing nodes to obtain average gradient data, and the average gradient data is cached in the network card memory, including: After each calculation to obtain a preset number of levels of gradient data, the preset number of levels of gradient data are transmitted to the network card through a cache refresh operation, so that the network card and other computing nodes synchronize the preset number of levels of gradient data to obtain the preset number of levels of average gradient data, and cache the preset number of levels of average gradient data to the network card memory.

15. An artificial intelligence model training device, characterized in that: A computing device applied to a system as claimed in any one of claims 1 to 10, wherein the apparatus comprises: A forward propagation computing module, used to read the training data in the memory of the computing device and forward propagate the training artificial intelligence model based on the training data; A back propagation calculation module, used for back propagating and calculating gradient data based on the result output by the artificial intelligence model, transmitting the gradient data to the network card through a cache refresh operation, so that the network card synchronizes the gradient data with other computing nodes to obtain average gradient data, and caches the average gradient data in the network card memory; An updating module is used to read the model parameters in the memory of the computing device and the average gradient data in the memory of the network card, and update the model parameters based on the average gradient data.

16. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the artificial intelligence model training method as described in any one of claims 11 to 14 when executing the computer program.

17. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, which, when executed, implements the steps of the artificial intelligence model training method as described in any one of claims 11 to 14.

18. A computer program product, characterized in that The computer program product includes a computer program, which, when executed, implements the steps of the artificial intelligence model training method as described in any one of claims 11 to 14.

Citation Information

Patent Citations

  • Method and device for controlling shared memory

    CN102651690A

  • Method and system for transmitting data between computing nodes and electronic equipment

    CN109828843A

  • Data transmission method, device and system and computer equipment

    CN116737069A

  • Data transmission method, device and system

    CN118018481A

  • Maintaining consistency of data between computing nodes of a distributed computer architecture

    US20210042292A1