Computing system, model training method and apparatus, and device, medium and product

Abstract

The present application relates to the technical field of computers. Disclosed are a computing system, a model training method and apparatus, and a device, a medium and a product. The artificial intelligence computing system comprises a plurality of computing nodes. Each computing node comprises a host, a computing device and a network interface card, wherein the computing device comprises a first hardware interface, a first device coherence engine and a first computing core, and is connected to the host by means of the first hardware interface; the network interface card comprises a second hardware interface and a second device coherence engine, and is connected to the host by means of the second hardware interface; and by means of a cache flush operation, the computing device transmits gradient data with a network interface card memory of the network interface card, and the network interface card is used for synchronizing the gradient data with other computing nodes. The present application improves the efficiency of transmitting gradient data between a computing device and a network interface card, thus improving the training efficiency for an artificial intelligence model.

Description

A computing system, model training method, device and equipment, medium and product

[0001] Cross-reference to Related Applications

[0002] The present application claims priority to the Chinese patent application No. 202411162431.8, filed on August 23, 2024, and entitled "A computing system, model training method, device and equipment, medium and product", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of computers, and in particular to a computing system, model training method, device and equipment, medium and product. BACKGROUND

[0004] In the related art, a PCIe bus is used to connect a network card and a GPU (Graphics Processing Unit), and the communication between the two uses a DMA (Direct Memory Access) mode, and the long data communication path results in low transmission efficiency of gradient data when training an artificial intelligence model in a distributed manner.

[0005] Therefore, how to solve the problems of low transmission efficiency of gradient data and low training efficiency of an artificial intelligence model when training an artificial intelligence model in a distributed manner. SUMMARY

[0006] The present application provides a computing system, model training method, device and equipment, medium and product.

[0007] The present application provides a computing system, comprising a plurality of computing nodes, and the plurality of computing nodes train an artificial intelligence model in a data parallel manner.

[0008] The computing node comprises a host, a computing device and a network card; the computing device comprises a first hardware interface, a first device consistency maintenance engine and a first computing core, the computing device is connected to the host through the first hardware interface, the first device consistency maintenance engine is used to maintain a consistency state with the host, and the first computing core is used to train an artificial intelligence model; the network card comprises a second hardware interface and a second device consistency maintenance engine, the network card is connected to the host through the second hardware interface, and the second device consistency maintenance engine is used to maintain a consistency state with the host; and

[0009] The computing device transmits gradient data between the refresh operation of the cache and the network card memory of the network card, and the network card is used to synchronize the gradient data with other computing nodes.

[0010] The computing device is a graphics processing unit.

[0011] The computing device further includes a computing device memory connected to the first computing core and the first device consistency maintenance engine through a memory bus, and the computing device memory is used to store training data and model parameters.

[0012] The network card further includes a storage controller, and the storage controller is used to access the network card memory corresponding to the network card, and the network card memory is used to cache gradient data.

[0013] The host includes a host agent.

[0014] The process in which the computing device transmits gradient data between the refresh operation of the cache and the network card memory of the network card is as follows: the computing device sends a first data request for transmitting gradient data to the host agent; the host agent modifies the request type of the first data request to obtain a second data request, and sends the second data request to the network card; the second device consistency maintenance engine in the network card obtains gradient data through the storage controller, encapsulates a first data response based on the gradient data, and sends the first data response to the host agent; the host agent modifies the response type of the first data response to obtain a second data response, and sends the second data response to the computing device; and the computing device obtains gradient data based on the second data response.

[0015] The network card further includes a second computing core, a network perception unit, and a network function unit.

[0016] The network function unit is used to implement network processing functions.

[0017] The network perception unit is used to obtain a current network state from the network function unit; and

[0018] The second computing core is used to determine a compression scale based on the current network state, compress gradient data to be transmitted according to the compression scale, so that the network card transmits the compressed gradient data to other computing nodes.

[0019] The second computing core is further used to: read gradient data calculated by the computing node and cached in the network card memory through the second device consistency maintenance engine, average the gradient data calculated by the computing node to obtain average gradient data, and cache the average gradient data in the network card memory.

[0020] The first computing core reads the average gradient data in the network card memory through the first device consistency maintenance engine and the second device consistency maintenance engine; and the first computing core reads the model parameters in the computing device memory and updates the model parameters according to the average gradient data.

[0021] The first computing core directly accesses the computing device memory.

[0022] The second device consistency maintenance engine is further configured to perform conflict processing in response to determining that the first computing core and the second computing core simultaneously access the network card memory.

[0023] The host is connected to the first hardware interface in the computing device and the second hardware interface in the network card through a computing fast link switch.

[0024] The artificial intelligence model includes a plurality of levels, and the first computing core transmits the gradient data of the preset number of levels to the network card through a refresh operation of the cache after calculating the gradient data of the preset number of levels, so that the network card synchronizes the gradient data of the preset number of levels with other computing nodes.

[0025] The present application provides an artificial intelligence model training method, applied to a computing device in the above computing system, the method comprising:

[0026] reading the training data in the computing device memory and forward propagating the training artificial intelligence model based on the training data;

[0027] calculating gradient data based on the results output by the artificial intelligence model, transmitting the gradient data to the network card through a refresh operation of the cache, so that the network card synchronizes the gradient data with other computing nodes to obtain average gradient data, and caching the average gradient data to the network card memory; and

[0028] reading the model parameters in the computing device memory and the average gradient data in the network card memory, and updating the model parameters based on the average gradient data.

[0029] The average gradient data in the network card memory is read, including:

[0030] The average gradient data in the network card memory is read through the first device consistency maintenance engine in the computing device and the second device consistency maintenance engine in the network card.

[0031] The average gradient data in the network card memory is read through the first device consistency maintenance engine in the computing device and the second device consistency maintenance engine in the network card, including:

[0032] sending, by a first device coherency maintenance engine in the computing device, a first data request to transmit average gradient data to a host agent in a host, so that the host agent modifies a request type of the first data request to obtain a second data request, and sends the second data request to a network interface card;

[0033] receiving a second data response sent by the host agent; wherein the second device coherency maintenance engine in the network interface card acquires the average gradient data through the storage controller, encapsulates the first data response based on the average gradient data, and sends the first data response to the host agent, and the host agent modifies a response type of the first data response to obtain the second data response; and

[0034] acquiring the average gradient data based on the second data response.

[0035] The artificial intelligence model includes a plurality of levels, the gradient data is calculated based on a result output by the artificial intelligence model, the gradient data is transmitted to the network interface card through a refresh operation of the cache, so that the network interface card synchronizes the gradient data with other computing nodes to obtain the average gradient data, and the average gradient data is cached in the network interface card memory, including:

[0036] After a preset number of levels of gradient data are calculated, the preset number of levels of gradient data are transmitted to the network interface card through a refresh operation of the cache, so that the network interface card synchronizes the preset number of levels of gradient data with other computing nodes to obtain the average gradient data of the preset number of levels, and the average gradient data of the preset number of levels is cached in the network interface card memory.

[0037] The present application provides an artificial intelligence model training device, which is applied to a computing device in the above computing system, and the device includes:

[0038] a forward propagation calculation module configured to read training data in a memory of the computing device, and forward propagate training of the artificial intelligence model based on the training data;

[0039] a backward propagation calculation module configured to calculate gradient data based on a result output by the artificial intelligence model, transmit the gradient data to the network interface card through a refresh operation of the cache, so that the network interface card synchronizes the gradient data with other computing nodes to obtain the average gradient data, and cache the average gradient data in the network interface card memory; and

[0040] an updating module configured to read model parameters in the memory of the computing device and the average gradient data in the network interface card memory, and update the model parameters based on the average gradient data.

[0041] The present application provides an electronic device, including:

[0042] one or more processors; and

[0043] A memory associated with the one or more processors, the memory to store computer-readable instructions that, when read and executed by the one or more processors, implement the steps of the artificial intelligence model training method as described above.

[0044] The present application provides a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium has computer-readable instructions stored thereon, the computer-readable instructions, when executed by one or more processors, implement the steps of the artificial intelligence model training method as described above.

[0045] The present application provides a computer program product comprising a computer program, the computer program, when executed by one or more processors, implements the steps of the artificial intelligence model training method as described above.

[0046] From the above scheme, the present application provides a computing system, comprising a plurality of computing nodes, the plurality of computing nodes train an artificial intelligence model in a data parallel manner; the computing node comprises a host, a computing device and a network card; the computing device comprises a first hardware interface, a first device consistency maintenance engine and a first computing core, the computing device is connected to the host through the first hardware interface, the first device consistency maintenance engine is used to maintain a consistency state with the host, and the first computing core is used to train the artificial intelligence model; the network card comprises a second hardware interface and a second device consistency maintenance engine, the network card is connected to the host through the second hardware interface, and the second device consistency maintenance engine is used to maintain a consistency state with the host; the computing device transmits gradient data between the cache and the network card memory of the network card through a refresh operation of the cache, and the network card is used to synchronize the gradient data with other computing nodes.

[0047] It should be understood that the above general description and the following detailed description are only exemplary and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor. The drawings are used to provide further understanding of the present disclosure and constitute a part of the specification. Together with the following embodiments, they are used to explain the present disclosure but do not constitute a limitation on the present disclosure. In the drawings:

[0049] Fig. 1 is a structural diagram of a computing node of a computing system according to an embodiment of the present application;

[0050] Fig. 2 is a structural diagram of a computing device according to an embodiment of the present application;

[0051] FIG. 3 is a structural diagram of a network card according to an embodiment of the present application;

[0052] FIG. 4 is a schematic diagram of a network card and a computing device accessing memory of the network card according to an embodiment of the present application;

[0053] FIG. 5 is a flowchart of a method for training an artificial intelligence model according to an embodiment of the present application;

[0054] FIG. 6 is a structural diagram of an apparatus for training an artificial intelligence model according to an embodiment of the present application;

[0055] FIG. 7 is a structural diagram of an electronic device according to an embodiment of the present application;

[0056] FIG. 8 is a structural diagram of a non-volatile computer readable storage medium according to an embodiment of the present application;

[0057] FIG. 9 is a structural diagram of a computer program product according to an embodiment of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application. In addition, in the embodiments of the present application, “first”, “second”, etc. are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0059] With the increasing size of artificial intelligence models, a huge number of parameters bring great challenges to the effective training of the models. A single GPU (Graphics Processing Unit) or even a single server node cannot store all the parameters of the model, so more machines are needed for distributed parallel training. Data parallelism usually stores a complete model on each machine, and the training data is divided into multiple parts according to the number of machines. After each computing node completes training on its own part of data, it synchronizes the model gradient data through network communication, and then performs the next round of training. The network communication performance between computing nodes determines the performance of model training. The emergence of intelligent network card technology offloads the protocol processing in network communication to the network card, reducing the burden on the host and improving the efficiency of network communication.

[0060] In the related art, the PCIe bus is used to connect the network card and the GPU, and the communication between the two is carried out in the DMA mode. The long data communication path results in low transmission efficiency of gradient data when the distributed artificial intelligence model is trained.

[0061] Therefore, the hardware interface conforming to CXL (Compute Express Link) is arranged in the computing device and the network card in each computing node, and the transmission of gradient data is realized between the computing device and the network card through CXL, thereby improving the transmission efficiency of gradient data between the computing device and the network card, and further improving the training efficiency of the artificial intelligence model. Further, the computing device and the network card maintain the consistency state with the host through the device consistency maintenance engine, respectively, so as to realize the cache consistency data communication between the network card and the computing device.

[0062] The embodiment of the present application discloses a computing system, comprising a plurality of computing nodes, the plurality of computing nodes train an artificial intelligence model in a data parallel manner; the computing node comprises a host, a computing device and a network card; the computing device comprises a first hardware interface, a first device consistency maintenance engine and a first computing core, the computing device is connected to the host through the first hardware interface, the first device consistency maintenance engine is used to maintain the consistency state with the host, and the first computing core is used to train the artificial intelligence model; the network card comprises a second hardware interface and a second device consistency maintenance engine, the network card is connected to the host through the second hardware interface, and the second device consistency maintenance engine is used to maintain the consistency state with the host; the computing device transmits gradient data between the cache and the network card memory of the network card through a cache refresh operation, and the network card is used to synchronize the gradient data with other computing nodes.

[0063] In the embodiment, the artificial intelligence computing system includes a plurality of computing nodes, each of which trains an artificial intelligence model in a data-parallel manner. In the process of training the artificial intelligence model, gradient data needs to be synchronized between the computing nodes. As shown in FIG. 1, each computing node includes a host, a computing device connected to the host, and a network card connected to the host. As some embodiments, the host connects a first hardware interface in the computing device and a second hardware interface in the network card through a switch. That is, the host connects the computing device and the network card through a compute express link switch (CXL Swtich). The computing device can be a GPU, which is used to train an artificial intelligence model based on training data of the computing node, calculate gradient data, transmit the calculated gradient data to the network card, synchronize the gradient data with other computing nodes, then transmit the synchronized gradient data to the computing device, and the computing device updates the model parameters based on the received gradient data. In distributed model training, gradient data refers to the partial derivative of the loss function with respect to the model parameters calculated by each node in the independent model parameter update process on multiple computing nodes. These gradient data are then averaged to obtain average gradient data, which guides the parameter update of the entire model, ensures that the model copies on the nodes can work cooperatively, and quickly converges to the optimal solution.

[0064] The computing device is shown in FIG. 2, including a first hardware interface, a first device coherence maintenance engine (DCOH), and a first computing core. The computing device is connected to the host through the first hardware interface, which is a CXL protocol compliant interface. The first hardware interface is responsible for parsing the transaction layer packets of CXL.io, CXL.cache, and CXL.mem, which are three sub-protocols of CXL. The first device coherence maintenance engine maintains coherence with the host through the CXL.cache protocol. The device coherence maintenance engine is an advanced system that ensures the stability and data consistency of distributed devices in a high-availability environment through real-time monitoring, data synchronization, fault detection and transfer, configuration management, optimization algorithms, redundancy mechanisms, automated repair, log auditing, and user interface, thereby improving the performance and reliability of the overall system. The first computing core is responsible for training an artificial intelligence model based on training data of the computing node, calculating gradient data, and updating model parameters based on the synchronized gradient data.

[0065] As some feasible embodiments, the computing device further includes a computing device memory connected to the first computing core and the first device coherence maintenance engine through a memory bus, which is used to store training data and model parameters.

[0066] In implementation, the first computing core is connected to the memory of the computing device through a memory bus, and the first device consistency maintenance engine is connected to the memory of the computing device through the memory bus. The first device consistency maintenance engine needs to maintain the consistency state of data on the memory of the computing device, and the first device consistency maintenance engine is connected to the first computing core to ensure the consistency state of cached data in the first computing core.

[0067] The network card is shown in FIG. 3, including a second hardware interface and a second device consistency maintenance engine. The network card is connected to the host through the second hardware interface, and the second hardware interface is an interface conforming to the CXL protocol. The second hardware interface is responsible for parsing transaction layer packets of three CXL sub-protocols CXL.io, CXL.cache and CXL.mem. The second device consistency maintenance engine maintains consistency with the host through the CXL.cache protocol.

[0068] As some possible implementations, the network card further includes a storage controller for accessing network card memory corresponding to the network card, and the network card memory is used for caching gradient data. In implementation, the network card includes part of on-chip memory space, i.e., network card memory, which is accessed through the storage controller in the network card and is used for caching gradient data. The physical memory of the network card and the physical memory of the host are uniformly managed by the host, and the device-side memory is referred to as HDM (Host-Device Memory). The consistency of memory access between the host and the network card is maintained through CXL.cache, and the management mode of the network card memory is HDM-D (Host-Device Memory with Direct Cache) mode.

[0069] As some possible implementations, the network card further includes a second computing core and a processor, and the processor includes a network perception unit and a network function unit. The network function unit is used to implement network processing functions, and the network perception unit is used to obtain a current network state from the network function unit. The second computing core is used to determine a compression scale based on the current network state, compress gradient data to be transmitted according to the compression scale, so that the network card transmits the compressed gradient data to other computing nodes.

[0070] In implementation, the network card further includes a second computing core, a network awareness unit, and a network function unit. The network function unit is configured to implement traditional network processing functions, such as parsing RDMA (Remote Direct Memory Access), TCP (Transmission Control Protocol), UDP (User Datagram Protocol) messages, traffic management, packet modification, and the like. The network awareness unit is configured to collect a current network state for the network function unit, and the current network state can be cached in a memory of the network card. The second computing core is responsible for adjusting a compression scale of gradient data to be transmitted according to the current network state. When the network environment is congested, a larger compression scale can be set, and when the network environment is good, a smaller compression scale can be set, thereby improving the transmission efficiency of the gradient data. For example, when the network environment is congested, the gradient data can be compressed for transmission in 8 bits, and when the network environment is good, the gradient data can be transmitted in the original 32-bit format. The compression of the gradient data is performed by the network card, and does not need to pass through the host, thereby reducing the burden of the host.

[0071] The CXL interface defines six channels of the CXL.cache protocol, including a D2H Req (device-to-host request channel), a D2H Resq (device-to-host response channel), a D2H Data (device-to-host data channel), a H2D Req (host-to-device request channel), a H2D Resq (host-to-device response channel), and a H2D Data (host-to-device data channel). In the D2H or H2D, H represents Host, and D represents Device. The host in the embodiment can be a CPU (Central Processing Unit). In order to realize consistent data communication between the network card and the computing device, such as consistent access of the computing device to the memory on the network card, a host agent module on the host side is needed to forward messages.

[0072] As some possible implementations, the host includes a host agent; the process of transmitting the gradient data between the computing device and the network memory of the network card through the flushing operation of the cache is as follows: the computing device sends a first data request for transmitting the gradient data to the host agent; the host agent modifies a request type of the first data request to obtain a second data request, and sends the second data request to the network card; a second device consistency maintenance engine in the network card acquires the gradient data through the storage controller, encapsulates a first data response based on the gradient data, and sends the first data response to the host agent; the host agent modifies a response type of the first data response to obtain a second data response, and sends the second data response to the computing device; and the computing device acquires the gradient data based on the second data response.

[0073] In implementation, the computing device first requests data on a certain physical address on the network card from the host through the channel of D2H Req, the host modifies the header of the D2H message, since the system physical address is uniformly addressed, the address part does not need to be changed, the host modifies the message into the H2D Req type and forwards it to the network card, the second hardware interface on the network card parses the message and then forwards it to the second device consistency maintenance engine, the second device consistency maintenance engine obtains the data by communicating with the storage controller, encapsulates the message into D2H Resp and D2H Data, and returns it to the host, and the host modifies the header to obtain the H2D Resp and H2D Data message and returns it to the computing device.

[0074] In the process of transmitting gradient data between the computing device and the network card, the computing device and the network card can communicate through the CXL.cache protocol, and the gradient is automatically transmitted to the network card memory through the flush operation of the cache, at this time the network card memory is equivalent to the secondary storage of the computing device cache, in this way, the long data transmission process of gradient data from the computing device cache→computing device memory→network card memory is avoided, and the transmission efficiency of the gradient data is improved.

[0075] As some possible implementation manners, the second computing core is further configured to read the gradient data calculated by the computing node and cached in the network card memory through the second device consistency maintenance engine, average the gradient data calculated by the computing node to obtain average gradient data, and cache the average gradient data in the network card memory. In implementation, the network card will perform the averaging operation on the gradient data after receiving the gradient data of other computing nodes, and then cache the calculated average gradient data in the network card memory. Compared with the related art, the gradient information that needs to be synchronized between machines is stored in the network card memory, and the size of the memory available to the computing device is indirectly expanded by using the network card memory.

[0076] As some possible implementation manners, the first computing core reads the average gradient data in the network card memory through the first device consistency maintenance engine and the second device consistency maintenance engine, the first computing core reads the model parameters in the computing device memory, and updates the model parameters according to the average gradient data. The first computing core directly accesses the computing device memory.

[0077] In implementation, as shown in FIG. 4, the second computing core in the network card accesses the network card memory of itself through the forwarding of the second device coherence maintenance engine, the first computing core in the computing device accesses the network card memory through the forwarding of the first device coherence maintenance engine and the second device coherence maintenance engine. The first computing core in the computing device can directly access the computing device memory of itself without the forwarding of the first device coherence maintenance engine.

[0078] As some possible implementation, the second device coherence maintenance engine is further configured to: in response to determining that the first computing core and the second computing core access the network card memory at the same time, perform conflict processing.

[0079] In implementation, when the computing device and the network card access the network card memory at the same time, the second device coherence maintenance engine in the network card manages the conflict situation that the network card memory is accessed by the network card and the computing device at the same time. Through the CXL.cache communication between the second device coherence maintenance engine and the first device coherence maintenance engine, if a memory page of the network card memory is accessed by the first computing core in the computing device, the second computing core in the network card cannot access the memory page.

[0080] As some possible implementation, the artificial intelligence model includes multiple levels, and the first computing core transmits the gradient data of the preset number of levels to the network card through the flushing operation of the cache after calculating the gradient data of the preset number of levels, so that the network card synchronizes the gradient data of the preset number of levels with other computing nodes.

[0081] It can be understood that in the training process of the artificial intelligence model, after the computing device calculates the loss value forwardly, it will calculate the gradient data of each level in turn from the back to the front according to the model network level. In response to determining that the model network depth is large, it can not be necessary to calculate the gradient data of the levels after the loss value and then synchronize through the network. Instead, after calculating a preset number of levels, it can be synchronized with other computing nodes in batches through the network card, thereby improving the synchronization efficiency of the gradient data.

[0082] The computing device and the network card in each computing node participate in the training process of the artificial intelligence model as follows: in the initialization stage, the network card stores the corresponding gradient according to the size of the model, and sets the value of the gradient to 0; after the forward propagation of the computing device is completed, during the backward propagation, the computing device calculates the gradient of each layer, which can be automatically synchronized to the network card memory through the cache flush operation; the network card memory runs the gradient compression strategy according to the network condition, and then synchronizes with other machines in real time. At the same time, the gradient information of other machines is also received by the current machine in real time. At this time, taking the synchronization operator of Ring AllReduce as an example, the network card will process in real time after receiving the message. After the gradient synchronization is completed, the network card calculates the average of the collected gradient, and the computing device reads the gradient information in the network card memory and adds the model parameters in the computing device memory to complete the update operation of the model parameters, and the next batch of model training is started.

[0083] The computing system provided by the embodiments of the present application sets the CXL-compliant hardware interface in the computing device and the network card in each computing node, and the gradient data transmission between the computing device and the network card is realized through CXL, thereby improving the gradient data transmission efficiency of the computing device and the network card, and further improving the training efficiency of the artificial intelligence model. Further, the computing device and the network card maintain the consistency state through the device consistency maintenance engine and the host respectively, realizing the cache consistency data communication between the network card and the computing device.

[0084] The embodiments of the present application disclose an artificial intelligence model training method, referring to FIG. 5, according to the flow chart of the artificial intelligence model training method shown in the embodiments of the present application, as shown in FIG. 5, comprising:

[0085] S101: reading the training data in the computing device memory, and forward propagating the training artificial intelligence model based on the training data;

[0086] The execution subject of the embodiment is the computing device in the computing node, for example, GPU. The computing device includes a first hardware interface, a first device consistency maintenance engine and a first computing core. The computing device is connected with the host through the first hardware interface, the first hardware interface is a CXL-compliant interface, and the first hardware interface is responsible for parsing the transaction layer message of CXL.io, CXL.cache and CXL.mem, which are three CXL sub-protocols. The first device consistency maintenance engine maintains consistency with the host through CXL.cache protocol. The first computing core is responsible for training the artificial intelligence model based on the training data of the computing node, calculating the gradient data, and updating the model parameters based on the synchronized gradient data.

[0087] In implementation, the computing device reads the training data from the computing device memory and uses the training data to train the artificial intelligence model through a forward propagation process. Forward propagation is a basic step in neural networks where input data is passed through layers of the model to generate an output result.

[0088] As some possible implementations, the computing device further includes a computing device memory connected to the first computing core and the first device coherency maintenance engine through a memory bus, and the computing device memory is used to store the training data and the model parameters. In implementation, the first computing core is connected to the computing device memory through the memory bus, and the computing device memory includes a plurality of memory blocks. The first device coherency maintenance engine is connected to the computing device memory through the memory bus, and the first device coherency maintenance engine needs to maintain the coherency state of the data on the computing device memory. At the same time, the first device coherency maintenance engine is connected to the first computing core to ensure the coherency state of the cached data in the first computing core.

[0089] S102: Based on the result output by the artificial intelligence model, the gradient data is calculated by backward propagation, and the gradient data is transmitted to the network card through the refresh operation of the cache, so that the network card synchronizes the gradient data with other computing nodes to obtain average gradient data, and the average gradient data is cached to the memory of the network card;

[0090] In implementation, the computing device performs backward propagation calculation based on the output result of the artificial intelligence model. Backward propagation is a key step in training neural networks, which is used to calculate the gradient of the loss function with respect to the model parameters. The calculated gradient data is transmitted to the network card through the cache refresh operation of the CXL (Compute Express Link) protocol. The CXL protocol allows high-speed, low-latency data transmission, thereby improving the efficiency of gradient synchronization. The network card synchronizes the gradient data with other computing nodes, then calculates the average gradient data, and caches the calculated average gradient data to the memory of the network card.

[0091] The network card includes a second hardware interface and a second device coherency maintenance engine. The network card is connected to the host through the second hardware interface, and the second hardware interface is a CXL protocol compliant interface. The second hardware interface is responsible for parsing the transaction layer packets of the three CXL sub-protocols CXL.io, CXL.cache and CXL.mem. The second device coherency maintenance engine maintains coherency with the host through the CXL.cache protocol.

[0092] As some possible implementation manners, the network card further comprises a memory controller configured to access a network card memory corresponding to the network card, and the network card memory is configured to cache the gradient data. In implementation, the network card comprises a part of on-chip memory space, i.e., the network card memory, and the network card memory is accessed by the memory controller in the network card and is configured to cache the gradient data. The physical memory of the network card and the physical memory of the host are uniformly managed by the host, the device-side memory is referred to as HDM, and the consistency of memory access between the host and the network card is maintained by CXL.cache. At this time, the management mode of the network card memory is HDM-D mode.

[0093] As some possible implementation manners, the network card further comprises a second computing core, a network perception unit and a network function unit. The network function unit is configured to implement a network processing function. The network perception unit is configured to obtain a current network state from the network function unit. The second computing core is configured to determine a compression scale based on the current network state, compress the gradient data to be transmitted according to the compression scale, and transmit the compressed gradient data to other computing nodes by the network card.

[0094] In implementation, the network card further comprises a second computing core, a network perception unit and a network function unit. The network function unit is configured to implement a traditional network processing function, such as parsing RDMA, TCP, UDP (User Datagram Protocol) packets, traffic management, data packet modification, etc. The network perception unit is configured to collect a current network state by the network function unit. The current network state can be cached in the network card memory. The second computing core is responsible for adjusting the compression scale of the gradient data to be transmitted according to the current network state. When the network environment is congested, a larger compression scale can be set. When the network environment is good, a smaller compression scale can be set. For example, when the network environment is congested, the gradient data can be compressed to 8 bits for transmission. When the network environment is good, the gradient data can be transmitted in the original 32-bit format. The compression of the gradient data is responsible by the network card, and does not need to pass through the host, thereby reducing the burden of the host.

[0095] S103: reading the model parameters in the memory of the computing device and the average gradient data in the network card memory, and updating the model parameters based on the average gradient data.

[0096] In implementation, the computing device reads the current model parameters from the memory of the computing device and reads the average gradient data of each layer calculated previously from the network card memory, and updates the model parameters of each layer based on the average gradient data of each layer, so as to reduce the loss value of the model and improve the prediction accuracy of the model.

[0097] As some possible implementations, reading the average gradient data in the NIC memory includes: reading the average gradient data in the NIC memory by a first device coherency maintenance engine in the computing device and a second device coherency maintenance engine in the NIC. In implementation, when the first computing core in the computing device accesses the NIC memory, it needs to be forwarded by the first device coherency maintenance engine and the second device coherency maintenance engine.

[0098] The CXL interface defines 6 channels of the CXL.cache protocol, including D2H Req (device-host request channel), D2H Resq (device-host response channel), D2H Data (device-host data channel), H2D Req (host-device request channel), H2D Resq (host-device response channel), and H2D Data (host-device data channel). H in D2H or H2D represents Host, and D represents Device. The host in the embodiment can be a CPU (Central Processing Unit). In order to realize the consistent data communication between the NIC and the computing device, such as the computing device accessing the memory on the NIC in a consistent manner, the message needs to be forwarded by the host agent module on the host side.

[0099] As some possible implementations, reading the average gradient data in the NIC memory includes: reading the average gradient data in the NIC memory by a first device coherency maintenance engine in the computing device and a second device coherency maintenance engine in the NIC. In implementation, when the first computing core in the computing device accesses the NIC memory, it needs to be forwarded by the first device coherency maintenance engine and the second device coherency maintenance engine.

[0100] In implementation, the computing device first requests the average gradient data on a certain physical address on the network card from the host through the channel of D2H Req, the host modifies the header of the D2H message, since the system physical address is uniformly addressed, the address part does not need to be changed, the host modifies it into a H2D Req type message and forwards it to the network card, the second hardware interface on the network card parses the message and then forwards it to the second device consistency maintenance engine, the second device consistency maintenance engine obtains the average gradient data by communicating with the storage controller, and then encapsulates the message into D2H Resp and D2H Data and returns it to the host, the host modifies the header to obtain H2D Resp and H2D Data and returns it to the computing device, and the computing device parses the received message to obtain the average gradient data.

[0101] In the process of transmitting the gradient data between the computing device and the network card, the computing device and the network card can communicate through the CXL.cache protocol, and the gradient is automatically transmitted to the network card memory through the flush operation of the cache, at this time the network card memory is equivalent to the secondary storage of the computing device cache, in this way, the long data transmission process from the computing device cache→computing device memory→network card memory is avoided, and the transmission efficiency of the average gradient data is improved.

[0102] As some possible implementations, the artificial intelligence model includes multiple levels, the gradient data is calculated based on the result output by the artificial intelligence model, and the gradient data is transmitted to the network card through the flush operation of the cache, so that the network card synchronizes the gradient data with other computing nodes to obtain the average gradient data, and the average gradient data is cached to the network card memory, including: after calculating the gradient data of a preset number of levels, the gradient data of the preset number of levels is transmitted to the network card through the flush operation of the cache, so that the network card synchronizes the gradient data of the preset number of levels with other computing nodes to obtain the average gradient data of the preset number of levels, and the average gradient data of the preset number of levels is cached to the network card memory.

[0103] In implementation, in response to determining that the model network depth is large, the gradient data of the levels can not be calculated once and then synchronized through the network, but can be synchronized in batches through the network card and other computing nodes after calculating a preset number of levels, thereby improving the synchronization efficiency of the gradient data.

[0104] As can be seen, in the embodiment, the computing device and the network card in each computing node realize the transmission of the gradient data through CXL, which improves the transmission efficiency of the gradient data between the computing device and the network card, and further improves the training efficiency of the artificial intelligence model.

[0105] An artificial intelligence model training device according to an embodiment of the present application is described below. The artificial intelligence model training device described below can be used in conjunction with the artificial intelligence model training method described above.

[0106] Referring to FIG. 6, a structural diagram of an artificial intelligence model training device according to an embodiment of the present application is shown in FIG. 6, which includes:

[0107] The forward propagation calculation module 100 is configured to read the training data in the memory of the computing device and forward-propagate the training artificial intelligence model based on the training data.

[0108] The backward propagation calculation module 200 is configured to calculate gradient data based on the result output by the artificial intelligence model, transmit the gradient data to the network card through a cache refresh operation, so that the network card synchronizes the gradient data with other computing nodes to obtain average gradient data, and cache the average gradient data in the memory of the network card.

[0109] The update module 300 is configured to read the model parameters in the memory of the computing device and the average gradient data in the memory of the network card, and update the model parameters based on the average gradient data.

[0110] As can be seen, in the embodiment, the computing device and the network card in each computing node implement transmission of gradient data through CXL, which improves the transmission efficiency of the computing device and the network card for gradient data, and further improves the training efficiency of the artificial intelligence model.

[0111] On the basis of the above embodiment, as some implementations, the update module 300 is configured to read the average gradient data in the memory of the network card through the first device consistency maintenance engine in the computing device and the second device consistency maintenance engine in the network card.

[0112] On the basis of the above embodiment, as some implementations, the update module 300 is configured to send, through the first device consistency maintenance engine in the computing device, a first data request for transmitting the average gradient data to a host agent in the host, so that the host agent modifies the request type of the first data request to obtain a second data request, and sends the second data request to the network card; receive the second data response sent by the host agent; wherein the second device consistency maintenance engine in the network card acquires the average gradient data through the storage controller, encapsulates the first data response based on the average gradient data, and sends the first data response to the host agent, the host agent modifies the response type of the first data response to obtain the second data response; and acquires the average gradient data based on the second data response.

[0113] On the basis of the above-mentioned embodiments, as some embodiments, the artificial intelligence model comprises a plurality of levels, and the back propagation calculation module 200 is configured to: after calculating gradient data of a preset number of levels each time, transmit the gradient data of the preset number of levels to the network card through a refresh operation of the cache, so that the network card synchronizes the gradient data of the preset number of levels with other computing nodes to obtain average gradient data of the preset number of levels, and caches the average gradient data of the preset number of levels to the network card memory.

[0114] As to the apparatus in the above-mentioned embodiments, the way in which each module performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

[0115] Based on the hardware implementation of the above-mentioned program modules, and in order to implement the method of the embodiments of the present application, the embodiments of the present application also provide an electronic device. FIG. 7 is a structural diagram of an electronic device according to an embodiment of the present application. As shown in FIG. 7, the electronic device comprises:

[0116] a communication interface 1 capable of interacting with other devices such as network devices and the like;

[0117] one or more processors 2 connected to the communication interface 1 to realize information interaction with other devices;

[0118] a memory 3 associated with the one or more processors 2, the memory 3 being configured to store computer readable instructions, the computer readable instructions being configured to realize the artificial intelligence model training method provided by one or more technical solutions when read and executed by the one or more processors 2. And the computer readable instructions are stored on the memory 3.

[0119] Of course, in actual application, various components in the electronic device are coupled together through a bus system 4. It can be understood that the bus system 4 is used to realize the connection and communication between the components. In addition to the data bus, the bus system 4 also includes a power bus, a control bus and a status signal bus. However, in order to clearly illustrate, various buses are marked as the bus system 4 in FIG. 7.

[0120] The memory 3 in the embodiments of the present application is used to store various types of data to support the operation of the electronic device. Examples of these data include: any computer readable instructions for operating on the electronic device.

[0121] It can be understood that the memory 3 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM). The magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM can be used, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), sync link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memory 3 described in the embodiments of the present application is intended to include, but not limited to, these and any other suitable types of memory.

[0122] The method disclosed in the embodiments of the present application can be applied in the processor 2 or implemented by the processor 2. The processor 2 can be an integrated circuit with signal processing capability. In the implementation, the steps of the above method can be completed by the integrated logic circuit of hardware in the processor 2 or the instructions in the form of software. The processor 2 described above can be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 2 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiments of the present application, the execution can be directly completed by the hardware decoding processor or by the combination of hardware and software modules in the decoding processor. The software module can be located in the storage medium, which is located in the memory 3. The processor 2 reads the readable instructions in the memory 3 and combines the hardware to complete the steps of the above method.

[0123] The processor 2 implements the corresponding flow in each method of the embodiments of the present application when executing the readable instructions. For brevity, it will not be repeated here.

[0124] In the exemplary embodiments, as shown in FIG. 8, the embodiments of the present application also provide a non-volatile computer readable storage medium storing computer readable instructions, which are executed by one or more processors 2 to complete the above method steps.

[0125] In the exemplary embodiments, as shown in FIG. 9, the embodiments of the present application also provide a computer program product including a computer program, which is executed by one or more processors 2 to complete the above method steps.

[0126] Those skilled in the art can understand that all or part of the steps of the above method embodiments can be completed by the hardware related to the computer readable instructions. The above computer readable instructions can be stored in a non-volatile storage medium, and when executed, execute the steps of the above method embodiments. Or, when the above integrated units of the present application are realized in the form of software function modules and sold or used as independent products, they can also be stored in a non-volatile storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products, which are stored in a non-volatile storage medium and include a number of instructions for making an electronic device (which can be a personal computer, a server, a network device, etc.) execute all or part of the methods described in the embodiments of the present application.

[0127] The above merely provides the implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the change or replacement within the technical range disclosed by the present application, which should be covered in the protection scope of the present application.

Claims

1. A computing system, comprising: The computing system comprises a plurality of computing nodes, and the plurality of computing nodes train an artificial intelligence model in a data parallel manner; Each computing node comprises a host, a computing device, and a network card; wherein The computing device comprises a first hardware interface, a first device consistency maintenance engine, and a first computing core, the computing device is connected to the host through the first hardware interface, the first device consistency maintenance engine is configured to maintain a consistency state with the host, and the first computing core is configured to train the artificial intelligence model; The network card comprises a second hardware interface and a second device consistency maintenance engine, the network card is connected to the host through the second hardware interface, and the second device consistency maintenance engine is configured to maintain a consistency state with the host; and the computing device transmits gradient data between a cache refresh operation and a network card memory of the network card, and the network card is configured to synchronize the gradient data with other computing nodes.

2. The computing system of claim 1, wherein, The computing device is a graphics processing unit.

3. The computing system of claim 1, wherein, The computing device further comprises a computing device memory connected to the first computing core and the first device consistency maintenance engine through a memory bus, and the computing device memory is configured to store training data and model parameters.

4. The computing system of claim 1, wherein, The network card further comprises a storage controller configured to access a network card memory corresponding to the network card, and the network card memory is configured to cache gradient data.

5. The computing system of claim 4, wherein, The host comprises a host agent; The computing device transmits gradient data between a cache refresh operation and a network card memory of the network card comprises: The computing device sends a first data request for transmitting gradient data to the host agent; The host agent modifies a request type of the first data request to obtain a second data request, and sends the second data request to the network card; The second device consistency maintenance engine in the network card acquires gradient data through the storage controller, encapsulates a first data response based on the gradient data, and sends the first data response to the host agent; The host agent modifies a response type of the first data response to obtain a second data response, and sends the second data response to the computing device; and The computing device acquires gradient data based on the second data response.

6. The computing system of claim 1, wherein, The network card further comprises a second computing core, a network perception unit, and a network function unit; The network function unit is configured to implement a network processing function; The network perception unit is configured to acquire a current network state from the network function unit; and The second computing core is configured to determine a compression scale based on the current network state, compress gradient data to be transmitted according to the compression scale, so that the network card transmits the compressed gradient data to other computing nodes.

7. The computing system of claim 6, wherein, The second computing core is further configured to: read gradient data calculated by the computing node and cached in the network card memory through the second device consistency maintenance engine, average the gradient data calculated by the computing node to obtain average gradient data, and cache the average gradient data in the network card memory.

8. The computing system of claim 7, wherein The first computing core reads the average gradient data in the network card memory through the first device consistency maintenance engine and the second device consistency maintenance engine. And The first computing core reads the model parameters in the computing device memory and updates the model parameters according to the average gradient data.

9. The computing system of claim 8, wherein, The first computing core directly accesses the computing device memory.

10. The computing system of claim 8, wherein, The second device consistency maintenance engine is also used for: in response to determining that the first computing core and the second computing core access the network card memory at the same time, performing conflict processing.

11. The computing system of claim 1, wherein, The host connects the first hardware interface in the computing device and the second hardware interface in the network card through a switch.

12. The computing system of claim 1, wherein, The artificial intelligence model includes multiple levels, and the first computing core transmits the gradient data of a preset number of levels to the network card through a cache refresh operation after calculating the gradient data of the preset number of levels, so that the network card synchronizes the gradient data of the preset number of levels with other computing nodes.

13. An artificial intelligence model training method, characterized by, The method applied to the computing device in the computing system of any one of claims 1 to 12, the method comprising: reading training data in the computing device memory and forward propagating a training artificial intelligence model based on the training data; based on the result output by the artificial intelligence model, backward propagating computing gradient data, transmitting the gradient data to the network card through a cache refresh operation, so that the network card synchronizes the gradient data with other computing nodes to obtain average gradient data, and caches the average gradient data to the network card memory; and reading the model parameters in the computing device memory and the average gradient data in the network card memory, and updating the model parameters based on the average gradient data. 14.The method of claim 13, wherein, Reading the average gradient data in the network card memory comprises: reading the average gradient data in the network card memory through the first device consistency maintenance engine in the computing device and the second device consistency maintenance engine in the network card. 15.The method of claim 14, wherein, Reading the average gradient data in the network card memory through the first device consistency maintenance engine in the computing device and the second device consistency maintenance engine in the network card comprises: sending a first data request for transmitting average gradient data to a host agent in the host through the first device consistency maintenance engine in the computing device, so that the host agent modifies the request type of the first data request to obtain a second data request, and sends the second data request to the network card; receiving the second data response sent by the host agent; wherein the second device consistency maintenance engine in the network card obtains average gradient data through a storage controller, encapsulates a first data response based on the average gradient data, and sends the first data response to the host agent, and the host agent modifies the response type of the first data response to obtain the second data response; and obtaining average gradient data based on the second data response. 16.The method of claim 13, wherein, The artificial intelligence model comprises a plurality of levels, gradient data is calculated based on a result output by the artificial intelligence model, the gradient data is transmitted to a network card through a refresh operation of a cache, so that the network card synchronizes gradient data with other computing nodes to obtain average gradient data, and the average gradient data is cached to a network card memory, comprising: After a preset number of levels of gradient data are calculated, the preset number of levels of gradient data are transmitted to the network card through the refresh operation of the cache, so that the network card synchronizes the preset number of levels of gradient data with other computing nodes to obtain preset number of levels of average gradient data, and the preset number of levels of average gradient data are cached to the network card memory.

17. An artificial intelligence model training apparatus comprising: The computing device is applied to the system in any one of claims 1 to 12, and the device comprises: a forward propagation calculation module configured to read training data in a memory of the computing device and forward propagate a training artificial intelligence model based on the training data; a backward propagation calculation module configured to calculate gradient data based on a result output by the artificial intelligence model, transmit the gradient data to a network card through a refresh operation of a cache, so that the network card synchronizes gradient data with other computing nodes to obtain average gradient data, and cache the average gradient data to a network card memory; and an update module configured to read model parameters in the memory of the computing device and the average gradient data in the network card memory, and update the model parameters based on the average gradient data.

18. An electronic device, comprising: comprise: one or more processors; and a memory associated with the one or more processors, the memory being configured to store computer-readable instructions that, when read and executed by the one or more processors, implement the steps of the artificial intelligence model training method in any one of claims 13 to 16. The non-volatile computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions, when executed by one or more processors, implement the steps of the artificial intelligence model training method in any one of claims 13 to 16.

19. A non-transitory computer readable storage medium, comprising: The computer program product comprises a computer program, and the computer program, when executed by one or more processors, implements the steps of the artificial intelligence model training method in any one of claims 13 to 16.

20. A computer program product, characterised in that, ​

Images