Data processing system and method, and medium

Abstract

The present application relates to the technical field of model training, and discloses a data processing system and method, and a medium. According to the present application, a model training task is executed by means of acceleration devices in a host, and training data, an intermediate result, and weight data are stored by means of a memory board in the host, so that the task does not need to be executed by the host, and the data does not need to be stored by the host, thereby reducing the load of the host. The acceleration devices execute the model training task, so that the model training efficiency can be improved. Only the host is needed for task allocation in the whole training process, and a large amount of computing and memory access is unloaded to the acceleration devices and the memory board, so that the downtime risk and the model weight data recovery difficulty in the training process can be reduced.

Description

Data processing system, method and medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on February 28, 2024, with application number 202410220698.1 and application name “A Data Processing System, Method and Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to the field of model training technology, and in particular to a data processing system, method and medium. Background Art

[0004] Currently, when using a host computer to perform model training tasks, the host memory needs to store a large amount of data. Intermediate results, weights, and other data from model training also need to be stored in the host memory. This increases the host load and reduces training efficiency. If a training program error occurs, saving a complete copy of the model weights requires a significant amount of memory access time, and there is a risk of losing the model weights in the event of a power outage.

[0005] Summary of the Invention

[0006] In view of this, the purpose of this application is to provide a data processing system, method and medium. The scheme is as follows:

[0007] In a first aspect, the present application provides a data processing system, comprising a plurality of hosts, wherein any host comprises: a plurality of acceleration devices and at least one memory board;

[0008] The multiple hosts include a control host, which divides the same model training task into multiple subtasks and distributes the subtasks to multiple hosts;

[0009] Multiple hosts use their own accelerators to execute the received subtasks in parallel, and use their own memory cards to store the training data, intermediate results, and weight data corresponding to the subtasks; and

[0010] The control host uses its own memory board to collect and process the weight data stored in the memory boards of multiple hosts, and writes the latest processed weight data back to the memory boards of multiple hosts.

[0011] In another aspect, any memory board includes: a non-volatile memory module, the non-volatile memory module including a memory controller and a non-volatile memory area;

[0012] The memory controller is used to perform memory allocation operations, memory release operations, data storage operations, address mapping operations and / or memory request scheduling on the non-volatile memory area; and

[0013] The non-volatile storage area is used to respond to memory allocation operations, memory release operations, data storage operations, address mapping operations and / or memory request scheduling performed by the storage controller.

[0014] On the other hand, the non-volatile storage area is divided into training data area, full model weight data area, subtask weight data area and other areas.

[0015] On the other hand, any memory board also includes: a board scheduler, which is used to achieve data consistency between the memory of the host to which the current memory board belongs and the non-volatile storage area in the current memory board through the CXL protocol.

[0016] On the other hand, any memory board further includes: a priority arbiter, which determines a priority according to a memory access address in a received memory access request and processes the memory access request according to the determined priority.

[0017] On the other hand, the memory access request is at least one of: a request from the host to which the current memory board belongs to access the non-volatile storage area in the current memory board, a request from the acceleration device in the host to which the current memory board belongs to access the non-volatile storage area in the current memory board, a request from the current memory board to access the non-volatile storage area in itself, and a request from the current memory board to access memory boards in other hosts.

[0018] On the other hand, the priority arbiter determines the priority as the highest in response to determining that the memory access request is a request from an acceleration device in a host to which the current memory board belongs to access a non-volatile storage area in the current memory board.

[0019] On the other hand, the priority arbiter determines the priority according to a preset policy in response to determining that the memory access request is not a request for the acceleration device in the host to which the current memory board belongs to access the non-volatile storage area in the current memory board.

[0020] On the other hand, any memory board also includes: a board calculation unit, which is used to collect and calculate weight data.

[0021] On the other hand, any memory board further includes: a network module, which is used to communicate with memory boards in other hosts through a target protocol.

[0022] On the other hand, the network module is used to:

[0023] Synchronize the weight data stored in the memory card of the network module to the memory cards in other hosts through the RDMA (Remote Direct Memory Access) protocol; and

[0024] Receive weight data sent by memory cards in other hosts through the RDMA protocol.

[0025] On the other hand, the control host initializes the memory boards in multiple hosts, configures the interaction mode, memory size and starting offset address of each memory board, uniformly addresses each memory board to obtain an addressing table, and enables access rights between each memory board.

[0026] On the other hand, any host or any memory board or any acceleration device accesses the target memory address through the base address and offset address corresponding to the target memory address, and the target memory address is the address of the non-volatile storage area in the non-current memory board.

[0027] On the other hand, any host maps the address of the non-volatile storage area in its own memory board to the host memory space through the CXL protocol, and realizes the conversion between the host memory address and the board memory address according to the mapped address mapping relationship.

[0028] On the other hand, any host divides the received subtasks into multiple task blocks according to the number of acceleration devices in itself, distributes the multiple task blocks to the acceleration devices in itself, and controls the acceleration devices in itself to run the multiple task blocks in parallel.

[0029] On the other hand, the control host calculates the average of the weight data stored in the memory boards in the multiple hosts, and uses the average as the latest weight data.

[0030] On the other hand, any acceleration device includes a device scheduler, which is used to achieve data consistency between the memory of the current acceleration device and the memory of the host to which the current acceleration device belongs through the CXL protocol.

[0031] On the other hand, any acceleration device includes multiple device computing units, which are used to execute received task blocks; the task blocks are obtained by dividing the subtasks received by the current acceleration device according to the number of acceleration devices in the host to which the current acceleration device belongs.

[0032] On the other hand, if any host or any memory board or any acceleration device determines that the target memory address is local according to the addressing table, the target memory address is accessed through the PCIE protocol; and / or if any host or any memory board or any acceleration device determines that the target memory address is not local according to the addressing table, the target memory address is accessed through the RDMA protocol.

[0033] On the other hand, any acceleration device stores the intermediate weights obtained during the training process of executing the subtask into the subtask weight data area of ​​the memory board of the same host; in response to determining that the subtask weight data area is full, the flag corresponding to the subtask weight data area is set from 0 to 1.

[0034] On the other hand, the board compute unit in any memory board is used to:

[0035] Read the intermediate weight in the local or remote subtask weight data area according to the addressing table, and set the flag corresponding to the corresponding subtask weight data area from 1 to 0; and

[0036] Calculate the mean of the read data and write the mean to the local or remote full model weight data area according to the addressing table.

[0037] In a second aspect, the present application provides a data processing method applied to a distributed system, the distributed system including a plurality of hosts, any host including: a plurality of acceleration devices and at least one memory board;

[0038] The multiple hosts include a control host, which divides the same model training task into multiple subtasks and distributes the subtasks to multiple hosts;

[0039] Multiple hosts use their own accelerators to execute the received subtasks in parallel, and use their own memory cards to store the training data, intermediate results, and weight data corresponding to the subtasks; and

[0040] The control host uses its own memory board to collect and process the weight data stored in the memory boards of multiple hosts, and writes the latest weight data obtained through processing into the memory boards of multiple hosts.

[0041] On the other hand, the network module in any memory board synchronizes the weight data stored in the current memory board to the memory boards in other hosts through the RDMA protocol; and receives the weight data sent by the memory boards in other hosts through the RDMA protocol.

[0042] On the other hand, the control host initializes the memory boards in multiple hosts, configures the interaction mode, memory size and starting offset address of each memory board, uniformly addresses each memory board to obtain an addressing table, and enables access rights between each memory board.

[0043] In a third aspect, the present application provides a non-transitory computer-readable storage medium, which stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the aforementioned disclosed data processing method is implemented.

[0044] From the above scheme, it can be seen that the present application provides a data processing system, including multiple hosts, any host includes: multiple acceleration devices and at least one memory board; the multiple hosts include a control host, the control host divides the same model training task into multiple subtasks, and distributes the multiple subtasks to multiple hosts; the multiple hosts use the multiple acceleration devices in themselves to execute the received subtasks in parallel, and use the memory boards in themselves to store the training data, intermediate results and weight data corresponding to the corresponding subtasks; the control host uses the memory boards in itself to collect and process the weight data stored in the memory boards in multiple hosts, and writes the latest processed weight data back to the memory boards in multiple hosts. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0046] FIG1 is a schematic diagram of a data processing system disclosed in this application;

[0047] FIG2 is a schematic diagram of a distributed cluster disclosed in this application;

[0048] FIG3 is a schematic diagram of a memory board disclosed in this application;

[0049] FIG4 is a schematic diagram of an acceleration device disclosed in this application;

[0050] FIG5 is a flow chart of a data processing method disclosed in this application;

[0051] FIG6 is a structural diagram of a server provided by this application;

[0052] FIG7 is a diagram of a terminal structure provided by this application;

[0053] FIG8 is a structural diagram of a non-transitory computer storage medium provided by the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other examples obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0055] Large-model training refers to the process of training deep learning models using large datasets and computing resources. Large models typically have more parameters and deeper network structures, enabling them to learn more complex and abstract feature representations. This gives large models greater expressive power when handling complex tasks and large amounts of data, better capturing underlying patterns and relationships within the data. Increasing the number of parameters and layers can improve model performance and effectiveness. For example, in natural language processing, large models can better understand and generate natural language, demonstrating greater semantic accuracy and fluency. In computer vision, large models can more accurately perform tasks such as object recognition and image generation. They also possess stronger generalization capabilities, meaning they perform well on unseen data. By increasing the model's capacity and complexity, large models can better adapt to diverse data distributions and task requirements, thereby improving their generalization capabilities to new data. Training on large amounts of data allows them to learn more versatile feature representations. This enables large models to perform better in transfer learning tasks, transferring learned knowledge to new tasks and domains.

[0056] At the same time, training large models faces several difficulties and challenges. The most obvious challenge is the extremely high demands placed on training equipment. These challenges primarily include the following: Processor performance: Large model training requires high-performance processors for computation. Graphics processors (GPUs) or dedicated AI accelerators are often used to accelerate model training. These processors offer parallel computing capabilities and high-speed floating-point operations, significantly accelerating training. Memory capacity: Model parameters in large model training are often very large, requiring a large amount of memory to store and update them. For a single compute node, the memory capacity must be sufficient to store all model parameters. If memory capacity is insufficient, model parallelization or distributed training techniques may be necessary to address memory limitations. Storage system: Large model training generates large amounts of data, requiring a fast storage system for storing and accessing data. High-speed solid-state drives (SSDs) or network storage (such as distributed file systems) can provide sufficient storage bandwidth and capacity to meet the requirements of large-scale training. Network connectivity: Large model training typically requires communication and synchronization between multiple compute nodes. Therefore, high-speed network connectivity is required to ensure efficient and stable data transmission between nodes. High-performance networks such as high-speed Ethernet can meet the requirements of large-scale training. Distributed computing architecture: For large-scale model training, a distributed computing architecture may be needed to distribute computing tasks across multiple computing nodes for parallel processing. This can improve training speed and efficiency. A distributed computing architecture requires features such as task scheduling, node management, and communication coordination.

[0057] Currently, when using a host to perform model training tasks, a large amount of data needs to be stored in the host memory. The intermediate results, weights, and other data of the model training also need to be stored in the host memory, which will increase the host load and slow down the training efficiency. When the training program fails, saving a complete copy of the model weights requires a lot of memory access time, and the model weights are at risk of being lost during power failure. To this end, this application provides a data processing solution that can reduce the host load and improve the efficiency of model training.

[0058] As shown in FIG1 , an embodiment of the present application discloses a data processing system including multiple hosts, each of which includes multiple acceleration devices and at least one memory card. The acceleration devices include FPGA (Field-Programmable Gate Array) accelerator cards, GPU (Graphics Processing Unit) accelerator cards, and the like.

[0059] The multiple hosts include a control host, which divides the same model training task into multiple subtasks and distributes the multiple subtasks to the multiple hosts. The multiple hosts use their own multiple acceleration devices to execute the received subtasks in parallel, and use their own memory boards to store the training data, intermediate results, and weight data corresponding to the corresponding subtasks. The control host uses its own memory board to collect and process the weight data stored by the memory boards in the multiple hosts, and writes the latest weight data obtained from the processing back to the memory boards in the multiple hosts. The memory board can be a memory expansion card implemented based on FPGA.

[0060] In this embodiment, model training tasks are used to train various intelligent models. These intelligent models can be of any structure and can be used to implement tasks such as data encryption, decryption, image recognition, and classification. The host is a server. A data processing system composed of multiple hosts can be a distributed system. Hosts in a distributed system can share resources, accelerating computation. This embodiment divides a model training task into several subtasks that run in parallel and distributes these subtasks across different host nodes, allowing them to run simultaneously on all of them, thereby accelerating computation. Furthermore, the distributed system features computation migration. If a host node is found to be overloaded, some jobs can be moved to other host nodes for execution, thereby reducing the load on the previous host node. This type of job migration is known as load balancing. Furthermore, distributed systems offer high reliability. If a node fails, the remaining nodes can continue to operate, preventing the entire system from collapsing due to the failure of one or a few nodes. Therefore, distributed systems have excellent fault tolerance. The system can also detect node failures and take appropriate measures to recover from them. Once the system determines the failed node, it will no longer use it to provide services until it returns to normal operation. The functions of the failed node can be completed by other nodes. In response to determining that the failed node has been restored or repaired, the system can smoothly integrate it into the system.

[0061] A single host may include at least one switch device that supports the CXL (Compute Express Link) protocol. The upstream port of the switch device is used to connect to the host, and the downstream port is used to connect to other switch devices, multiple accelerator devices, and at least one memory card. After the downstream port of the switch device completes configuration of the configuration space and allocation of the BAR (Base Address Register) bus base address, each downstream device can periodically broadcast its free memory information to each other via the PCIE (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard) interface, thereby enabling direct PCIE communication between downstream devices. The switch device can effectively facilitate communication and interconnection between the host and its downstream devices. PCIE is a high-speed serial point-to-point dual-channel, high-bandwidth transmission method, and the connected devices are allocated exclusive channel bandwidth. PCI Express also has multiple specifications, ranging from PCI Express x1 to PCI Express x32, to meet the needs of low-speed and high-speed devices that will appear in the future. The PCI Express interface is the PCIE 3.0 interface, with an 8 Gbps bit rate, approximately double the bandwidth of the previous generation. It also incorporates a range of key features, such as transmitter and receiver equalization and clock data recovery, to improve data transmission and protection. PCI Express bus links support full-duplex communication between any two endpoints. A channel consists of two differential signal pairs—one for receiving data and the other for transmitting data—as well as a differential reference clock pair. Therefore, each channel consists of four data lines. Conceptually, each channel functions as a full-duplex byte stream, simultaneously transmitting 8-bit byte packets in both directions between link endpoints. A physical PCI Express link may contain from one to 32 lanes.

[0062] The CXL protocol, compatible with the PCIe standard, addresses cache and memory access coherency issues across heterogeneous devices, enabling fast global access to the accelerator device's memory, the host's memory, and the CPU's cache by CXL-enabled devices. CXL technology maintains memory coherency between the CPU's memory space and the memory on connected devices, enabling resource sharing (or memory pooling) for higher performance, reduced software stack complexity, and lower overall system cost. Therefore, the CXL interface enables communication between the CPU and accelerator devices such as GPUs and FPGAs, resulting in higher data access efficiency and lower local data access latency. CXL consists of three dynamically multiplexed subprotocols on a single link: an I / O (input / output) protocol similar to PCIe (i.e., CXL.io), a cache protocol (i.e., CXL.cach), and a memory access protocol (i.e., CXL.memory). Depending on the specific accelerator device usage model, all or only one of the subprotocols can be enabled. The CXL.io protocol is required for operations such as discovery and enumeration, error reporting, and host physical address lookup. A key advantage of CXL is that it provides a low-latency, high-bandwidth path for acceleration devices to access the system. CXL's memory cache coherence allows memory resources to be shared between the host CPU and the acceleration devices.

[0063] In some examples, downstream acceleration devices of a switch device in a host can communicate directly with each other using the DMA (Direct Memory Access) method without going through the host. In response to any acceleration device needing to transmit data, the other acceleration devices in the current host used to process the data are determined based on the free memory capacity and the function of the acceleration card, and the memory free information of other acceleration devices is queried in the host; then the data to be transmitted is written into the memory of other acceleration devices using the direct memory access controller and free memory information in the host. The switch device is a multi-network port device whose main function is to forward data between different ports, and a data interface is also reserved for docking with other devices.

[0064] In some embodiments, any memory board includes: a non-volatile storage module, the non-volatile storage module including a storage controller and a non-volatile storage area; the storage controller is used to perform memory allocation operations, memory release operations, data storage operations, address mapping operations and / or memory request scheduling on the non-volatile storage area; the non-volatile storage area is used to respond to the memory allocation operations, memory release operations, data storage operations, address mapping operations and / or memory request scheduling performed by the storage controller. The non-volatile storage area is divided into a training data area, a full model weight data area, a subtask weight (the weight of a certain layer in the model) data area, and other areas.

[0065] In some embodiments, any memory board also includes: a board scheduler. The board scheduler can be implemented using an FPGA or an ASIC (Application Specific Integrated Circuit) with solidified related functions. The board scheduler is used to achieve data consistency between the memory of the host to which the current memory board belongs and the non-volatile storage area in the current memory board through the CXL protocol. In some embodiments, the board scheduler uses the data consistency standard provided by the CXL protocol to implement a related engine, thereby using this engine to instantly synchronize the data in the host memory and the non-volatile storage area in the memory board.

[0066] In some embodiments, any memory board further includes a priority arbiter. The priority arbiter comprises a priority list, a priority queue, and an arbitration module. The priority list is used to store priority levels and corresponding priority events. The priority queue comprises multiple queues, one for each priority level, which are used to store access events for each priority level in a first-in, first-out manner. The arbitration module is used to resolve access conflicts and store lower-priority access events in the corresponding access queue. The priority arbiter can be implemented using FPGA hardware or software.

[0067] The priority arbiter determines a priority based on the memory access address in the received memory access request and processes the memory access request according to the determined priority. The memory access request is: a request from the host to which the current memory board belongs to access the non-volatile storage area in the current memory board, a request from an acceleration device in the host to which the current memory board belongs to access the non-volatile storage area in the current memory board, a request from the current memory board to access its own non-volatile storage area, and / or a request from the current memory board to access memory boards in other hosts. It can be seen that the present system supports the host to which the current memory board belongs to access the non-volatile storage area in the current memory board, the acceleration device in the host to which the current memory board belongs to access the non-volatile storage area in the current memory board, the current memory board to access its own non-volatile storage area, and / or mutual access between memory boards of different hosts. In some embodiments, if the priority arbiter determines that the memory access request is a request from an acceleration device in the host to which the current memory board belongs to access the non-volatile storage area in the current memory board, then the priority is determined to be the highest. In some embodiments, in response to determining that the memory access request is not a request from an accelerator device in a host to which the current memory board belongs to access a non-volatile storage area in the current memory board, the priority arbiter determines a priority according to a preset policy. The preset policy may prioritize the different requests, for example: a request from an accelerator device in a host to which the current memory board belongs to access a non-volatile storage area in the current memory board has the highest priority, a request from the current memory board to access its own non-volatile storage area has the second highest priority, a request from memory boards of different hosts to access each other has the third highest priority, and a request from the host to which the current memory board belongs to access a non-volatile storage area in the current memory board has the lowest priority.

[0068] The board's computing unit can be a multi-core parallel ASIC, or an ASIC or FPGA that supports deep learning algorithms. The board's computing unit is used to collect and calculate weight data. It can obtain weight data output by the accelerator device in the host where the current board is located and / or accelerator devices in other hosts, and perform calculations on it.

[0069] The network module is used to communicate with the memory boards in other hosts through the target protocol. The network module can be an FPGA that supports protocols such as TCP / RDMA or an ASIC module that integrates the network protocol. The network module synchronizes the weight data stored in the memory board to which the network module belongs to the memory board in other hosts through the RDMA protocol or the TCP (Transmission Control Protocol) protocol; and receives the weight data sent by the memory boards in other hosts through the RDMA protocol. RDMA can reduce network latency and transfer data directly to the non-volatile storage area of ​​different memory boards, so that data can be quickly transmitted on different memory boards without causing any impact on the host operating system. In this way, the host's processing function is not required, thereby freeing up the host's memory and bandwidth and improving the host system performance.

[0070] In some embodiments, a control host initializes the memory boards in multiple hosts, configures the interaction mode, memory size, and starting offset address of each memory board, uniformly addresses each memory board to obtain an addressing table, and grants access rights to each memory board. Any host, any memory board, or any acceleration device can access a target memory address using the base address and offset address corresponding to the target memory address. The target memory address is the address of a non-volatile storage area in a non-current memory board.

[0071] In some embodiments, any host maps the address of the non-volatile storage area in its own memory card to the host memory space through CXL, and implements conversion between the host memory address and the card memory address based on the mapped address mapping relationship.

[0072] In some implementations, any host divides a received subtask into multiple task blocks based on the number of accelerators within the host, distributes the multiple task blocks to the accelerators within the host, and controls the accelerators within the host to execute the multiple task blocks in parallel. This allows different accelerators within the same host to execute the subtasks received by the host in parallel, thereby accelerating task processing.

[0073] In some implementations, the control host calculates an average of the weight data stored in the memory boards of the multiple hosts, and uses the average as the latest weight data.

[0074] In some embodiments, any acceleration device includes a device scheduler. The device scheduler can be implemented using an FPGA or an ASIC with embedded related functions. The device scheduler is used to achieve data consistency between the memory of the current acceleration device and the memory of the host to which the current acceleration device belongs via the CXL protocol.

[0075] In some embodiments, any accelerator device includes multiple device computing units. These device computing units can be multi-core parallel ASICs, or ASICs or FPGAs that support deep learning algorithms. These multiple device computing units are configured to execute received task blocks; these task blocks are generated by dividing the subtasks received by the current accelerator device into the number of accelerator devices in the host to which the current accelerator device belongs.

[0076] In some examples, the control host also includes a data transmission device. The data transmission device is used to: directly connect to a memory address of the corresponding remote device according to the memory request of at least one remote device, so that the host can directly access the model training task and its related data stored in the remote device with the help of the data transmission device. Among them, the data transmission device includes: an address resolution module and multiple memory access modules; each memory access module is used to directly connect to a memory address of the corresponding remote device according to the memory request of at least one remote device, and supports time-sharing multiplexing of different remote devices connected to it, and each remote device directly accessed by each memory access module shares the processor and acceleration device of the control host. The memory access module includes an RDMA unit.

[0077] In some embodiments, the processor of the control host is further configured to: query an idle memory access module in the data transmission device based on a memory request sent by any remote device; in response to finding an idle memory access module, generate an address configuration operation for the idle memory access module and send the address configuration operation to the idle memory access module; accordingly, the idle memory access module is configured to: configure the memory address range corresponding to any remote device carried by the address configuration operation in itself according to the address configuration operation, and establish a remote memory access connection with the current remote device; accordingly, the address resolution module is configured to: record the mapping relationship between the memory address range, the current remote device, and the memory access module configured with the memory address range. The processor is further configured to: generate an address configuration operation for the idle memory access module in the data transmission device based on a memory request sent by any remote device, and send the address configuration operation to the data transmission device; accordingly, the data transmission device is configured to: cause the idle memory access module to configure the memory address range corresponding to any remote device carried by the address configuration operation in itself according to the address configuration operation, and establish a remote memory access connection with the current remote device;

[0078] In some embodiments, the host controller's processor is further configured to: in response to a query that no free memory access module is found, return a request failure message to the corresponding remote device. In some embodiments, the host controller's processor is further configured to: detect the memory space size of a memory address range based on a memory request sent by any remote device; determine a memory mode that matches the memory space size; and manage the corresponding memory space according to the memory mode. In some embodiments, the host controller's processor is further configured to: set a configurable address range size for each memory access module in the data transmission device.

[0079] On the other hand, any host or any memory board or any acceleration device determines that the target memory address belongs to the local area according to the addressing table, and then accesses the target memory address through the PCIE protocol; and / or any host or any memory board or any acceleration device determines that the target memory address does not belong to the local area according to the addressing table, and then accesses the target memory address through the RDMA protocol. Any acceleration device stores the intermediate weights obtained during the training process of executing the subtask into the subtask weight data area of ​​the memory board of the same host; in response to the subtask weight data area being full, the flag corresponding to the subtask weight data area is set from 0 to 1. Accordingly, the board calculation unit in any memory board reads the intermediate weights in the local or remote subtask weight data area according to the addressing table, and sets the flag corresponding to the corresponding subtask weight data area from 1 to 0; calculates the mean of the read data, and writes the mean into the local or remote full model weight data area according to the addressing table.

[0080] This embodiment uses the acceleration device in the host to perform model training tasks, and uses the memory board in the host to store training data, intermediate results and weight data, so that the host does not need to perform tasks or store data, thereby reducing the load on the host. The acceleration device performs model training tasks to improve model training efficiency. The entire training process only requires the host to perform task allocation, and a large amount of computing and memory access work is offloaded to the acceleration device and memory board, which can reduce the risk of downtime during training and the difficulty of recovering model weight data.

[0081] Referring to Figure 2, this embodiment provides a distributed cluster consisting of several host nodes equipped with CXL-PMEM (i.e., memory boards) and CXL-GPUs (i.e., acceleration devices). Each host node includes a CXL switch (connection device), a GPU accelerator with several CXL interfaces, and one or more CXL memory expansion devices (i.e., memory boards) with integrated FPGAs.

[0082] Each host can connect to multiple cascaded CXL switches, supporting up to 4096 memory boards. Accelerators and memory boards are designed as CXL type 2 devices (including I / O and cache) and connected to the host via a CXL switch. Due to their device type, the internal memory of the memory board is exposed to the accelerator and the corresponding host. The memory board cannot see the host memory, but the accelerator can access the host memory. The memory board and accelerator can access each other; the host can access the memory board, and vice versa.

[0083] Please refer to Figure 3. The memory board is a memory expansion card implemented by FPGA, which includes the following four modules: computing unit, storage scheduler (i.e., board scheduler), PMEM (Persistent Memory) storage module (i.e., non-volatile storage module) and network module.

[0084] The storage scheduler implements the device coherency engine (dcoh) through CXL to maintain memory consistency with the host. This includes a memory access queue and a priority arbiter. The memory card is designed with a memory access queue and a priority arbiter. Memory access on this device primarily involves the following: the host reads and writes to the memory card's PMEM storage module; the accelerator reads and writes to the memory card's PMEM storage module; the memory card reads and writes to its own PMEM storage module; and the memory card reads and writes to the PMEM storage module of other nodes' memory cards.

[0085] Priority Arbiter: This determines priority based on memory access addresses and read / write commands, placing concurrent memory access requests into a memory access queue. Because forward and reverse computations take up the most time during training, to ensure training latency, the memory access operations of the accelerator device accessing the memory card are prioritized, while other memory access operations are hidden in the forward and reverse computations.

[0086] Computing unit: mainly completes the weight collection and calculation of each GPU during training.

[0087] Engine dcoh: This is used for consistency tracking logic between device memory and host memory. The CXL standard ensures consistent memory access, ensuring that the memory accessed by the host and accelerator device is up-to-date and consistent.

[0088] The network module is used to support remote access between memory boards in different hosts. Network protocols such as RDMA / TCP can be used. In this example, RDMA is used as the communication method for description.

[0089] The main functions of the controller in the PEME storage module include memory allocation / release, data placement, address mapping, and memory request scheduling. PMEM is used to store model weights and training data. Because PMEM does not lose data during power failure and its unit storage cost is lower than DDR, it is more suitable for large model training scenarios. PMEM supports access to extended memory in remote memory boards and uniformly addresses memory boards in network nodes. The boards use the CXL.IO protocol. The network-connected memory board addresses can be mapped to the host memory space through the IO MMU. Users can use the virtual address space after address conversion to connect to the memory board.

[0090] As shown in Figure 4, the acceleration device is a GPU compute unit that supports the CXL protocol. It includes the following modules: a memory consistency engine (DCOH), an I / O FIFO (Input Output First Input First Output), a GPGPU (General Purpose GPU) compute module (device compute unit), and DDR (Double Data Rate). The GPGPU compute unit primarily performs forward and reverse computations during training; the I / O FIFO is responsible for pre-fetching training data / model weights and writing model weights during training to the memory card; the memory consistency engine (DCOH) implements consistency tracking logic between device memory and host memory, and DDR (i.e., the subtask weight data area) stores intermediate weights used in the cached computation process.

[0091] In some examples, a model training process includes: before training, the extended memory of the host node in the cluster needs to be initialized, and the extended memory of the node needs to be written into the memory configuration file of each node. The configuration file includes the interaction mode, size and starting offset address of each memory board card of the current host.

[0092] For example, there are three host nodes and the extended memory size of each node is 0x10000000. For memory information, see Table 1.

[0093] Table 1

[0094] Through this unified addressing, each node has access to the extended memory of other nodes, and the base address + offset memory address in each node's memory board points to a unique physical memory. As shown in Table 1, a single memory board supports both PCIE communication and RDMA communication.

[0095] Start training: Since the model is very large and cannot be stored in the memory of a single GPU, the training adopts a node-based model parallel strategy to split the model to the GPU of each node for running. In some embodiments, the GPU in each node runs a shared split model. The model consists of multiple layers, and each layer is split into N GPUs for calculation. As a distributed training, one of the nodes is selected as the master (control host), which is responsible for starting the training task and sending training parameters to other nodes. Each node is responsible for burning and starting the kernel program of the acceleration device and memory board, and passing the training parameters received from the master to the acceleration device and memory board program.

[0096] Initializing training parameter configuration involves allocating memory on the memory card based on the model size, divided into four areas: training data, model weights, layer weights, and other memory areas. Because the model training dataset is extremely large, it needs to be stored in batches on each node's memory card. Because PCIE read latency is lower than RDMA, the training data is stored in the non-volatile storage area of ​​the local memory card.

[0097] Initialize each node's local weight block. This memory is used to store the local weights currently being calculated. It exists in the memory of each node's local memory board, reducing communication latency and being allocated to local DDR storage. This improves read and write performance compared to PMEM. This portion of memory is accessed by the master node for mean calculation and, after updating the data, is written back. The updated data is the mean of the node's data. To prevent read and write conflicts, this area is designed as a ping-pong cache, and a flag bit is also designed. When the accelerator updates the local weight, the flag bit is set to 1. When the computing unit in the memory board detects the flag as 1, it performs mean calculation and writes it to the model weight area, then sets it to 0. Because the accelerator updates the weights for a certain layer of the model, and the mean calculation time of the memory board is much smaller than the accelerator's update time (the calculation / training time of that layer of the model), access conflicts are avoided.

[0098] Reading training data into GPU memory: This operation is performed by the I / O FIFO module of the accelerator device. This module is responsible for pre-fetching training data from the training data area of ​​the memory board and storing the training data in the data FIFO. The training data is then read by the GPGPU computing module of the accelerator device.

[0099] Read model weights in segments: Since the model is large, model segmentation is used to divide the weights of each layer into each accelerator device according to the number of GPUs of a single node, which can be calculated according to the amount of computation. Assuming that each node has 2 GPUs, the layer is split onto two GPUs according to the amount of computation. This ensures that the two GPUs complete the computation at the same time, reducing the waiting time required for synchronization. This operation is also completed by the I / O FIFO module of the accelerator device. The module is responsible for pre-fetching training data from the model weight area of ​​the memory board, which stores the mean value of the last trained model weight, storing the training data in the weight FIFO, and then reading it by the GPGPU computing module of the accelerator device; by reading the model weights in layers / blocks, the local GPU memory usage can be reduced, and only a small amount of model weights and intermediate data need to be cached locally.

[0100] Forward calculation: Each node performs forward calculation based on the read data and weights to calculate the model loss of each node.

[0101] Partial reverse calculation: The reverse calculation is also performed in blocks and completed by the GPGPU module of the acceleration device.

[0102] GPU updates local weights: After each node calculates the weight, it stores the weight in the graphics memory of the acceleration device.

[0103] The model weights are stored in the node memory board DDR: This operation is completed by the accelerator device I / O FIFO module, which records the pingpong address last sent to the memory board, reads the accelerator device local memory weights and stores them in the corresponding weight area of ​​pmem memory. When the cache area is full, it will be marked as set to 1.

[0104] Update PMEM model weights: The memory board's calculation module reads the ping-pong cache of each node's weight, calculates the average, and stores it in the model weight memory in the PMEM memory, setting the flag to 0. Because the memory board's average calculation time is much shorter than the accelerator's update time, a timer mode is used to read the flag bit.

[0105] Repeat the above process until the backpropagation of all model blocks is completed.

[0106] As can be seen, this embodiment stores training data, intermediate training results, and training weights on a memory card device via a CXL bus PMEM memory expansion device. Utilizing cross-node CXL mem unified addressing, distributed training devices only need to store one copy of the model weights, significantly saving model cache space and enabling distributed large-model training in small-scale clusters. The entire training process requires only a small number of CPUs, with a large amount of computing and memory access work offloaded to the CXL device, significantly saving GPU memory, computing power, and bandwidth. PMEM's power-loss-resistant feature can significantly reduce the risk of downtime during training and the difficulty of recovering model weight data, achieving the effect of expanding GPU memory and CPU memory, thereby supporting the training of large models in small-scale clusters.

[0107] The following introduces a data processing method provided in an embodiment of the present application. The data processing method described below can be referenced with other embodiments described in this document.

[0108] As shown in FIG5 , an embodiment of the present application discloses a data processing method, which is applied to a distributed system. The distributed system includes multiple hosts, and any host includes: multiple acceleration devices and at least one memory board.

[0109] S501. The multiple hosts include a control host, which divides the same model training task into multiple subtasks and distributes the multiple subtasks to multiple hosts.

[0110] S502. The multiple hosts use the multiple acceleration devices in their own hosts to execute the received subtasks in parallel, and use the memory boards in their own hosts to store the training data, intermediate results and weight data corresponding to the corresponding subtasks.

[0111] S503. The control host uses its own memory board to collect and process the weight data stored in the memory boards of multiple hosts, and writes the latest weight data obtained through processing into the memory boards of multiple hosts.

[0112] In some embodiments, the network module in any memory board synchronizes the weight data stored in the current memory board to the memory boards in other hosts through the RDMA protocol; and receives the weight data sent by the memory boards in other hosts through the RDMA protocol.

[0113] In some embodiments, the control host initializes the memory boards in multiple hosts, configures the interaction mode, memory size and starting offset address of each memory board, uniformly addresses each memory board to obtain an addressing table, and enables access rights between each memory board.

[0114] In some embodiments, any memory board includes: a non-volatile storage module, the non-volatile storage module including a storage controller and a non-volatile storage area;

[0115] The memory controller is used to perform memory allocation operations, memory release operations, data storage operations, address mapping operations and / or memory request scheduling on the non-volatile memory area;

[0116] The non-volatile storage area is used to respond to memory allocation operations, memory release operations, data storage operations, address mapping operations and / or memory request scheduling performed by the storage controller.

[0117] In some embodiments, the non-volatile storage area is divided into a training data area, a full model weight data area, a subtask weight data area, and other areas.

[0118] In some embodiments, any memory board further includes: a board scheduler, which is configured to achieve data consistency between the memory of the host to which the current memory board belongs and the non-volatile storage area of ​​the current memory board through the CXL protocol.

[0119] In some implementations, any memory board further includes a priority arbitrator.

[0120] The priority arbiter determines a priority based on the memory access address in the received memory access request and processes the memory access request according to the determined priority. The memory access request is: a request from the host to which the current memory board belongs to access the non-volatile storage area in the current memory board, a request from an acceleration device in the host to which the current memory board belongs to access the non-volatile storage area in the current memory board, a request from the current memory board to access its own non-volatile storage area, and / or a request from the current memory board to access memory boards in other hosts. It can be seen that the present system supports the host to which the current memory board belongs to access the non-volatile storage area in the current memory board, the acceleration device in the host to which the current memory board belongs to access the non-volatile storage area in the current memory board, the current memory board to access its own non-volatile storage area, and / or mutual access between memory boards of different hosts. In some embodiments, if the priority arbiter determines that the memory access request is a request from an acceleration device in the host to which the current memory board belongs to access the non-volatile storage area in the current memory board, then the priority is determined to be the highest. In some embodiments, in response to determining that the memory access request is not a request from an accelerator device in a host to which the current memory board belongs to access a non-volatile storage area in the current memory board, the priority arbiter determines a priority according to a preset policy. The preset policy may prioritize the different requests, for example: a request from an accelerator device in a host to which the current memory board belongs to access a non-volatile storage area in the current memory board has the highest priority, a request from the current memory board to access its own non-volatile storage area has the second highest priority, a request from memory boards of different hosts to access each other has the third highest priority, and a request from the host to which the current memory board belongs to access a non-volatile storage area in the current memory board has the lowest priority.

[0121] In some embodiments, any memory board further includes: a board computing unit, which is used to collect and calculate weight data.

[0122] In some embodiments, any memory board further includes: a network module, the network module being configured to communicate with memory boards in other hosts via a target protocol.

[0123] In some embodiments, the network module synchronizes the weight data stored in the memory board to which the network module belongs to the memory board in other hosts through the RDMA protocol; and receives the weight data sent by the memory board in other hosts through the RDMA protocol.

[0124] In some embodiments, the control host initializes the memory boards in multiple hosts, configures the interaction mode, memory size and starting offset address of each memory board, uniformly addresses each memory board to obtain an addressing table, and enables access rights between each memory board.

[0125] In some embodiments, any host or any memory board or any acceleration device accesses the target memory address through the base address and offset address corresponding to the target memory address, and the target memory address is the address of a non-volatile storage area in a non-current memory board.

[0126] In some embodiments, any host maps the address of the non-volatile storage area in its own memory card to the host memory space through the CXL protocol, and implements conversion between the host memory address and the card memory address based on the mapped address mapping relationship.

[0127] In some implementations, any host divides a received subtask into multiple task blocks based on the number of accelerators within the host, distributes the multiple task blocks to the accelerators within the host, and controls the accelerators within the host to execute the multiple task blocks in parallel. This allows different accelerators within the same host to execute the subtasks received by the host in parallel, thereby accelerating task processing.

[0128] In some implementations, the control host calculates an average of the weight data stored in the memory boards of the multiple hosts, and uses the average as the latest weight data.

[0129] In some implementations, any acceleration device includes a device scheduler, and the device scheduler is configured to achieve data consistency between a memory of the current acceleration device and a memory of a host to which the current acceleration device belongs, through a CXL protocol.

[0130] In some embodiments, any acceleration device includes multiple device computing units, which are used to execute received task blocks; the task blocks are obtained by dividing the subtasks received by the current acceleration device according to the number of acceleration devices in the host to which the current acceleration device belongs.

[0131] On the other hand, any host or any memory board or any acceleration device determines that the target memory address belongs to the local according to the addressing table, and then accesses the target memory address through the PCIE protocol; and / or any host or any memory board or any acceleration device determines that the target memory address does not belong to the local according to the addressing table, and then accesses the target memory address through the RDMA protocol. Any acceleration device stores the intermediate weights obtained during the training process of executing the subtask into the subtask weight data area of ​​the memory board of the same host; in response to determining that the subtask weight data area is full, the flag corresponding to the subtask weight data area is set from 0 to 1. Accordingly, the board calculation unit in any memory board reads the intermediate weights in the local or remote subtask weight data area according to the addressing table, and sets the flag corresponding to the corresponding subtask weight data area from 1 to 0; calculates the mean of the read data, and writes the mean into the local or remote full model weight data area according to the addressing table.

[0132] Among them, regarding the working process of each module and unit in this embodiment, reference can be made to the corresponding content disclosed in the aforementioned embodiments, and no further details will be given here.

[0133] It can be seen that this embodiment uses the acceleration device in the host to perform model training tasks, and uses the memory board in the host to store training data, intermediate results and weight data, so that the host does not need to perform tasks or store data, thereby reducing the load on the host. The acceleration device performs model training tasks to improve the efficiency of model training. The entire training process only requires the host to perform task allocation, and offloads a large amount of computing and memory access work to the acceleration device and memory board, which can reduce the risk of downtime during training and the difficulty of recovering model weight data.

[0134] The following describes an electronic device provided in an embodiment of the present application. The electronic device described below can be cross-referenced with other embodiments described herein. The electronic device in this embodiment can be the host, memory card, or acceleration device in the above embodiments.

[0135] The present application discloses an electronic device, including:

[0136] one or more processors; and

[0137] A memory associated with one or more processors, the memory is used to store computer-readable instructions, and the computer-readable instructions implement the method disclosed in any of the above embodiments when read and executed by the one or more processors.

[0138] Furthermore, an embodiment of the present application also provides an electronic device. The electronic device can be either a server as shown in FIG6 or a terminal as shown in FIG7 . FIG6 and FIG7 are both structural diagrams of electronic devices according to an exemplary embodiment, and the contents of the diagrams are not to be construed as limiting the scope of use of the present application.

[0139] Figure 6 is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server may include: at least one processor, at least one memory, a power supply, a communication interface, an input / output interface, and a communication bus. The memory is used to store computer-readable instructions, which are loaded and executed by the processor to implement the relevant steps of the data processing disclosed in any of the aforementioned embodiments.

[0140] In this embodiment, the power supply is used to provide operating voltage for each hardware device on the server; the communication interface can create a data transmission channel between the server and external devices, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not limited here; the input and output interface is used to obtain external input data or output data to the outside world, and its interface type can be selected according to application needs and is not limited here.

[0141] In addition, the memory as a carrier for resource storage can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon include operating system, computer-readable instructions and data, etc. The storage method can be temporary storage or permanent storage.

[0142] The operating system manages and controls the server's hardware devices and computer-readable instructions, enabling the processor to operate and process data in memory. It can be Windows Server, NetWare, Unix, Linux, or other operating systems. In addition to computer-readable instructions for implementing the data processing methods disclosed in any of the aforementioned embodiments, the computer-readable instructions may also include computer-readable instructions for performing other specific tasks. Data may include application update information and other data, such as application developer information.

[0143] FIG7 is a schematic structural diagram of a terminal provided in an embodiment of the present application. The terminal may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0144] Generally, the terminal in this embodiment includes: one or more processors and memories associated with the one or more processors.

[0145] Among them, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA, and PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor may be integrated with a GPU, which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0146] The memory may include one or more computer-readable storage media, which may be non-transitory. The memory may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory is at least used to store the following computer-readable instructions, wherein, after the computer-readable instructions are loaded and executed by the processor, the relevant steps in the data processing method performed by the terminal side disclosed in any of the aforementioned embodiments can be implemented. In addition, the resources stored in the memory may also include an operating system and data, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system may include Windows, Unix, Linux, etc. The data may include but is not limited to update information of the application.

[0147] In some embodiments, the terminal may further include a display screen, an input and output interface, a communication interface, a sensor, a power supply, and a communication bus.

[0148] Those skilled in the art will appreciate that the structure shown in FIG. 7 does not limit the terminal and may include more or fewer components than shown in the figure.

[0149] The following describes a readable storage medium provided in an embodiment of the present application. The readable storage medium described below can be cross-referenced with other embodiments described herein. The readable storage medium is a computer-readable storage medium that, as a carrier for storing resources, can be a read-only memory, random access memory, a magnetic disk, or an optical disk. The resources stored thereon include an operating system, computer-readable instructions, and data, and the storage method can be either temporary or permanent.

[0150] As shown in Figure 8, a structural diagram of a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores computer-readable instructions, which, when executed by one or more processors, implement the data processing method disclosed in the aforementioned embodiments.

[0151] In this embodiment, the computer-readable instructions executed by the processor can implement the following steps: performing memory allocation operations, memory release operations, data storage operations, address mapping operations, and / or memory request scheduling on the non-volatile storage area. Responding to the memory allocation operations, memory release operations, data storage operations, address mapping operations, and / or memory request scheduling performed by the storage controller. The non-volatile storage area is divided into a training data area, a full model weight data area, a subtask weight data area, and other areas.

[0152] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0153] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of readable storage medium known in the art.

[0154] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A data processing system, characterized in that: The system comprises a plurality of hosts, wherein any host comprises: a plurality of acceleration devices and at least one memory card; The multiple hosts include a control host, which divides the same model training task into multiple subtasks and distributes the multiple subtasks to the multiple hosts; The multiple hosts use their own multiple acceleration devices to execute the received subtasks in parallel, and use their own memory boards to store the training data, intermediate results and weight data corresponding to the corresponding subtasks; and The control host uses the memory board in itself to collect and process the weight data stored in the memory boards in the multiple hosts, and writes the latest weight data obtained by processing back to the memory boards in the multiple hosts.

2. The system according to claim 1, wherein: Any memory board includes: a non-volatile storage module, wherein the non-volatile storage module includes a storage controller and a non-volatile storage area; The storage controller is used to perform memory allocation operations, memory release operations, data storage operations, address mapping operations and / or memory request scheduling on the non-volatile storage area; and The non-volatile storage area is used to respond to memory allocation operations, memory release operations, data storage operations, address mapping operations and / or memory request scheduling performed by the storage controller.

3. The system according to claim 2, characterized in that The non-volatile storage area is divided into a training data area, a full model weight data area, a subtask weight data area and other areas.

4. The system according to claim 2, wherein: Any memory board also includes: a board scheduler, which is used to achieve data consistency between the memory of the host to which the current memory board belongs and the non-volatile storage area in the current memory board through the computational fast link CXL protocol.

5. The system according to claim 2, wherein: Any memory board also includes: a priority arbiter, which determines a priority according to a memory access address in a received memory access request and processes the memory access request according to the determined priority.

6. The system according to claim 5, characterized in that The memory access request is at least one of: a request from the host to which the current memory board belongs to access the non-volatile storage area in the current memory board, a request from an acceleration device in the host to which the current memory board belongs to access the non-volatile storage area in the current memory board, a request from the current memory board to access its own non-volatile storage area, and a request from the current memory board to access a memory board in another host.

7. The system according to claim 5, characterized in that The priority arbiter determines that the priority is the highest in response to determining that the memory access request is a request for an acceleration device in a host to which the current memory board belongs to access a non-volatile storage area in the current memory board.

8. The system according to claim 5, wherein: The priority arbiter determines the priority according to a preset policy in response to determining that the memory access request is not a request for an acceleration device in a host to which the current memory board belongs to access a non-volatile storage area in the current memory board.

9. The system according to claim 2, wherein: Any memory board also includes: a board calculation unit, which is used to collect and calculate weight data.

10. The system according to claim 2, wherein: Any memory board also includes: a network module, which is used to communicate with memory boards in other hosts through a target protocol.

11. The system according to claim 10, wherein: The network module is used for: Synchronize the weight data stored in the memory card of the network module to the memory cards in other hosts through the RDMA protocol; as well as Receive weight data sent by memory boards in other hosts through the RDMA protocol.

12. The system according to any one of claims 1 to 11, characterized in that The control host initializes the memory boards in the multiple hosts, configures the interaction mode, memory size and starting offset address of each memory board, uniformly addresses each memory board to obtain an addressing table, and allows access rights between each memory board.

13. The system according to claim 12, wherein: Any host or any memory board or any acceleration device accesses the target memory address through the base address and offset address corresponding to the target memory address, and the target memory address is the address of the non-volatile storage area in the non-current memory board.

14. The system according to claim 13, wherein: Any host or any memory board or any acceleration device determines that the target memory address belongs to the local according to the addressing table, and then accesses the target memory address through the PCIE protocol; and / or If any host, any memory board, or any acceleration device determines according to the addressing table that the target memory address is not local, the target memory address is accessed through the RDMA protocol.

15. The system according to any one of claims 1 to 11, characterized in that Any host maps the address of the non-volatile storage area in its own memory board to the host memory space through the CXL protocol, and realizes the conversion between the host memory address and the board memory address according to the address mapping relationship obtained by mapping.

16. The system according to any one of claims 1 to 11, characterized in that Any host divides the received subtasks into multiple task blocks according to the number of acceleration devices in the host, distributes the multiple task blocks to the acceleration devices in the host, and controls the acceleration devices in the host to run the multiple task blocks in parallel.

17. The system according to any one of claims 1 to 11, characterized in that The control host calculates an average value of the weight data stored in the memory boards of the plurality of hosts, and uses the average value as the latest weight data.

18. The system according to any one of claims 1 to 11, characterized in that Any acceleration device includes a device scheduler, which is used to achieve data consistency between the memory of the current acceleration device and the memory of the host to which the current acceleration device belongs through the CXL protocol.

19. The system according to claim 18, wherein: Any acceleration device includes multiple device computing units, which are used to execute received task blocks; the task blocks are obtained by dividing the subtasks received by the current acceleration device according to the number of acceleration devices in the host to which the current acceleration device belongs.

20. The system according to claim 19, wherein: Any acceleration device stores the intermediate weights obtained during the training process of executing the subtask into the subtask weight data area of ​​the memory card of the same host; In response to determining that the subtask weight data area is full, a flag corresponding to the subtask weight data area is set from 0 to 1.

21. The system according to claim 20, wherein: The board compute unit in any memory board is used to: Read the intermediate weight in the local or remote subtask weight data area according to the addressing table, and The flag is set from 1 to 0; as well as Calculate the mean of the read data and write the mean to the local or remote full model weight data area according to the addressing table.

22. A data processing method, characterized in that: Applied to a distributed system, the distributed system includes multiple hosts, and any host includes: multiple acceleration devices and at least one memory card; The multiple hosts include a control host, which divides the same model training task into multiple subtasks and distributes the multiple subtasks to the multiple hosts; The multiple hosts use their own multiple acceleration devices to execute the received subtasks in parallel, and use their own memory boards to store the training data, intermediate results and weight data corresponding to the corresponding subtasks; and The control host uses the memory board in itself to collect and process the weight data stored in the memory boards in the multiple hosts, and writes the latest weight data obtained by processing into the memory boards in the multiple hosts.

23. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer-readable instructions, which implement the method of claim 22 when executed by one or more processors.

Images