A data processing system and method
By using a high-speed computing link interface and persistent memory between storage and computing devices, memory access operations are optimized, solving the problem of CPU overload during model training. This enables efficient large-scale single-machine model training, reducing resource consumption and training costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INSPUR (BEIJING) ELECTRONICS INFORMATION IND CO LTD
- Filing Date
- 2023-11-28
- Publication Date
- 2026-06-19
AI Technical Summary
In existing technologies, the computational resource requirements for model training data are high, leading to CPU overload, long training cycles, and increased costs, especially when multi-GPU computing is offloaded to the CPU.
By designing both storage and computing devices to include high-speed computing link interfaces, data interaction is achieved through high-speed computing links. Combined with persistent memory and ping-pong caching, the response priority of memory access operations is optimized, reducing the resource consumption of the host CPU and saving cache space and bandwidth of the computing devices.
It enables large-scale model training on a single machine, reducing CPU resource consumption, lowering training costs, improving training efficiency, and reducing the risk of downtime.
Smart Images
Figure CN117608838B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing, and in particular to a data processing system and method. Background Technology
[0002] Currently, with the development of BERT (Bidirectional Encoder Representations from...)
[0003] Transformers (a type of self-encoding language model) and GPT (Generative Pre-Trained Language Modeling)
[0004] The increasing amount of training data for models like the Transformer (a generative pre-trained Transformer model) places a greater demand on computational resources for training, requiring significant amounts of GPUs (Graphics Processing Units) and storage. Due to the massive amount of training data for large models, training cycles are lengthy. To reduce the training time and cost of large models, current solutions offload training data and GPU computations to the CPU (Central Processing Unit) for large-scale model training. Offloading training data to CPU memory consumes CPU bandwidth and memory access time when reading training data. Offloading GPU computations to the CPU increases CPU load, and when multiple GPU computations need to be offloaded, it can lead to CPU overload.
[0005] Therefore, how to provide a solution to the above-mentioned technical problems is a problem that needs to be solved by those skilled in the art. Summary of the Invention
[0006] The purpose of this invention is to provide a data processing system and method that can reduce the resource consumption of the host's central processing unit and save the cache space, computing power and bandwidth of the computing device.
[0007] To address the aforementioned technical problems, the present invention provides a data processing system, comprising a host, at least one storage device equipped with a high-speed computing link interface, and at least one computing device equipped with the high-speed computing link interface, wherein the storage device, the computing device, and the host are connected via the high-speed computing link interface, wherein:
[0008] The host is used to acquire training data and write the training data into the storage device;
[0009] The computing device is configured to obtain the training data and model weight data from the storage device, obtain local weight data based on the training data and model weight data, and write the local weight data into the storage device.
[0010] The storage device is used to store the training data and the local weight data, and to obtain and store the model weight data based on all the local weight data.
[0011] In one exemplary embodiment, the storage device includes a first storage medium, which is a storage medium in which data stored is not lost after power failure;
[0012] The host is further configured to divide the first storage medium into multiple persistent storage areas, determine any one of the persistent storage areas as the first persistent storage area, and configure the first persistent storage area as a ring memory;
[0013] The process of writing the training data into the storage device includes:
[0014] The training data is written to the circular memory until the address pointed to by the write pointer and the address pointed to by the read pointer of the circular memory are the same;
[0015] The process of obtaining the training data from the storage device includes:
[0016] The training data is obtained from the circular memory.
[0017] In one exemplary embodiment, the process of writing the training data into the storage device includes:
[0018] The training data is divided into multiple sub-data groups; the difference in the amount of data between any two sub-data groups is less than a preset threshold.
[0019] The process of writing the training data into the circular memory until the address pointed to by the write pointer and the address pointed to by the read pointer of the circular memory are the same includes:
[0020] For each of the sub-data groups, the sub-data group is written into the circular memory until the address pointed to by the write pointer and the address pointed to by the read pointer of the circular memory are the same.
[0021] In an exemplary embodiment, the process of dividing the first storage medium into a plurality of persistent storage regions includes:
[0022] The first storage medium is divided into multiple persistent storage areas according to the size of the preset model.
[0023] In one exemplary embodiment, the first storage medium is persistent memory.
[0024] In an exemplary embodiment, the host is further configured to identify any one of the persistent storage regions other than the first persistent storage region as a second persistent storage region;
[0025] The process of obtaining and storing the model weight data based on all the local weight data includes:
[0026] The model weight data is obtained based on all the local weight data and stored in the second persistent storage area;
[0027] The process of obtaining model weight data from the storage device includes:
[0028] Retrieve model weight data from the second persistent storage area.
[0029] In one exemplary embodiment, the storage device further includes a second storage medium, which is a storage medium for which stored data is lost upon power failure;
[0030] The host is also configured to divide the second storage medium into multiple non-persistent storage areas, each non-persistent storage area corresponding to a computing device.
[0031] The process of writing the local weight data into the storage device includes:
[0032] The local weight data is written to the non-persistent storage area that corresponds to it one-to-one.
[0033] In one exemplary embodiment, the process of dividing the second storage medium into multiple non-persistent storage regions includes:
[0034] Determine the number of computing devices connected to the storage device;
[0035] The second storage medium is divided into the stated number of non-persistent storage regions.
[0036] In one exemplary embodiment, the host is further configured to configure each of the non-persistent storage regions as a ping-pong cache.
[0037] In one exemplary embodiment, the second storage medium is a double-rate synchronous dynamic random access memory.
[0038] In one exemplary embodiment, the storage device further includes:
[0039] The calculation module is used to obtain the local weight data stored in each of the non-persistent storage areas, calculate the average value of each of the local weight data, and obtain the model weight data.
[0040] The control module is used to write the model weight data into the second persistent storage area.
[0041] In one exemplary embodiment, the storage device further includes a scheduling module for determining the response priority of received memory access operations to the first storage medium and / or the second storage medium;
[0042] The control module is also configured to respond to the memory access operation in descending order of response priority.
[0043] In one exemplary embodiment, the computing device has the highest priority in responding to memory access operations on the first storage medium and / or the second storage medium.
[0044] In one exemplary embodiment, the scheduling module includes:
[0045] A priority arbiter is used to determine the response priority of a received memory access operation and write the memory access operation with the tag of the response priority into the corresponding cache queue.
[0046] A cache queue is used to output the memory access operations in descending order of the response priority;
[0047] The process of responding to the memory access operation in descending order of response priority includes:
[0048] The memory access operation is responded to by the output of the cache queue.
[0049] In an exemplary embodiment, the process of determining the response priority of the received memory access operation to the first storage medium and / or the second storage medium includes:
[0050] Determine the memory access address and memory access type of the received memory access operation on the first storage medium and / or the second storage medium;
[0051] The priority of the memory access operation is determined based on the memory access address and the memory access type.
[0052] In one exemplary embodiment, the computing device includes a first-in-first-out (FIFO) module, a processing module, and a storage module, wherein:
[0053] The first-in-first-out module is used to obtain the training data and model weight data from the storage device, and write the local weight data stored in the storage module into the storage device.
[0054] The processing module is used to calculate the local weight data based on the training data and the model weight data;
[0055] The storage module is used to store the local weight data.
[0056] In one exemplary embodiment, the first-in-first-out module includes:
[0057] Data cache queue and weight cache queue;
[0058] The reading unit is configured to read training data of a first preset size from the storage device, write the training data into the data cache queue until the data cache queue meets the first data full condition, read model weight data of a second preset size from the storage device, and write the model weight data into the weight cache queue until the weight cache queue meets the second data full condition.
[0059] The processing module is specifically used to obtain the training data from the data cache queue, obtain the model weight data from the weight cache queue, and calculate the local weight data based on the training data and the model weight data.
[0060] In an exemplary embodiment, the process of writing the local weight data stored in the storage module to the storage device includes:
[0061] Determine the historical address of the last write to the storage device, and determine the current address based on the historical address;
[0062] Write the local weight data stored in the storage module to the storage device at the current address.
[0063] In one exemplary embodiment, the host includes a central processing unit, and the data processing system further includes a high-speed computing link switch, which is connected to the central processing unit, the high-speed computing link interface of the computing device, and the high-speed computing link interface of the storage device, respectively.
[0064] In one exemplary embodiment, both the storage device and the computing device are high-speed computing link-2 devices.
[0065] To address the aforementioned technical problems, the present invention also provides a data processing method applied to a data processing system as described in any of the above claims. The data processing system includes a host, at least one storage device equipped with a high-speed computing link interface, and at least one computing device equipped with the high-speed computing link interface. The storage device, the computing device, and the host are connected via the high-speed computing link interface. The data processing method includes:
[0066] The training data is obtained through the host and written to the storage device;
[0067] The computing device obtains the training data and model weight data from the storage device, obtains local weight data based on the training data and model weight data, and writes the local weight data into the storage device.
[0068] The training data and the local weight data are stored in the storage device, and the model weight data is obtained and stored based on all the local weight data.
[0069] In one exemplary embodiment, the storage device includes a first storage medium, which is a storage medium in which data stored is not lost after power failure;
[0070] The data processing method further includes:
[0071] The host divides the first storage medium into multiple persistent storage areas, designates any one of the persistent storage areas as the first persistent storage area, and configures the first persistent storage area as a ring memory.
[0072] The process of writing the training data into the storage device includes:
[0073] The training data is written to the circular memory until the address pointed to by the write pointer and the address pointed to by the read pointer of the circular memory are the same;
[0074] The process of obtaining the training data from the storage device through the computing device includes:
[0075] The training data is obtained from the ring memory using the computing device.
[0076] In one exemplary embodiment, the data processing method further includes:
[0077] The host determines any one of the persistent storage areas, excluding the first persistent storage area, as the second persistent storage area.
[0078] The process of obtaining and storing the model weight data based on all the local weight data includes:
[0079] The model weight data is obtained based on all the local weight data and stored in the second persistent storage area;
[0080] The process of obtaining model weight data from the storage device through the computing device includes:
[0081] The model weight data is obtained from the second persistent storage area through the computing device.
[0082] In one exemplary embodiment, the storage device further includes a second storage medium, which is a storage medium for which stored data is lost upon power failure;
[0083] The data processing method further includes:
[0084] The host divides the second storage medium into multiple non-persistent storage areas, and each non-persistent storage area corresponds to a computing device.
[0085] The process of writing the local weight data into the storage device includes: writing the local weight data into a non-persistent storage area that corresponds to itself.
[0086] In one exemplary embodiment, after the host divides the second storage medium into multiple non-persistent storage areas, the data processing method further includes:
[0087] The host configures each of the non-persistent storage areas as a ping-pong cache.
[0088] In one exemplary embodiment, the storage device further includes a scheduling module and a control module, and the data processing method further includes:
[0089] The scheduling module determines the response priority of the received memory access operations to the first storage medium and / or the second storage medium.
[0090] The control module responds to the memory access operation in descending order of response priority.
[0091] In an exemplary embodiment, the computing device includes a first-in-first-out (FIFO) module, a processing module, and a storage module. The process of obtaining the training data and model weight data from the storage device via the computing device, obtaining local weight data based on the training data and the model weight data, and writing the local weight data into the storage device includes:
[0092] The training data and model weight data are obtained from the storage device through the first-in-first-out module, and the local weight data stored in the storage module is written to the storage device.
[0093] The processing module calculates the local weight data based on the training data and the model weight data.
[0094] The local weight data is stored through the storage module.
[0095] In an exemplary embodiment, the process of writing the local weight data stored in the storage module to the storage device includes:
[0096] Determine the historical address of the last write to the storage device, and determine the current address based on the historical address;
[0097] Write the local weight data stored in the storage module to the storage device at the current address.
[0098] This invention provides a data processing system in which both the storage device and the computing device include high-speed computing link interfaces. These interfaces connect the two devices, enabling data interaction and reducing CPU resource consumption on the host computer. Furthermore, by leveraging cache consistency between the high-speed computing link devices, all computing devices only need to store one copy of the local weight data, saving cache space, computing power, and bandwidth. This invention also provides a data processing method with the same beneficial effects as the aforementioned data processing system. Attached Figure Description
[0099] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0100] Figure 1 This is a schematic diagram of the structure of a data processing system provided by the present invention;
[0101] Figure 2 This is a schematic diagram of the structure of a storage device provided by the present invention;
[0102] Figure 3 This is a schematic diagram of the structure of a computing device provided by the present invention;
[0103] Figure 4 This is a flowchart illustrating the steps of a data processing method provided by the present invention. Detailed Implementation
[0104] The core of this invention is to provide a data processing system and method that can reduce the resource occupation of the host's central processing unit and save the cache space, computing power and bandwidth of the computing device.
[0105] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0106] Firstly, please refer to Figure 1 , Figure 1 This is a schematic diagram of the structure of a data processing system provided by the present invention. The data processing system includes:
[0107] It includes a host 1, at least one storage device 2 equipped with a high-speed computing link interface, and at least one computing device 3 equipped with a high-speed computing link interface. The storage device 2, computing device 3, and host 1 are connected via the high-speed computing link interface, wherein:
[0108] Host 1 is used to acquire training data and write the training data to storage device 2.
[0109] The computing device 3 is used to obtain training data and model weight data from the storage device 2, obtain local weight data based on the training data and model weight data, and write the local weight data into the storage device 2.
[0110] Storage device 2 is used to store training data and local weight data, and to obtain and store model weight data based on all local weight data.
[0111] In this embodiment, the data processing system includes a host 1 and multiple nodes. Each node includes at least one computing device 3, at least one storage device 2, and a switch. The computing device 3 and storage device 2 in the node interact with the host 1 through the switch. Both the computing device 3 and storage device 2 include a high-speed computing link interface, which enables the host 1 to directly access the memory space of the storage device 2, and the computing device 3 to directly access the memory space of the storage device 2. Specifically, the computing device 3 and storage device 2 are configured as high-speed computing link-2 type, that is, each computing device 3 and storage device 2 is equipped with a functional module to support the protocol corresponding to CXL-2 type. Thanks to the device types of computing device 3 and storage device 2, the internal memory of storage device 2 can be exposed in the local memory of computing device 3.
[0112] In this embodiment, host 1 is used to acquire training data and write it into storage device 2. The working principle of one computing device 3 is explained below as an example; other computing devices 3 are similar. Computing device 3 acquires training data and model weight data obtained from the previous calculation from storage device 2, calculates local weight data based on the training data and model weight data, and writes the local weight data into storage device 2. It can be understood that all computing devices 3 connected to storage device 2 through the high-speed computing link interface will write their own calculated local weight data into storage device 2. Storage device 2 calculates model weight data based on all local weight data and updates the model weight data for the next calculation.
[0113] As can be seen, in this embodiment, both the storage device 2 and the computing device 3 are designed to include a high-speed computing link interface. The two are connected through the high-speed computing link interface to realize data interaction between the storage device 2 and the computing device 3, reducing the CPU resource occupation of the host 1. At the same time, by utilizing the cache consistency between the high-speed computing link devices, all computing devices 3 only need to store one copy of the local weight data, saving the cache space, computing power and bandwidth of the computing devices 3.
[0114] Please refer to Figure 2 and Figure 3 , Figure 2 This is a schematic diagram of the structure of a storage device 2 provided by the present invention. Figure 3 This is a schematic diagram of the structure of a computing device 3 provided by the present invention, based on the above embodiments:
[0115] In one exemplary embodiment, the storage device 2 includes a first storage medium 21, which is a storage medium in which data stored is not lost after power failure;
[0116] The host 1 is also used to divide the first storage medium 21 into multiple persistent storage areas, determine any one of the persistent storage areas as the first persistent storage area, and configure the first persistent storage area as a ring memory;
[0117] The process of writing training data to storage device 2 includes:
[0118] The training data is written to the circular memory until the address pointed to by the write pointer and the address pointed to by the read pointer in the circular memory are the same.
[0119] The process of obtaining training data from storage device 2 includes:
[0120] Retrieve training data from the circular memory.
[0121] In this embodiment, the storage device 2 includes a first storage medium 21, which is a storage medium whose stored data is not lost after power failure, such as PMEM (Persistent Memory). The host 1 divides the first storage medium 21 into multiple persistent storage areas. Each persistent storage area is used to store a certain type of data for training, such as using one persistent storage area to store training data and another persistent storage area to store model weight data. In this embodiment, the persistent storage area used to store training data is defined as the first persistent storage area, and the first persistent storage area is configured as a circular memory. It can be understood that the circular memory includes a read pointer and a write pointer, wherein the read pointer is used to mark the position when reading data from the circular memory, and the write pointer is used to mark the position when writing data to the circular memory.
[0122] In this embodiment, when writing training data to the circular memory, the writing is paused when the read pointer and write pointer coincide (i.e., they both point to the same location). This ensures that the training data is always new. The training data is written to the circular memory in batches until all training data has been written. Furthermore, the computing device 3 can retrieve the training data from the circular memory according to the pre-determined address for subsequent calculations.
[0123] It is understandable that, given the large amount of training data, storing it in persistent memory can reduce training costs and ensure that the training data is not lost after a power outage, thus improving model training efficiency.
[0124] In one exemplary embodiment, the process of writing training data to storage device 2 includes:
[0125] The training data is divided into multiple sub-data groups; the difference in the amount of data between any two sub-data groups is less than a preset threshold.
[0126] The process of writing training data into a circular memory until the address pointed to by the write pointer and the address pointed to by the read pointer in the circular memory includes:
[0127] For each sub-data group, write the sub-data group to the circular memory until the address pointed to by the write pointer and the address pointed to by the read pointer in the circular memory are the same.
[0128] In this embodiment, considering the large amount of training data, the training data is first divided into multiple sub-data groups, each with a roughly equal amount of data. Then, the training data from each sub-data group is sequentially written into the circular memory. During the writing process, it is necessary to ensure that the write pointer and read pointer coincide, thereby guaranteeing that the training data is always up-to-date. It can be understood that if the amount of data in a sub-data group still exceeds the amount of data that the circular memory can support writing, the sub-data groups can be further divided, and the further divided training data can be written into the circular memory.
[0129] In one exemplary embodiment, the process of dividing the first storage medium 21 into a plurality of persistent storage regions includes:
[0130] The first storage medium 21 is divided into multiple persistent storage areas according to the size of the preset model.
[0131] In this embodiment, the first storage medium 21 is divided according to the size of a preset model, so that the capacity of each persistent storage area can support the writing of data of its corresponding category.
[0132] In an exemplary embodiment, host 1 is further configured to determine any one of the multiple persistent storage regions, other than the first persistent storage region, as the second persistent storage region;
[0133] The process of obtaining and storing model weight data based on all local weight data includes:
[0134] The model weight data is obtained based on all local weight data and stored in the second persistent storage area;
[0135] The process of obtaining model weight data from storage device 2 includes:
[0136] Retrieve model weight data from the second persistent storage area.
[0137] In this embodiment, a second persistent storage area is also defined among the multiple persistent storage areas for storing model weight data. Since the model weight data is also quite large, storing it in persistent memory can reduce training costs. Furthermore, the model weight data will not be lost after power failure, improving model training efficiency. Accordingly, the computing device 3 reads the model weight data from the predetermined address of the second persistent storage area for subsequent calculations.
[0138] In one exemplary embodiment, the storage device 2 further includes a second storage medium 22, which is a storage medium for which stored data is lost after power failure;
[0139] The host 1 is also used to divide the second storage medium 22 into multiple non-persistent storage areas, each of which corresponds to a computing device 3.
[0140] The process of writing local weight data to storage device 2 includes:
[0141] Write the local weight data to the non-persistent storage area that corresponds to it one-to-one.
[0142] In this embodiment, the storage device 2 further includes a second storage medium 22, which is a storage medium for data loss after a data drop. In an exemplary embodiment, the second storage medium 22 can be a double-rate synchronous dynamic random access memory. The host 1 divides the second storage medium 22 into multiple non-persistent storage areas. Each computing device 3 connected to the storage device 2 corresponds to one non-persistent storage area. After calculating the local weight data, each computing device 3 writes the local weight data into its corresponding non-persistent storage area so that the storage device 2 can subsequently calculate the model weight data based on the local weight data stored in all non-persistent storage areas.
[0143] In one exemplary embodiment, the process of dividing the second storage medium 22 into a plurality of non-persistent storage regions includes:
[0144] Determine the number of computing devices 3 connected to storage device 2;
[0145] The second storage medium 22 is divided into a number of non-persistent storage areas.
[0146] It is understandable that, assuming the number of computing devices 3 connected to storage device 2 is n, where n is a positive integer, the second storage medium 22 is divided into n non-persistent storage areas.
[0147] In one exemplary embodiment, host 1 is further configured to configure each non-persistent storage region as a ping-pong cache.
[0148] In this embodiment, the non-persistent storage area is configured as a ping-pong cache to minimize read-write conflicts that could increase training latency. Local weight data is stored using a double-rate synchronous dynamic random access memory, which improves read-write performance compared to persistent memory.
[0149] It is understandable that the ping-pong cache includes two caches. When data is written to one cache, the data in the other cache can be read. After the data is written, the two caches are swapped, and then data is written and read again, giving the computing module 23 enough time to process the data and improving efficiency.
[0150] In one exemplary embodiment, the storage device 2 further includes:
[0151] Calculation module 23 is used to obtain local weight data stored in each non-persistent storage area, calculate the average value of each local weight data, and obtain model weight data.
[0152] The control module is used to write model weight data into the second persistent storage area.
[0153] In this embodiment, the calculation module 23 reads local weight data from the cache currently used for reading data in each non-persistent storage area, calculates the average value of each local weight data to obtain model weight data, and the control module writes the model weight data into the second persistent storage area. Correspondingly, when the calculation device 3 finishes calculating the local weight data, it writes the local weight data into the cache currently used for writing data in its corresponding non-persistent storage area.
[0154] In one exemplary embodiment, the storage device 2 further includes a scheduling module 24 for determining the response priority of received memory access operations to the first storage medium 21 and / or the second storage medium 22;
[0155] The control module is also used to respond to memory access operations in descending order of response priority.
[0156] It can be understood that memory access operations on storage device 2 include, but are not limited to, host 1 writing to the first persistent storage area of storage device 2 and reading from the second persistent storage area of storage device 2; computing device 3 reading the address space of the first persistent storage area of storage device 2, computing device 3 reading both the first and second persistent storage areas of storage device 2, computing device 3 writing to the address space of the non-persistent storage area of storage device 2; storage device 2 reading and writing to local memory, and accessing memory in the other area. Response priorities are set for each of the above memory access operations, with higher priority operations being processed first.
[0157] In one exemplary embodiment, the computing device 3 has the highest priority in responding to memory access operations of the first storage medium 21 and / or the second storage medium 22.
[0158] Considering that the forward and backward computations take the longest time in model training, the memory access operation of computing device 3 is given the highest priority to ensure training latency, while other memory access operations are hidden in the forward and backward computation processes.
[0159] In one exemplary embodiment, the scheduling module 24 includes:
[0160] The priority arbiter is used to determine the response priority of the received memory access operation and write the memory access operation with the response priority tag into the corresponding cache queue.
[0161] A cache queue is used to output memory access operations in descending order of response priority.
[0162] The process of responding to memory access operations in descending order of priority includes:
[0163] Memory access operations responded to by the cache queue.
[0164] In one exemplary embodiment, the process of determining the priority of received responses to memory access operations on the first storage medium 21 and / or the second storage medium 22 includes:
[0165] Determine the memory access address and memory access type of the received memory access operation to the first storage medium 21 and / or the second storage medium 22;
[0166] The priority of memory access operations is determined based on the memory access address and memory access type.
[0167] It is understood that the scheduling module 24 includes a priority arbitrator, which is used to determine the response priority of the memory access operation based on the memory access address and memory access type. The scheduling module 24 also includes multiple cache queues, including a cache queue for storing memory access operations from computing device 3 to storage device 2. This cache queue has the highest priority. When there is data in this cache queue, the memory access operations in this cache queue are processed first. Other memory access operations can be placed in other cache queues, or separate cache queues can be set for different cache operations, and then the priority of the cache queue is determined based on the response priority of the cache operation.
[0168] In one exemplary embodiment, the computing device 3 includes a first-in-first-out module 31, a processing module 32, and a storage module 33, wherein:
[0169] The first-in-first-out module 31 is used to obtain training data and model weight data from storage device 2, and write the local weight data stored in storage module 33 into storage device 2.
[0170] Processing module 32 is used to calculate local weight data based on training data and model weight data;
[0171] Storage module 33 is used to store local weight data.
[0172] In one exemplary embodiment, the first-in-first-out module 31 includes:
[0173] Data cache queue and weight cache queue;
[0174] The reading unit is used to read training data of a first preset size from the storage device 2, write the training data into the data cache queue until the data cache queue meets the first data full condition, and read model weight data of a second preset size from the storage device 2, write the model weight data into the weight cache queue until the weight cache queue meets the second data full condition.
[0175] The processing module 32 is specifically used to obtain training data from the data cache queue, obtain model weight data from the weight cache queue, and calculate local weight data based on the training data and model weight data.
[0176] In this embodiment, the computing device 3 includes a first-in-first-out (FIFO) module 31, a processing module 32, and a storage module 33. The storage module 33 is the local memory of the computing device 3, including a memory controller and DDR (Double Data Rate). The FIFO module 31 reads training data into the storage module 33 of the computing device 3. Specifically, the FIFO module 31 prefetches training data from the first persistent storage area of the storage module 33 and stores the training data in the data cache queue, which is then read by the processing module 32 of the computing device 3. The reading logic of the FIFO module 31 is that when the data cache queue is not full, it continuously reads training data from the first persistent storage area and writes it into the data cache queue. The FIFO module 31 is also used to read model weight data in segments. FIFO module 31 is responsible for prefetching training data from the second persistent storage area of storage device 2, which stores the average weights of the previously trained model. The model weight data is stored in a weight cache queue and then read by the processing module 32 of computing device 3. By segmenting the model weight data, the memory usage of the local computing device 3 can be reduced; only a small amount of model weight data and intermediate data needs to be cached locally. The processing module 32 performs forward and backward calculations based on the model weight data and training data, and stores the calculated local weight data into storage module 33.
[0177] In one exemplary embodiment, the process of writing the local weight data stored in the storage module 33 to the storage device 2 includes:
[0178] Determine the historical address of the last write to storage device 2, and determine the current address based on the historical address;
[0179] Write the local weight data stored in storage module 33 to storage device 2 at the current address.
[0180] Considering that the storage area used to store local weight data is a ping-pong cache, the first-in-first-out module 31 determines the historical address of the last time the local weight data was written to its corresponding non-persistent storage area, determines the current address based on the historical address, the current address is the address of the last time it was written, and then writes the currently calculated local weight data into the address of the last time it was written.
[0181] In summary, this invention stores training data, intermediate training results, and training weights in storage device 2 via a high-speed computing link bus and persistent memory expansion device. Utilizing cache consistency between high-speed computing link devices, all computing devices 3 only need to store one copy of the model weight data, significantly saving model cache space and enabling single-machine large model training. By setting a first-in-first-out module 31 in computing device 3 and configuring memory access optimization methods such as ping-pong caching, circular queue caching, memory access priority, and memory access queues in storage device 2, memory access efficiency is improved, training latency is reduced, and training efficiency is enhanced. The entire training process requires only a small amount of CPU involvement, with a large amount of computation and memory access work offloaded to the cxl device, thus significantly saving GPU memory, computing power, and bandwidth. Furthermore, the power-loss non-data loss characteristic of PMEM can greatly reduce the risk of system crashes during training and the difficulty of recovering model weight data.
[0182] Secondly, please refer to Figure 4 , Figure 4 The present invention also provides a flowchart of a data processing method, applicable to a data processing system as described above. The data processing system includes a host, at least one storage device with a high-speed computing link interface, and at least one computing device with a high-speed computing link interface. The storage device, computing device, and host are connected via the high-speed computing link interface. The data processing method includes:
[0183] S401: Obtain training data from the host and write the training data to the storage device;
[0184] S402: Obtain training data and model weight data from storage device through computing device, obtain local weight data based on training data and model weight data, and write local weight data to storage device;
[0185] S403: Store training data and local weight data through a storage device, obtain model weight data based on all local weight data, and store it.
[0186] In one exemplary embodiment, the storage device includes a first storage medium, which is a storage medium in which data stored is not lost after power failure;
[0187] Data processing methods also include:
[0188] The host divides the first storage medium into multiple persistent storage areas, and any one of the persistent storage areas is designated as the first persistent storage area, which is then configured as a ring memory.
[0189] The process of writing training data to a storage device includes:
[0190] The training data is written to the circular memory until the address pointed to by the write pointer and the address pointed to by the read pointer in the circular memory are the same.
[0191] The process of retrieving training data from storage devices includes:
[0192] Retrieve training data from the circular memory.
[0193] In one exemplary embodiment, the process of writing training data to a storage device includes:
[0194] The training data is divided into multiple sub-data groups; the difference in the amount of data between any two sub-data groups is less than a preset threshold.
[0195] The process of writing training data into a circular memory until the address pointed to by the write pointer and the address pointed to by the read pointer in the circular memory includes:
[0196] For each sub-data group, write the sub-data group to the circular memory until the address pointed to by the write pointer and the address pointed to by the read pointer in the circular memory are the same.
[0197] In one exemplary embodiment, the process of dividing the first storage medium into multiple persistent storage regions includes:
[0198] The first storage medium is divided into multiple persistent storage areas according to the size of the preset model.
[0199] In one exemplary embodiment, the first storage medium is persistent memory.
[0200] In one exemplary embodiment, the data processing method further includes:
[0201] The host determines any one of the multiple persistent storage regions, excluding the first persistent storage region, as the second persistent storage region.
[0202] The process of obtaining and storing model weight data based on all local weight data includes:
[0203] The model weight data is obtained based on all local weight data and stored in the second persistent storage area;
[0204] The process of retrieving model weight data from storage devices includes:
[0205] Retrieve model weight data from the second persistent storage area.
[0206] In one exemplary embodiment, the storage device further includes a second storage medium, which is a storage medium for which stored data is lost after power failure;
[0207] This data processing method also includes:
[0208] The host divides the second storage medium into multiple non-persistent storage areas, and each non-persistent storage area corresponds to a computing device.
[0209] The process of writing local weight data to a storage device includes:
[0210] Write the local weight data to the non-persistent storage area that corresponds to it one-to-one.
[0211] In one exemplary embodiment, the process of dividing the second storage medium into multiple non-persistent storage regions includes:
[0212] Determine the number of computing devices connected to the storage device;
[0213] The second storage medium is divided into a number of non-persistent storage areas.
[0214] In one exemplary embodiment, the data processing method further includes:
[0215] Configure each non-persistent storage area as a ping-pong cache via the host.
[0216] In one exemplary embodiment, the second storage medium is a double-rate synchronous dynamic random access memory.
[0217] In one exemplary embodiment, the data processing method further includes:
[0218] The model weight data is obtained by acquiring the local weight data stored in each non-persistent storage area through the computing module in the storage device, calculating the average value of each local weight data, and obtaining the model weight data.
[0219] The model weight data is written to the second persistent storage area through the control module in the storage device.
[0220] In one exemplary embodiment, the data processing method further includes:
[0221] The scheduling module in the storage device determines the response priority of the received memory access operations to the first storage medium and / or the second storage medium.
[0222] The control module responds to memory access operations in descending order of response priority.
[0223] In one exemplary embodiment, the computing device responds with the highest priority to memory access operations on the first storage medium and / or the second storage medium.
[0224] In an exemplary embodiment, the process of determining the response priority of a received memory access operation to a first storage medium and / or a second storage medium by a scheduling module in the storage device includes:
[0225] The priority of the received memory access operation is determined by the priority arbitrator in the scheduling module, and the memory access operation marked with the response priority is written into the corresponding cache queue.
[0226] The memory access operations are output in descending order of response priority through the cache queue in the scheduling module;
[0227] The process of responding to memory access operations in descending order of priority includes:
[0228] Memory access operations responded to by the cache queue.
[0229] In one exemplary embodiment, the process of determining the priority of a received memory access operation to a first storage medium and / or a second storage medium includes:
[0230] Determine the memory access address and memory access type of the received memory access operation on the first storage medium and / or the second storage medium;
[0231] The priority of memory access operations is determined based on the memory access address and memory access type.
[0232] In one exemplary embodiment, the process of obtaining training data and model weight data from a storage device via a computing device, obtaining local weight data based on the training data and model weight data, and writing the local weight data to the storage device includes:
[0233] The first-in-first-out (FIFO) module retrieves training data and model weight data from the storage device and writes the local weight data stored in the storage module to the storage device.
[0234] The processing module calculates local weight data based on the training data and model weight data.
[0235] Local weight data is stored through a storage module.
[0236] In an exemplary embodiment, the first-in-first-out (FIFO) module includes a data cache queue, a weight cache queue, and a read unit. The process of obtaining training data and model weight data from the storage device through the FIFO module and writing the local weight data stored in the storage module into the storage device includes:
[0237] The first preset size of training data is read from the storage device by the reading unit and written into the data cache queue until the data cache queue meets the first data full condition. The second preset size of model weight data is read from the storage device and written into the weight cache queue until the weight cache queue meets the second data full condition.
[0238] The process by which the processing module calculates local weight data based on training data and model weight data includes:
[0239] The processing module retrieves training data from the data cache queue and model weight data from the weight cache queue, and calculates local weight data based on the training data and model weight data.
[0240] In one exemplary embodiment, the process of writing local weight data stored in the storage module to the storage device includes:
[0241] Determine the historical address of the last write to the storage device, and determine the current address based on the historical address;
[0242] Write the local weight data stored in the storage module to the storage device at the current address.
[0243] In one exemplary embodiment, the host includes a central processing unit (CPU), and the data processing system further includes a high-speed computing link switch, which is connected to the CPU, the high-speed computing link interface of the computing device, and the high-speed computing link interface of the storage device, respectively.
[0244] In one exemplary embodiment, both the storage device and the computing device are high-speed computing link-2 type devices.
[0245] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0246] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data processing system, characterized by The system includes a host, at least one storage device equipped with a high-speed computing link interface, and at least one computing device equipped with the high-speed computing link interface, wherein the storage device, the computing device, and the host are connected via the high-speed computing link interface, and wherein: The host is used to acquire training data and write the training data into the storage device; The computing device is configured to obtain the training data and model weight data from the storage device, obtain local weight data based on the training data and model weight data, and write the local weight data into the storage device. The storage device is used to store the training data and the local weight data, and to obtain and store the model weight data based on all the local weight data; The storage device includes a first storage medium and a second storage medium, wherein the first storage medium is a storage medium in which data stored is not lost after power failure, and the second storage medium is a storage medium in which data stored is lost after power failure; The host is further configured to divide the first storage medium into multiple persistent storage regions, determine any one of the persistent storage regions as a first persistent storage region, configure the first persistent storage region as a ring memory, determine any one of the multiple persistent storage regions except the first persistent storage region as a second persistent storage region, divide the second storage medium into multiple non-persistent storage regions, and the non-persistent storage regions correspond one-to-one with the computing device; The process of writing the training data into the storage device includes: The training data is written to the circular memory until the address pointed to by the write pointer and the address pointed to by the read pointer of the circular memory are the same; The process of writing the local weight data into the storage device includes: Write the local weight data into the non-persistent storage area that corresponds to it one-to-one; The process of obtaining and storing the model weight data based on all the local weight data includes: The model weight data is obtained based on all the local weight data and stored in the second persistent storage area; The storage device also includes: The calculation module is used to read the local weight data from each of the non-persistent storage areas, calculate the average value of each of the local weight data, and obtain the model weight data; The control module is used to write the model weight data into the second persistent storage area; The storage device further includes a scheduling module for determining the response priority of received memory access operations to the first storage medium and / or the second storage medium; The control module is also configured to respond to the memory access operation in descending order of response priority; The computing device has the highest priority in responding to memory access operations on the first storage medium and / or the second storage medium; The scheduling module includes: A priority arbiter is used to determine the response priority of a received memory access operation and write the memory access operation with the tag of the response priority into the corresponding cache queue. A cache queue is used to output the memory access operations in descending order of the response priority; The process of responding to the memory access operation in descending order of response priority includes: The memory access operation is responded to by the output of the cache queue; The process of determining the priority of the received responses to memory access operations on the first storage medium and / or the second storage medium includes: Determine the memory access address and memory access type of the received memory access operation on the first storage medium and / or the second storage medium; The priority of the memory access operation is determined based on the memory access address and the memory access type.
2. The data processing system of claim 1, wherein, The process of obtaining the training data from the storage device includes: The training data is obtained from the circular memory.
3. The data processing system of claim 2, wherein, The process of writing the training data into the storage device includes: The training data is divided into multiple sub-data groups; the difference in the amount of data between any two sub-data groups is less than a preset threshold. The process of writing the training data into the circular memory until the address pointed to by the write pointer and the address pointed to by the read pointer of the circular memory are the same includes: For each of the sub-data groups, the sub-data group is written into the circular memory until the address pointed to by the write pointer and the address pointed to by the read pointer of the circular memory are the same.
4. The data processing system of claim 2, wherein, The process of dividing the first storage medium into multiple persistent storage regions includes: The first storage medium is divided into multiple persistent storage areas according to the size of the preset model.
5. The data processing system of claim 2, wherein, The first storage medium is persistent memory.
6. The data processing system according to claim 2, characterized in that, The process of obtaining model weight data from the storage device includes: Retrieve model weight data from the second persistent storage area.
7. The data processing system of claim 1, wherein, The process of dividing the second storage medium into multiple non-persistent storage regions includes: Determine the number of computing devices connected to the storage device; The second storage medium is divided into the stated number of non-persistent storage regions.
8. The data processing system of claim 1, wherein, The host is also used to configure each of the non-persistent storage areas as a ping-pong cache.
9. The data processing system of claim 1, wherein, The second storage medium is a double-rate synchronous dynamic random access memory.
10. The data processing system of any of claims 1-9, wherein, The computing device includes a first-in-first-out module, a processing module, and a storage module, wherein: The first-in-first-out module is used to obtain the training data and model weight data from the storage device, and write the local weight data stored in the storage module into the storage device. The processing module is used to calculate the local weight data based on the training data and the model weight data; The storage module is used to store the local weight data.
11. The data processing system according to claim 10, characterized in that, The first-in-first-out module includes: Data cache queue and weight cache queue; The reading unit is configured to read training data of a first preset size from the storage device, write the training data into the data cache queue until the data cache queue meets the first data full condition, read model weight data of a second preset size from the storage device, and write the model weight data into the weight cache queue until the weight cache queue meets the second data full condition. The processing module is specifically used to obtain the training data from the data cache queue, obtain the model weight data from the weight cache queue, and calculate the local weight data based on the training data and the model weight data.
12. The data processing system according to claim 10, characterized in that, The process of writing the local weight data stored in the storage module to the storage device includes: Determine the historical address of the last write to the storage device, and determine the current address based on the historical address; Write the local weight data stored in the storage module to the storage device at the current address.
13. The data processing system according to claim 10, characterized in that, The host includes a central processing unit, and the data processing system further includes a high-speed computing link switch, which is connected to the central processing unit, the high-speed computing link interface of the computing device, and the high-speed computing link interface of the storage device.
14. The data processing system according to claim 10, characterized in that, Both the storage device and the computing device are high-speed computing link-2 type devices.
15. A data processing method, characterized by, The data processing system is applied to any one of claims 1-14. The data processing system includes a host, at least one storage device with a high-speed computing link interface, and at least one computing device with the high-speed computing link interface. The storage device, the computing device, and the host are connected through the high-speed computing link interface. The storage device includes a first storage medium and a second storage medium. The first storage medium is a storage medium in which data stored is not lost after power failure, and the second storage medium is a storage medium in which data stored is lost after power failure. The data processing method includes: The training data is obtained through the host and written to the storage device; The computing device obtains the training data and model weight data from the storage device, obtains local weight data based on the training data and model weight data, and writes the local weight data into the storage device. The training data and the local weight data are stored in the storage device, and the model weight data is obtained and stored based on all the local weight data. The host divides the first storage medium into multiple persistent storage areas, designates any one of the persistent storage areas as the first persistent storage area, configures the first persistent storage area as a ring memory, designates any one of the multiple persistent storage areas other than the first persistent storage area as the second persistent storage area, and divides the second storage medium into multiple non-persistent storage areas, each of which corresponds one-to-one with the computing device. The process of writing the training data into the storage device includes: The training data is written to the circular memory until the address pointed to by the write pointer and the address pointed to by the read pointer of the circular memory are the same; The process of writing the local weight data into the storage device includes: Write the local weight data into the non-persistent storage area that corresponds to it one-to-one; The process of obtaining and storing the model weight data based on all the local weight data includes: The model weight data is obtained based on all the local weight data and stored in the second persistent storage area; The data processing method further includes: The local weight data is read from each of the non-persistent storage areas by the computing module in the storage device, and the average value of each of the local weight data is calculated to obtain the model weight data. The model weight data is written to the second persistent storage area via the control module in the storage device; The storage device further includes a scheduling module, which determines the response priority of received memory access operations to the first storage medium and / or the second storage medium. The control module responds to the memory access operations in descending order of response priority; the computing device has the highest response priority to memory access operations on the first storage medium and / or the second storage medium. The scheduling module includes a priority arbiter and a cache queue. The priority arbiter is used to determine the response priority of the received memory access operation and write the memory access operation marked with the response priority into the corresponding cache queue. The cache queue is used to output the memory access operation in descending order of the response priority. The process of responding to the memory access operation in descending order of response priority includes: The memory access operation is responded to by the output of the cache queue; The process of determining the priority of the received responses to memory access operations on the first storage medium and / or the second storage medium includes: Determine the memory access address and memory access type of the received memory access operation on the first storage medium and / or the second storage medium; The priority of the memory access operation is determined based on the memory access address and the memory access type.
16. The data processing method according to claim 15, characterized in that, The process of obtaining the training data from the storage device through the computing device includes: The training data is obtained from the ring memory using the computing device.
17. The data processing method according to claim 16, characterized in that, The process of obtaining model weight data from the storage device through the computing device includes: The model weight data is obtained from the second persistent storage area through the computing device.
18. The data processing method of claim 16, wherein, The process of writing the local weight data into the storage device includes: The local weight data is written to the non-persistent storage area that corresponds to it one-to-one.
19. The data processing method of claim 17, wherein, After the host divides the second storage medium into multiple non-persistent storage areas, the data processing method further includes: The host configures each of the non-persistent storage areas as a ping-pong cache.
20. The data processing method according to any one of claims 15-19, characterized by, The computing device includes a first-in-first-out (FIFO) module, a processing module, and a storage module. The process of obtaining the training data and model weight data from the storage device using the computing device, obtaining local weight data based on the training data and model weight data, and writing the local weight data into the storage device includes: The training data and model weight data are obtained from the storage device through the first-in-first-out module, and the local weight data stored in the storage module is written to the storage device. The processing module calculates the local weight data based on the training data and the model weight data. The local weight data is stored through the storage module.
21. The data processing method according to claim 20, characterized in that, The process of writing the local weight data stored in the storage module to the storage device includes: Determine the historical address of the last write to the storage device, and determine the current address based on the historical address; Write the local weight data stored in the storage module to the storage device at the current address.