Storage device and training data reading method, medium, product and system

Abstract

The application discloses a kind of storage device and training data reading method, medium, product, system, it is related to storage technical field.Therein, method includes dividing training data into multiple data blocks, randomization processing to each data block, obtain first random access order, and each data block is prefetched to cache according to the order.When receiving training data reading request, each data block in cache is split into sub data block satisfying training data reading request, randomization processing to each sub data block, obtain second random access order, and determine the target data corresponding to training data reading request from each sub data block according to second random access order.The application can solve the problem that related technology does not match in the scene of deep learning training, random small file access and high bandwidth demand of storage system, both can randomly read the large amount of small files required in training process, and can meet high bandwidth demand, improve overall storage performance.

Description

Technical Field

[0001] This invention relates to the field of storage technology, and in particular to a storage device and a method, medium, product, and system for reading training data. Background Technology

[0002] As the computing performance of accelerators such as GPUs continues to improve, the datasets used in deep learning training far exceed the memory capacity of storage devices, making it impossible to load them all into memory for data caching. Furthermore, deep learning training datasets consist primarily of numerous small files, requiring the random reading of a large number of scattered small files in each training round. This data access pattern places a significant burden on storage systems. In addition, the growth rate of storage device bandwidth lags significantly behind the growth rate of accelerator computing performance, and the performance gap between the two continues to widen, making it difficult for storage devices to meet the accelerator's data input and output bandwidth requirements. Summary of the Invention

[0003] This invention provides a storage device and a training data reading method, medium, product, and system that combines large block sequential prefetching of the backend disk with random small block hits at the frontend, ensuring both the randomness of the training data and meeting high bandwidth requirements, thereby improving the overall storage I / O (input / output) performance.

[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a method for reading training data from a storage device, comprising: Determine the training data corresponding to the target training task and divide the training data into multiple sub-training data blocks; Randomize each sub-training data block to obtain the first random access order corresponding to each sub-training data block; Before receiving a training data read request generated by the training task, each sub-training data block is prefetched into the cache according to the first random access order; When a training data read request is received, the sub-training data blocks in the cache are split into sub-data blocks to be read to satisfy the training data read request; Each sub-data block to be read is randomized to obtain a second random access order that is different from the first random access order. The target data corresponding to the training data reading request is determined from each sub-data block to be read according to the second random access order.

[0005] Another aspect of the present invention provides a storage device, comprising: Memory, used to store training data and computer programs; A cache is used to cache data read from memory; The communication interface is used to receive training data reading requests and send target data. The controller, which is connected to the memory, coupled with the buffer and the communication interface, is used to implement the steps of the training data reading method of the above-mentioned storage device when executing a computer program.

[0006] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a controller of the storage device, implements the steps of the training data reading method of the storage device described above.

[0007] The present invention also provides a computer program product, including a computer program that, when executed by a controller of a storage device, implements the steps of the above-described method for reading training data from a storage device.

[0008] Finally, this invention provides a training data reading system, including a server and a storage device; Before the target training task is executed, the server sends a notification message to the storage device through the storage device's communication interface to provide the training data required for the target training task. During the execution of the target training task, when a training data read request is generated, the server sends the training data read request to the storage device's controller through the communication interface. The storage device's memory pre-stores the training data corresponding to the server's training task. When the controller receives the training data read request and executes the computer program stored in the memory, it implements the steps of the above-mentioned training data read method of the storage device and sends the target data corresponding to the training data read request to the server through the communication interface.

[0009] The advantage of the technical solution provided by this invention lies in that the storage device predetermines the training data corresponding to the deep learning training task and divides the training data into multiple sub-training data blocks. The storage device randomizes each sub-training data block to obtain a first random access order, which represents the prefetch order of each sub-training data block. Before receiving a training data read request generated by the training task, the storage device prefetches each sub-training data block from the storage medium to the cache according to the first random access order, so that the data transfer process from the storage medium to the cache is completed in advance, avoiding the time delay caused by reading data from the storage medium only after receiving the read request. When the storage device receives a training data read request, it splits each sub-training data block in the cache into sub-data blocks to be read that meet the data amount indicated by the request, so that the data that originally existed in the cache in large blocks is converted into small blocks of data suitable for the granularity of the training task. The storage device randomizes each sub-data block to be read to obtain a second random access order different from the first random access order, and determines the target data corresponding to the training data read request from each sub-data block to be read according to the second random access order, so that the data returned to the training task from the front end has randomness. Therefore, the storage device accesses the storage medium in large blocks sequentially during the data transfer process from the storage medium to the cache, ensuring full utilization of the storage medium's data read bandwidth. During the data transfer process from the cache to the training task, it supplies data in small blocks randomly, ensuring the data randomness required by the training task. This decouples the conflicting high bandwidth and data randomness requirements in deep learning training scenarios to different levels within the storage device for separate processing. It ensures both large-block sequential access on the storage medium side to obtain high bandwidth and small-block random data supply on the training task side to meet the training generalization capability requirements, thereby improving the IO performance of the storage device in deep learning training scenarios.

[0010] Furthermore, this invention also provides corresponding storage devices, computer-readable storage media, computer program products, and training data reading systems for the training data reading method of storage devices, further making the method more practical, and the storage devices, computer-readable storage media, computer program products, and training data reading systems have corresponding advantages. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the present invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1A schematic diagram of the hardware composition framework applicable to the storage system provided by the present invention; Figure 2 A flowchart illustrating a method for reading training data from a storage device provided by the present invention; Figure 3 This invention provides a schematic diagram of training data segmentation in an exemplary application scenario. Figure 4 A schematic diagram of a random sequence array in an exemplary application scenario provided by the present invention; Figure 5 A schematic diagram of a consistent hash ring in an exemplary application scenario provided by the present invention; Figure 6 A structural framework diagram of an exemplary embodiment of the training data reading device for the storage device provided by the present invention; Figure 7 A structural diagram of an exemplary embodiment of the storage device provided by the present invention; Figure 8 A structural diagram of an exemplary embodiment of the training data reading system provided by the present invention; Figure 9 This is a schematic diagram of the framework of the training data reading method for the storage device provided by the present invention in an exemplary application scenario. Detailed Implementation

[0013] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. In this specification and the aforementioned drawings, the terms "first," "second," "third," "fourth," etc., are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. The term "exemplary" means "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior to or better than other embodiments.

[0014] The training process of deep learning models relies on repeatedly reading data samples from the training dataset to update the model parameters. The training dataset typically consists of a massive number of small files; for example, in image recognition tasks, the training dataset may contain millions of image files, each ranging in size from tens of KB to several MB. During training, each iteration requires randomly selecting a batch of samples from the training dataset to input into the graphics processor for computation. To ensure the model has good generalization ability, the order in which the training dataset is read needs to remain random in each training round to avoid overfitting due to a fixed data order.

[0015] However, with the continuous improvement of the computing performance of accelerators such as graphics processing units (GPUs), data storage has become a performance bottleneck in the deep learning training and inference process. First, the datasets used for deep learning training typically reach terabytes (TB) or even petabytes (PB), far exceeding the memory capacity of storage devices, making it impossible to load all the data into memory for caching. Second, deep learning training datasets are primarily composed of numerous small files. Each training round requires randomly reading a large number of scattered small files. This data access pattern leads to storage devices frequently processing a large number of small input / output requests, significantly increasing the number of input / output requests and consequently extending the response time of the storage devices. Third, the growth rate of storage device bandwidth significantly lags behind the growth rate of GPU computing performance, and the performance gap between the two continues to widen, making it difficult for storage devices to meet the data I / O bandwidth requirements of GPUs.

[0016] The storage controller manages the hard drives and the data stored on them, and also provides data access services to the server. For example... Figure 1 As shown, during the process of providing data access to the server, to ensure data security, data input / output requests are processed sequentially through multiple business modules such as the cache module, other I / O stack modules (e.g., redundant arrays of independent disks), and thin stack, ultimately reading data from or writing data to the hard drive. The front end of the storage controller connects to the server via external cards such as Fibre Channel or Remote Direct Memory Access, while the back end connects to the hard drive via external cards such as Serial Minicomputer Interface or Non-Volatile Memory Fast Channel. When a deep learning application on the server generates a read / output request, the request is first sent to the storage device controller through the front end interface. After recognizing the read request, the controller searches the cache based on the volume to which the requested data belongs and its address within the volume. If the data is found in the cache, it is directly returned to the server, completing the processing of one read / output request; this is called a cache hit. If the data is not stored in the cache, the cache module passes the read I / O request to the lower-level module until the data is read from the backend storage disk. After being processed layer by layer by the I / O stack, the data is returned to the server, completing the processing of one read I / O request. Therefore, storage caching plays a crucial role in storage I / O performance. Read operations in cache hit scenarios can significantly improve I / O performance by shortening the I / O path.

[0017] Deep learning training scenarios employ random small-file data usage to ensure the model's generalization ability. As long as the training data provided to the deep learning training process is random, the model's generalization ability can be guaranteed. In other words, the data reading requirements for deep learning training tasks include two aspects: firstly, data needs to be supplied randomly in small blocks; secondly, the overall bandwidth of the data supply needs to be high enough to match the computing speed of the graphics processor. However, random small-block data access and high bandwidth are two mutually restrictive performance indicators. Random small-block data access means that the storage device needs to frequently address different physical locations on the storage medium, reading only a small amount of data after each address, resulting in underutilization of the storage medium's data transfer bandwidth; high bandwidth, on the other hand, requires the storage medium to transmit data in continuous large blocks to reduce the number of addressing attempts and the proportion of addressing time. These two requirements are difficult to meet simultaneously on the same storage device.

[0018] When handling data read requests in deep learning training scenarios, storage devices using related technologies face challenges. Deep learning training requires random access to numerous small files to ensure model generalization ability. However, this random access leads to a surge in the number of input / output requests for storage devices, resulting in underutilization of storage bandwidth. Simultaneously, the bandwidth requirements for data reads in deep learning training are constantly increasing, while the bandwidth growth rate of storage devices lags behind the growth rate of graphics processing unit (GPU) performance, leading to a widening performance gap. These issues cause storage devices to struggle to meet the input / output performance demands of deep learning training tasks.

[0019] To match random small file access with high bandwidth requirements, this invention offloads all data selection and scheduling functions from the server to the storage device. The storage device, leveraging these functions, converts data access below its own cache into large-block data access, and converts host-side data supply into small-block random hits. This ensures data randomness while improving storage bandwidth utilization and overall storage performance. Having described the technical solution of this invention, the various non-limiting embodiments are described in detail below with reference to the accompanying drawings and specific implementation details.

[0020] Please see first. Figure 2 According to the present invention, a method for reading training data from a storage device can be implemented as a computer program product, installed and run in the controller of the storage device, for use in training data reading and processing in deep learning training scenarios. In some embodiments of the method, the method includes the following steps: S201: Determine the training data corresponding to the target training task and divide the training data into multiple sub-training data blocks.

[0021] In this context, the target training task refers to the training task that the host device, such as a server, is currently processing or about to process. The training task is a deep learning training process running on the host device, which needs to read training data from storage devices to update model parameters. Training data refers to the set of data samples required for the training task, which is stored in storage devices according to a specific address order. Sub-training data blocks refer to multiple data blocks obtained by dividing the training data according to storage address ranges; each sub-training data block is a part of the training data.

[0022] In this step, before the deep learning training begins, the host device can send the training data required for this training task via a message. The storage device obtains the training dataset used for this training through this message. The storage device determines the start and end addresses of the training dataset in the storage space, thereby determining the address range of the training data in the storage address space, and divides the training data into multiple sub-training data blocks according to the address range.

[0023] S202: Randomize each sub-training data block to obtain the first random access order corresponding to each sub-training data block.

[0024] The first random access order refers to the order in which the sub-training data blocks are accessed. This order is not based on the address order of the sub-training data blocks in the storage address space, but is rearranged after randomization. The sub-training data blocks originally had consecutive sequential numbers according to their storage address order. After randomization, the access order of the sub-training data blocks becomes random, and this random order is the first random access order. The randomization method is not limited, and those skilled in the art can flexibly choose one according to the actual situation.

[0025] S203: Before receiving a training data read request generated by the training task, prefetch each sub-training data block into the cache according to the first random access order.

[0026] In this context, caching refers to a high-speed storage area within a storage device used to temporarily store data read from the storage medium, with access speeds exceeding those of the storage medium itself. Prefetching refers to the operation of reading sub-blocks of training data from the storage medium into the cache before receiving a request to read training data. This operation ensures that the data exists in the cache before the request arrives, thereby shortening the request response time.

[0027] Before receiving a training data read request from the training task, the storage device sequentially reads each sub-training data block from the storage medium and writes it to the cache according to the order indicated by the first random access order. Since the prefetch operation is performed before receiving the training data read request, when a subsequent training data read request arrives, the required data already exists in the cache, eliminating the need to wait for reading from the storage medium. The sequential access to the storage medium in large blocks during the process from the storage medium to the cache ensures full utilization of the storage medium's data read bandwidth.

[0028] S204: When a training data read request is received, the sub-training data blocks in the cache are split into sub-data blocks to be read to satisfy the training data read request.

[0029] Among them, the sub-data block to be read refers to multiple data blocks obtained by splitting the sub-training data block in the cache according to the amount of data indicated by the training data read request. The amount of data in each sub-data block to be read is sufficient to meet the amount of data required for a single read of the training task.

[0030] After the data is prefetched into the cache in the previous step, upon receiving a training data read request, the cached sub-training data blocks are split into sub-data blocks to be read to satisfy the training data read request. The training data read request contains information about the amount of data required for a single training session. Based on this data amount information, the storage device splits the cached sub-training data blocks into multiple sub-data blocks to be read, ensuring that the amount of data in each sub-data block meets the data requirements for a single training session.

[0031] S205: Randomize each sub-data block to be read to obtain a second random access order that is different from the first random access order, and determine the target data corresponding to the training data reading request from each sub-data block to be read according to the second random access order.

[0032] The second random access order refers to the order in which the sub-data blocks to be read are accessed. This order is not based on the address order of the sub-data blocks in the storage address space, but is rearranged after randomization. The target data refers to the data determined from the sub-data blocks to be read that needs to be sent to the host device in response to the training data read request.

[0033] In this step, based on the sequential access of the storage medium in large blocks during the transfer from the storage medium to the cache, the storage device performs randomization processing on each sub-block of data to be read, generating a second random access order. This second random access order differs from the first random access order; that is, the sub-blocks of data to be read are arranged in a new random order during the transfer from the cache to the host device. The storage device selects data from each sub-block of data to be read according to the second random access order as the target data corresponding to the training data read request. This ensures that data is supplied randomly in small blocks during the transfer from the cache to the training task, guaranteeing the randomness of the data required for the training task.

[0034] In the technical solution provided in this embodiment, the storage device predetermines the training data corresponding to the deep learning training task and divides the training data into multiple sub-training data blocks. The storage device randomizes each sub-training data block to obtain a first random access order, which represents the prefetching order of each sub-training data block. Before receiving a training data read request generated by the training task, the storage device prefetches each sub-training data block from the storage medium to the cache according to the first random access order, so that the data transfer process from the storage medium to the cache is completed in advance, avoiding the time delay caused by reading data from the storage medium only after receiving the read request. When the storage device receives a training data read request, it splits each sub-training data block in the cache into sub-data blocks to be read that meet the data amount indicated by the request, so that the data that originally existed in the cache in large blocks is converted into small blocks of data suitable for the granularity of the training task. The storage device randomizes each sub-data block to be read to obtain a second random access order different from the first random access order, and determines the target data corresponding to the training data read request from each sub-data block to be read according to the second random access order, so that the data returned to the training task from the front end has randomness. Therefore, the storage device accesses the storage medium in large blocks sequentially during the data transfer process from the storage medium to the cache, ensuring full utilization of the storage medium's data read bandwidth. During the data transfer process from the cache to the training task, it supplies data in small blocks randomly, ensuring the data randomness required by the training task. This decouples the conflicting high bandwidth and data randomness requirements in deep learning training scenarios to different levels within the storage device for separate processing. It ensures both large-block sequential access on the storage medium side to obtain high bandwidth and small-block random data supply on the training task side to meet the training generalization capability requirements, thereby improving the IO performance of the storage device in deep learning training scenarios.

[0035] Based on the above embodiments, this embodiment further defines the implementation method of randomization processing for each sub-training data block, which may include the following: Obtain the entropy source data, perform a hash operation on the entropy source data to obtain the first random seed; input the first random seed into the pseudo-random number generator to generate the first random access order.

[0036] Entropy source data refers to the initial data used to generate randomness. In cryptographically secure random number generators, entropy source data typically originates from unpredictable factors in the system environment, such as hardware noise, system clock jitter, and device interruption times. The operating system kernel collects this environmental noise and uses it as the entropy source for the random number generator. Hash operation refers to the operation of converting input data of arbitrary length into a fixed-length output value using a hash function. Hash functions are one-way and collision-resistant, meaning that the input value cannot be deduced from the output value, and the probability of different input values ​​producing the same output value is extremely low. A pseudo-random number generator is a program that generates a sequence of random numbers using a deterministic algorithm. Starting from a random seed, the program generates a series of numbers that statistically approximate true random numbers through mathematical operations. In this embodiment, the pseudo-random number generator may, for example, employ the Mason tween algorithm.

[0037] In one implementation, the storage device reads a cryptographically secure pseudo-random number generator interface provided by the operating system kernel, such as reading a specified device file, to obtain random numbers as entropy source data. This entropy source data is generated based on unpredictable factors such as hardware noise in the system environment, exhibiting good randomness. For example, the storage device inputs the obtained entropy source data into a secure hash algorithm to perform cryptographic hash calculations, converting the entropy source data into a fixed-length hash value as the first random seed. Through hash operations, the randomness of the entropy source data is further enhanced, making it difficult for attackers to deduce the entropy source data from the random seed, thereby improving the security of the random seed. For example, the storage device inputs the first random seed as the initial state into a Mason tween algorithm pseudo-random number generator, which outputs multiple random numbers based on the first random seed through iterative calculations. The storage device arranges each sub-training data block in ascending order of the output random numbers; the resulting order is the first random access order. Because the output sequence of this pseudo-random number generator has good uniformity and independence, the first random access order statistically exhibits a random sequence.

[0038] As shown above, the storage device uses unpredictable factors such as hardware noise in the system environment as the entropy source data, ensuring the initial randomness of the random seed. Hash operations convert the entropy source data into fixed-length hash values, enhancing the security of the random seed and making it difficult for attackers to deduce the entropy source data from the random seed. The pseudo-random number generator outputs a random number sequence based on the random seed, exhibiting good uniformity and independence, making the first random access order statistically approximate a truly random sequence, thus ensuring the randomness of the access order of each sub-training data block during the prefetching phase.

[0039] Based on the above embodiments, this embodiment further defines the randomization processing implementation method for each sub-data block to be read, which may include the following: Obtain a second random seed; the second random seed is different from the first random seed; input the second random seed into the pseudo-random number generator to generate a second random access order.

[0040] The second random seed refers to the random seed used to generate the second random access order. This seed is different from the first random seed to ensure that there is no correlation between the first and second random access orders. The pseudo-random number generator can be the same generator used in the above embodiments. That is, this embodiment uses the same pseudo-random number generation algorithm, but uses a different random seed to generate a different random sequence that is statistically independent of the first random access order.

[0041] In one implementation, the storage device obtains the second random seed in a different way than obtaining the first random seed. For example, the storage device can obtain a new random number as the second random seed by reading the random number generator interface of the operating system kernel, or by generating the second random seed through a combination of system parameters such as the system clock and process identifier. The second random seed is different from the first random seed, that is, the values ​​of the two seeds are not equal. For example, the storage device uses the second random seed as the initial state input to the same Mason tween algorithm pseudo-random number generator as in the above embodiment, which outputs multiple random numbers based on the second random seed. The storage device rearranges each sub-data block to be read according to the output random numbers, and the rearranged order is the second random access order. Since the second random seed is different from the first random seed, and the output sequence of the pseudo-random number generator is statistically one-to-one with the input seed and has good independence, the second random access order and the first random access order are statistically independent random sequences.

[0042] As can be seen from the above, this embodiment uses a second random seed, which is different from the first random seed, to input the same pseudo-random number generator. This makes the order in which each sub-data block to be read is output from the cache to the training task independent of the access order of each sub-training data block in the prefetching stage, further enhancing the randomness of the order of data received by the training task.

[0043] Based on the above embodiments, this embodiment further defines the method for dividing the training data, which may include the following: Upon receiving a training task start notification, the start and end addresses of the training data required for the target training task in the storage address space are determined based on the training task start notification. Based on the address range between the start and end addresses, the training data is divided into multiple sub-training data blocks according to the address mapping granularity of the storage medium. The address mapping granularity is determined based on the redundant array type and stripe capacity of the storage medium.

[0044] The training task start notification refers to the message sent by the server to the storage device when starting a deep learning training task. This notification informs the storage device of the identification information of the training dataset used in this training. The start address and end address refer to the starting and ending positions of the address range occupied by the training dataset in the storage address space, respectively. The address range between these two positions constitutes the complete storage area of ​​the training dataset. Address mapping granularity refers to the smallest address mapping unit used by the storage device when mapping logical addresses to physical storage media. It is the smallest logical data unit used by the storage device when performing data read and write operations. This granularity determines the smallest data access unit when the storage device accesses the storage media, such as the minimum amount of data processed at one time when a disk or RAID (Redundant Array of Independent Disks) card reads and writes data at the underlying level. Address mapping granularity can be determined based on the physical characteristics of the storage medium, the type of redundant array, and the stripe capacity of the redundant array. Examples include sectors (512B) of a hard disk drive, pages (4KB) of an SSD, stripe size of a RAID array (e.g., 1MB), or the size of a data block (chunk) in distributed storage. Redundant arrays refer to the technology of combining multiple physical storage media into a single logical unit, improving data reliability and access performance through redundant data distribution. Stripe capacity refers to the amount of data contained in a single stripe in a redundant array; this parameter determines how data is distributed across multiple physical storage media.

[0045] In this embodiment, before initiating the deep learning training task, the server sends a training task initiation notification to the storage device. This notification includes the identification information of the training dataset used in this training. Upon receiving the notification, the main controller in the storage device queries the metadata information of the training dataset based on the identification information. This metadata information records the start and end addresses of the training dataset in the storage address space. The main controller determines the address range of the training data in the storage address space based on the query results. In one implementation, the storage device obtains the redundant array configuration parameters of the current storage medium, including the redundant array type and stripe capacity. Based on the redundant array type and stripe capacity, the storage device determines the address mapping granularity. For example, if the redundant array type is a standalone disk redundant array and the stripe capacity is a specified value, then the address mapping granularity is that stripe capacity value. The storage device divides the training dataset into multiple sub-training data blocks starting from the start address and using the stripe capacity as the unit, according to this address mapping granularity. Each sub-training data block occupies a contiguous address space in the storage medium and has the same amount of data.

[0046] As can be seen from the above, this embodiment can accurately determine the start and end addresses of the training dataset in the storage address space by querying metadata based on the identification information. The storage device divides the training data into multiple sub-training data blocks according to the address mapping granularity of the storage medium. Since the address mapping granularity is determined based on the redundant array type and stripe capacity, the boundary of each sub-training data block is aligned with the physical data layout of the storage medium. This ensures that when reading each sub-training data block from the storage medium, each input / output operation of the storage medium can completely read one or more stripes, avoiding the additional addressing overhead caused by cross-strip reading.

[0047] Based on the above embodiments, when the optimal access granularity (address mapping granularity) of the backend storage is inconsistent with the data consumption granularity (batch size) of the frontend training task, this embodiment further defines how to select an optimal block size that takes both into account, which may include the following: When the address mapping granularity is inconsistent with the training data volume, obtain the amount of training data required for a single batch in the target training task; compare the address mapping granularity of the storage medium with the training data volume, and determine the data volume of the sub-training data block according to the greatest common divisor of the address mapping granularity and the training data volume, so that the data volume of the sub-training data block is equal to the greatest common divisor.

[0048] The data volume required for a single batch refers to the total amount of data samples used in one iteration of a deep learning training task, i.e., the amount of data required for a single read request in the training task. The greatest common divisor (GCD) is the largest value that can simultaneously divide both the address mapping granularity and the data volume required for a single batch. The sub-training data block size refers to the amount of data contained in each sub-training data block when dividing the training data.

[0049] In one implementation, the training task initiation notification also includes information on the amount of training data required for a single batch in the deep learning training task. The main controller of the storage device extracts this information from the training task initiation notification to obtain the amount of training data required for a single batch. For example, the storage device obtains the address mapping granularity value and the amount of training data required for a single batch. The storage device calculates the greatest common divisor (GCD) of the two values ​​and uses the GCD as the data size of the sub-training data blocks used when partitioning the training data. For instance, if the address mapping granularity is a first specified value and the amount of training data required for a single batch is a second specified value, then the GCD is a third specified value, and the storage device partitions the training data using the third specified value as the data size for each sub-training data block.

[0050] As can be seen from the above, the final sub-training data block data size determined in this embodiment is divided by both the address mapping granularity and the data size required for a single batch. This ensures that the storage device can align with the physical layout of the storage medium when accessing each sub-training data block, and that the data required for each batch contains exactly an integer number of sub-training data blocks, thus avoiding data fragmentation and read amplification problems during the access process.

[0051] Based on the above embodiments, this embodiment further defines the implementation method of prefetching each sub-training data block into the cache, which may include the following: The system detects the data transfer rate between the storage device and the host device that issues the target training task; based on the data transfer rate, it controls the rate at which each sub-training data block is prefetched from the storage medium.

[0052] Data transfer rate refers to the amount of data transferred between the storage device and the host device per unit time. This rate reflects the speed at which the storage device supplies data to the host device. Prefetch rate refers to the speed at which the storage device reads data from the storage medium to the cache, that is, the amount of data that the storage device reads from the storage medium and writes to the cache per unit time.

[0053] In one implementation, the storage device's communication interface continuously monitors the transmission rate on the data transmission link, representing the speed at which the storage device sends data to the host device via the communication interface. The storage device's controller reads the measured value of this transmission rate from the communication interface. In an exemplary implementation where the storage device controls the prefetching rate of each sub-training data block, the storage device's controller determines the control method for the prefetch rate based on the detected transmission rate. When the transmission rate is high, the controller increases the speed at which sub-training data blocks are read from the storage medium and written to the cache; when the transmission rate is low, the controller decreases the prefetch rate.

[0054] As can be seen from the above, by associating the prefetch rate with the data transfer rate, the storage device can ensure the supply of front-end data while avoiding the prefetch operation from consuming too much bandwidth resources and affecting the access of other applications.

[0055] Based on the above embodiments, this embodiment further defines how to control the rate of prefetching each sub-training data block from the storage medium, which may include the following: Obtain the preset rate adjustment parameters; determine the data prefetch rate based on the preset rate adjustment parameters and the data transmission rate, so that the ratio of the data prefetch rate to the data transmission rate is not higher than the sum of 1.2 and the preset floating value.

[0056] The preset rate adjustment parameter is a pre-defined proportional coefficient used to calculate the data prefetch rate based on the data transmission rate. The preset float value is an offset used to adjust the upper limit of the rate adjustment parameter, allowing the rate adjustment parameter to fluctuate around the reference value.

[0057] In one implementation, a preset rate adjustment parameter is stored in the storage device's configuration file, which the system administrator can pre-configure based on the storage device's bandwidth resources and application scenarios. The storage device's controller reads this parameter from the configuration file during initialization. The controller uses the product of the data transfer rate and the preset rate adjustment parameter as the data prefetch rate, ensuring that the ratio of the data prefetch rate to the data transfer rate is no higher than the sum of 1.2 and a preset floating value.

[0058] As can be seen from the above, this embodiment controls the upper limit of the prefetch rate by using preset rate adjustment parameters and preset floating values, so that the prefetch rate is controlled within an acceptable range, which not only meets the bandwidth requirements of the prefetch operation, but also leaves bandwidth resources for other applications.

[0059] Based on the above embodiments, this embodiment further defines the implementation process of prefetching each sub-training data block into the cache, which may include the following: Continuously monitor the data transfer rate of the storage device sending data to the host device through the communication interface; when the data transfer rate increases compared to the previous moment, increase the rate of prefetching sub-training data blocks from the storage medium; when the data transfer rate decreases compared to the previous moment, decrease the rate of prefetching sub-training data blocks from the storage medium.

[0060] Continuous monitoring refers to the process of repeatedly measuring the data transmission rate at fixed time intervals. Through continuous monitoring, the trend of the transmission rate changing over time can be obtained. The communication interface refers to the hardware interface in the storage device used for sending and receiving data with the host device, such as a Fibre Channel interface or a remote direct memory access interface.

[0061] In one implementation, the communication interface of the storage device periodically measures the rate at which it sends data to the host device at preset time intervals and reports the measurement results to the controller in real time. The controller records each measurement result, forming time-series data of the data transmission rate. When the data transmission rate increases relative to the previous moment, the storage device increases the rate at which it prefetches sub-training data blocks from the storage medium. When the data transmission rate decreases relative to the previous moment, the storage device decreases the rate at which it prefetches sub-training data blocks from the storage medium. In one implementation, the controller of the storage device compares the transmission rate monitored at the current moment with the transmission rate monitored at the previous moment. If the transmission rate at the current moment is greater than the transmission rate at the previous moment, the controller increases the prefetch rate, that is, increases the speed at which it reads sub-training data blocks from the storage medium and writes them to the cache. If the transmission rate at the current moment is less than the transmission rate at the previous moment, the controller decreases the prefetch rate, that is, decreases the speed at which it reads sub-training data blocks from the storage medium and writes them to the cache. In one implementation, the adjustment magnitude of the prefetch rate is proportional to the change magnitude of the transmission rate, that is, the greater the change in the transmission rate, the greater the adjustment magnitude of the prefetch rate.

[0062] As can be seen from the above, the prefetch rate of this embodiment can be dynamically adjusted according to the change of data transmission rate, so that the prefetch operation is always matched with the front-end data reading rate, avoiding cache overflow or bandwidth waste caused by excessively high prefetch rate, and also avoiding insufficient cached data and front-end waiting caused by excessively low prefetch rate.

[0063] Based on the above embodiments, this embodiment further defines the implementation method of splitting each sub-training data block into sub-data blocks to be read, which may include the following: The training data read request also includes information on the amount of training data in a single session. Each sub-training data block in the cache is divided into multiple sub-data blocks to be read according to the amount of training data in a single session.

[0064] The single training data volume refers to the amount of data requested in a single read request during a deep learning training task. This data volume is determined by the batch size of the training task and represents the total number of data samples required for one iteration of the training task. The data volume of each sub-data block to be read is equal to the single training data volume.

[0065] In this embodiment, the storage device extracts single-time training data volume information from the received training data read request. This information indicates the amount of data required for this training task to read. The storage device splits each sub-training data block in the cache into multiple sub-data blocks to be read according to the single-time training data volume, and the data volume of each sub-data block to be read is equal to the single-time training data volume. In one implementation, each sub-training data block in the cache has a first data volume, and the single-time training data volume is less than the first data volume. The storage device sequentially divides each sub-training data block into multiple consecutive intervals according to the single-time training data volume, and each interval constitutes a sub-data block to be read. Each sub-data block to be read occupies a consecutive address interval in the original sub-training data block, and the sub-data blocks to be read cover the entire sub-training data block without overlap or omission. The data volume of each sub-data block to be read is equal to the single-time training data volume. The data volume of the last sub-data block to be read may be less than or equal to the single-time training data volume, depending on whether the data volume of the sub-training data block is divisible by the single-time training data volume.

[0066] As can be seen from the above, the amount of data in each sub-data block to be read in this embodiment is equal to the amount of training data in a single session. Each read request of the training task can directly obtain the sub-data block to be read from the cache that is exactly equal to the amount of data requested. There is no need to perform additional data pruning or padding operations before sending the data to the training task, which shortens the output path of data from the cache to the host.

[0067] Based on the above embodiments, after determining the target data corresponding to the training data reading request from each sub-data block to be read, this embodiment sends the target data to the host device; after the target data is successfully sent, the sub-data blocks to be read that make up the target data are deleted from the cache.

[0068] In this context, the host device refers to the server that runs deep learning training tasks. This device is connected to the storage device via a communication network, sending data read requests to the storage device and receiving the target data returned by the storage device. Cache deletion refers to the operation of removing data from the cache, which releases the cache storage space occupied by the deleted data.

[0069] In one implementation, the storage device controller organizes the target data determined from each sub-data block to be read into a continuous data stream and sends this data stream to the host device that issues the training task via a communication interface. Upon receiving the target data, the host device uses it for a single iteration of training the deep learning model. After the target data is successfully sent, the storage device removes the sub-data blocks that make up the target data from its cache. In another implementation, the storage device controller monitors the completion status of the communication interface. Upon confirming that the target data has been successfully sent to the host device, the controller determines the storage location of each sub-data block that makes up the target data in the cache, marks these sub-data blocks as invalid, and releases the cache storage space they occupy. The released cache storage space can then be used to store other sub-training data blocks prefetched from the storage medium subsequently.

[0070] As can be seen from the above, this embodiment immediately releases the cached data that has been accessed, avoiding the data from remaining in the cache and occupying cache space, thereby improving the utilization efficiency and hit rate of the cache.

[0071] Based on the above embodiments, this embodiment further defines the prefetching implementation method for each sub-training data block, which may include the following: The storage device includes multiple controllers, which together provide data access services for the training data. According to the correspondence between each sub-training data block and each controller, each controller prefetches its corresponding sub-training data block from the storage medium to the cache according to the first random access order.

[0072] In this context, a controller refers to a hardware unit within the storage device used to perform data input / output control and data processing. Multiple controllers can form a cluster to collectively provide data access services for the training data. The correspondence refers to the mapping relationship between each sub-training data block and each controller, which determines which controller is responsible for prefetching and provisioning each sub-training data block. In one implementation, the storage device assigns a corresponding controller to each sub-training data block, based on a consistent hashing algorithm. Each controller is only responsible for prefetching its corresponding sub-training data block. Each controller reads its corresponding sub-training data block from the storage medium and writes it to the cache within the prefetch time window, following the order indicated by the first random access order.

[0073] As shown above, the storage device prefetches its corresponding sub-training data blocks according to the correspondence between each sub-training data block and each controller. Multiple controllers perform the prefetching operation in parallel, and each controller only needs to process its own corresponding data block. The prefetching task is distributed among multiple controllers according to the data block correspondence, avoiding the performance bottleneck caused by a single controller undertaking all prefetching tasks and improving the throughput of the prefetching stage.

[0074] Based on the above embodiments, before determining the correspondence between each sub-training data block and each controller, for a scenario where the storage device includes a master controller and at least one slave controller, the master controller constructs a data block query structure based on the number of sub-training data blocks, and each element in the data block query structure corresponds one-to-one with each sub-training data block; the sub-training data block index of each sub-training data block is stored sequentially into the data block query structure according to its position number in the first random access order, and the data block query structure is synchronized to each slave controller.

[0075] In this system, the master controller refers to the controller node that performs management and coordination functions among multiple controllers, while the slave controller refers to the controller node that performs data read and write operations and is coordinated by the master controller. Data synchronization between the master and slave controllers can be achieved through methods such as high-speed interconnect buses or shared memory. The data block query structure is a data organization format used to store and query the correspondence between sub-training data blocks and their random access order. This structure allows each controller to quickly determine the sub-training data blocks to be prefetched based on the random access order. The data volume of this query structure is small (containing only index values, not actual data), therefore the synchronization overhead is acceptable. The position number refers to the position of the sub-training data block in the first random access order, used to identify the order in which the data block is accessed. Synchronization refers to the process of copying data from the master controller and sending it to each slave controller. The synchronization operation ensures that each slave controller obtains a data copy consistent with that of the master controller.

[0076] In one implementation, the main controller obtains the total number of sub-training data blocks and constructs a data block query structure containing the same number of elements as the total number. Each element in the data block query structure corresponds to a sub-training data block. The data block query structure can be implemented as an array, with the array length equal to the total number of sub-training data blocks, and each index position in the array corresponding to an identifier of a sub-training data block. For example, such as... Figure 3 As shown, the 1GB training data is divided into 1MB blocks, resulting in 1024 blocks. Each block's ID is a consecutive sequence of 1024 numbers from 0 to 1023. After randomization, each block's ID becomes a series of 1024 random numbers. This random number sequence can be stored in an array the same size as the number of data blocks, as shown below. Figure 4As shown, each index in the array corresponds to the ID of a sub-training data block. For example, the master controller traverses each position number starting from the first position according to a first random access order, storing the index value of the sub-training data block to be accessed at that position number at the corresponding array index. After the traversal, each position number in the data block query structure records the index of the sub-training data block corresponding to that position number. The master controller sends the complete data block query structure to each slave controller, which stores it in its local memory upon receiving it for subsequent prefetch operations.

[0077] As shown above, in this embodiment, the data block query structure constructed by the master controller records the correspondence between the index of each sub-training data block and the position number in the first random access order, and synchronizes this data structure to each slave controller, so that each slave controller can query the complete first random access order locally. Each slave controller does not need to request order information from the master controller during the prefetching process, reducing the communication overhead between controllers and improving the parallel efficiency of the prefetching operation.

[0078] Based on the above embodiments, this embodiment further defines the correspondence between each sub-training data block and the controller, which may include the following: Each sub-training data block corresponds to a sub-training data block index. The sub-training data block index is used to identify the position of the corresponding sub-training data block in the address order of the training data and to obtain the number of controllers contained in the storage device. The sub-training data block index value of each sub-training data block is divided by the number of controllers, and the controller corresponding to the corresponding sub-training data block is determined based on the remainder.

[0079] The sub-training data block index is a unique identifier assigned to each sub-training data block, used to identify the position of the corresponding sub-training data block in the address order of the training data. The number of controllers refers to the total number of controllers contained in the storage device. The division operation refers to the mathematical operation of dividing the sub-training data block index value by the number of controllers and then rounding down.

[0080] In one implementation, the main controller of the storage device counts the total number of controllers in the cluster during system initialization and stores this number in the system configuration information. When it is necessary to determine the controller corresponding to each sub-training data block, the main controller reads this number from the system configuration information. For each sub-training data block, the storage device obtains its sub-training data block index value, divides this index value by the total number of controllers, and calculates the remainder. This remainder serves as the identifier of the target controller. The storage device assigns the sub-training data block to the controller whose identifier is equal to the remainder.

[0081] As can be seen from the above, by using the above method, each sub-training data block is evenly distributed among multiple controllers, and the load of each controller remains balanced.

[0082] Based on the above embodiments, this embodiment further specifies how to determine the correspondence between each sub-training data block and the controller after the controller of the storage device is adjusted, which may include the following: When the number of controllers changes, obtain the updated number of controllers; then, based on the updated number of controllers, redetermine the controllers corresponding to each sub-training data block.

[0083] The change in the number of controllers refers to a change in the total number of controllers during the operation of the storage device due to reasons such as controller node failure or the addition of new controller nodes. The updated number of controllers refers to the total number of controllers after the change.

[0084] In one implementation, the storage device monitors state changes of controller nodes in the cluster. When a new controller node is detected joining the cluster or an existing controller node is removed from the cluster, the storage device's master controller recalculates the total number of currently available controllers to obtain an updated controller count. The storage device then redetermines the controllers corresponding to each sub-training data block based on the updated controller count. In one implementation, the master controller uses the updated controller count for consistent hash calculation. For each sub-training data block, the master controller divides the index value of the sub-training data block by the updated controller count and calculates the remainder. The master controller then reassigns the sub-training data block to a controller whose identifier is equal to the new remainder.

[0085] For example, such as Figure 5 As shown, the storage device includes four controllers, and the hash ring has four values, which are the node IDs, denoted as `node_id`, with values ​​ranging from 0 to 3. Adding a controller to the storage device adds a value to the hash ring, changing the node ID to 0-4. Removing a controller from the storage device removes a value from the hash ring, changing the node ID to 0-2. Each controller uses a randomized sequence array index... Figure 4 The index value is determined, and the index of the random sequence array is mapped to the controller through the relation node_id==index%node_total, where node_total is the total number of controllers.

[0086] As can be seen from the above, when the number of controllers changes, it is only necessary to recalculate the mapping relationship between each sub-training data block and the controller. The affected data block range is small, and there is no need to regenerate the random access order or re-prefetch data blocks, which makes the cluster have good scalability and fault tolerance.

[0087] Based on the above embodiments, this embodiment further defines an exemplary method for determining the training data corresponding to the target training task, which may include the following: The storage device includes a main controller and at least one slave controller. The main controller obtains the identification information of the target training task and determines the address range of the training data in the storage address space of the storage device based on the identification information.

[0088] In one implementation, when the server initiates a deep learning training task, it sends a training task initiation notification to the main controller of the storage device. This notification contains identification information for the target training task, which uniquely identifies the training dataset used for this training. Upon receiving the notification, the main controller extracts the identification information from it. Based on the identification information, the main controller queries the metadata storage to obtain the start and end addresses of the training dataset in the storage address space, thereby determining the address range of the training data in the storage device.

[0089] As can be seen from the above, in this embodiment, the main controller serves as the unified entry point for the storage device to receive training task information. After obtaining the identification information of the target training task, it queries the metadata based on the identification information to determine the address range of the training data in the storage address space. The main controller determines the data address range during the training task startup phase, enabling subsequent sub-training data block partitioning and randomization processing to be performed based on accurate address boundaries, thus avoiding data block partitioning errors caused by uncertain address ranges.

[0090] It should be noted that there is no strict order of execution between the steps in this invention. As long as they conform to the logical order, these steps can be executed simultaneously or in a certain preset order. Figure 2 This is just an illustrative example and does not mean that this is the only possible execution order.

[0091] This invention also provides a corresponding apparatus for the training data reading method of a storage device, further enhancing the practicality of the method. The apparatus can be described from both a functional module perspective and a hardware perspective. The following describes the training data reading apparatus for a storage device provided by this invention. This apparatus is used to implement the training data reading method for a storage device provided by this invention. In this embodiment, the training data reading apparatus for a storage device may include or be divided into one or more program modules. These one or more program modules are stored in a storage medium and executed by one or more processors to complete the training data reading method for a storage device disclosed in this embodiment. The program module referred to in this embodiment refers to a series of computer program instruction segments capable of performing a specific function, which is more suitable than the program itself for describing the execution process of the training data reading apparatus for a storage device in the storage medium. The following description will specifically introduce the functions of each program module in this embodiment. The training data reading apparatus for a storage device described below can be referred to in correspondence with the training data reading method for a storage device described above.

[0092] From the perspective of functional modules, see Figure 6 , Figure 6 A structural diagram of the training data reading device for the storage device provided in this embodiment, in one specific implementation, shows that the device may include: The training data determination module 601 is used to determine the training data corresponding to the target training task. The training data partitioning module 602 is used to divide the training data into multiple sub-training data blocks; The first randomization processing module 603 is used to randomize each sub-training data block to obtain the first random access order corresponding to each sub-training data block. The prefetch module 604 is used to prefetch each sub-training data block into the cache according to the first random access order before receiving the training data reading request generated by the training task; The cache block splitting module 605 is used to split each sub-training data block in the cache into sub-data blocks to be read to satisfy the training data reading request when a training data reading request is received. The second randomization processing module 606 is used to randomize each sub-data block to be read so as to obtain a second random access order that is different from the first random access order. The read request response module 607 is used to determine the target data corresponding to the training data read request from each sub-data block to be read according to the second random access order.

[0093] The training data reading device for the storage device mentioned above is described from the perspective of a functional module. Furthermore, the present invention also provides an electronic device, described from a hardware perspective. This electronic device includes a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-described embodiments of the training data reading method for the storage device.

[0094] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the training data reading method embodiments of any of the above storage devices when running.

[0095] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0096] Embodiments of this application also provide a computer program product, which includes a computer program configured to execute the steps in the embodiments of the training data reading method for any of the above-described storage devices when running.

[0097] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which is configured to execute the steps in the training data reading method embodiments of any of the above storage devices when running.

[0098] The present invention also provides a storage device, see [link to relevant documentation]. Figure 7 It may include a memory 701 for storing training data and a computer program; a buffer 702 for buffering data read from the memory 701; a communication interface 703 for receiving training data read requests and sending target data; and a controller 704 connected to the memory 701, coupled to the buffer 702 and the communication interface 703, for implementing the steps of the training data read method of the storage device described in any of the above embodiments when executing the computer program.

[0099] In this system, memory 701 refers to a storage medium, such as a hard disk or solid-state drive, used for persistently storing training data. Cache 702 refers to a high-speed storage medium used for temporarily storing data read from memory, with an access speed higher than that of the memory. Communication interface 703 refers to a hardware interface used for receiving training data read requests and sending target data; this interface supports data communication between the storage device and the host device. Controller 704 refers to a hardware processor used to execute computer programs stored in the memory to control the overall operation of the storage device. In one embodiment, the storage device is an independent disk redundant array storage system, where memory 701 consists of multiple physical hard disks, which are combined into logical storage units by a redundant array controller. Cache 702 is dynamic random access memory, with an access speed higher than that of hard disks. Communication interface 703 is a Fibre Channel interface or Ethernet interface used to connect the storage area network between the storage device and the host device. Controller 704 is a dedicated processor embedded in the storage device, running the storage operating system and input / output processing programs. In another embodiment, the storage device is a distributed storage cluster. Memory 701 consists of hard disks distributed across multiple physical nodes, with the hard disks of each node interconnected through a network to form a unified storage pool. The cache 702 is the memory on each node. The communication interface 703 is a remote direct memory access network interface. The controller 704 is the processor on each physical node, and the processors work together to provide a unified data access service.

[0100] Finally, the present invention also provides a training data reading system, please refer to [link to relevant documentation]. Figure 8 This includes server 801 and storage device 802; Before the target training task is executed, server 801 sends a notification message about the training data required for the target training task to storage device 802 through the communication interface of storage device 802. During the execution of the target training task, when a training data read request is generated, the server 801 sends the training data read request to the controller of storage device through the communication interface. The memory of storage device 802 stores the training data corresponding to the training task of the server in advance. When the controller receives the training data read request and executes the computer program stored in the memory, it implements the steps of the training data read method of storage device described in any of the above embodiments, and sends the target data corresponding to the training data read request to server 801 through the communication interface.

[0101] In this context, a server refers to a host device running a deep learning training task, which is connected to a storage device via a communication network. A notification message is a message sent by the server to the storage device before starting the training task, informing it of the training dataset information. Target data refers to the data determined by the storage device from the sub-data blocks to be read based on the training data read request. In one implementation, when starting a deep learning training task, the server sends a training task start notification to the management interface of the storage device. This notification message includes the identification information of the training dataset used in this training, the capacity information of the training dataset, and the amount of data required for a single batch in the training task. The main controller of the storage device receives the notification message and prepares the training data read service based on the information in the notification message. In one implementation, the server periodically generates training data read requests according to the training progress. Each training data read request includes the amount of data required for this read and the logical location information of the data in the training dataset. The server sends the request to the communication interface of the storage device via the storage area network, and the controller of the storage device receives the request from the communication interface. In one implementation, the controller determines the target data from the cache according to the method steps and sends the target data to the server through the communication interface. After receiving the target data, the server uses it for a single iteration of training the deep learning model.

[0102] For example, in an application scenario where the storage device includes a master controller and multiple slave controllers, the server 801 sends a training data read request to the master controller; the master controller receives the training data read request and determines the sub-data blocks that each slave controller needs to provide based on the correspondence between each sub-training data block and each controller; each slave controller determines the sub-data block to be read from its corresponding cached sub-training data blocks based on the amount of training data in the training data read request, and sends the sub-data block to be read to the master controller; the master controller combines the received sub-data blocks to be read into the target data corresponding to the training data read request, and sends the target data to the server 801 through the communication interface.

[0103] In one implementation, server 801 sends a training data read request to the communication port of the main controller of the storage device via a storage area network (SLAN). The main controller, acting as the entry point for the storage device to provide services, receives all data access requests from the server. Upon receiving the training data read request, the main controller parses the single training data volume information and the logical location information of the data within the training dataset contained in the request. Based on the logical location information, the main controller determines which sub-training data blocks correspond to that location, and then determines the slave controller where each relevant sub-training data block resides based on the correspondence between the sub-training data blocks and each controller. The main controller sends the sub-data block information to be read from each slave controller to the corresponding slave controller. Each slave controller, upon receiving the sub-data block read instruction from the main controller, searches for the corresponding sub-training data block in its local cache. The slave controller splits the sub-training data block into multiple sub-data blocks to be read, and determines the sub-data block to be read this time. The slave controller sends the sub-data block to be read to the main controller via the high-speed interconnect network within the storage device. After receiving the sub-data blocks to be read from each slave controller, the main controller concatenates the multiple sub-data blocks into complete target data according to their logical order in the training data. The main controller then sends the concatenated target data to server 801 via the communication interface.

[0104] To enable those skilled in the art to more clearly understand the technical solution of the present invention, an exemplary embodiment is also provided, such as... Figure 9 As shown, it may include the following: In this embodiment, the server acts as the host device, connected to the storage device controller via an external card such as FC (Fibre Channel). The controller, in turn, is connected to the backend storage disk via an external card such as SAS (Serial Attached SCSI Host Bus Adapter). When an application on the server (such as deep learning) generates a read I / O request, the request is first sent to one of the storage device controllers, such as controller A, via FC. After recognizing the read request, controller A searches the cache based on the volume to which the requested data belongs and its address within that volume. If the data is found, it is directly returned to the server, completing one host read I / O request processing, known as a cache hit. If the data is not in the cache, the cache module further passes the read I / O request to the lower-level modules until the data is read from the backend storage disk and processed by the next layer of the I / O stack before being returned to the server, completing one host read I / O request processing.

[0105] In this embodiment, the storage device can pre-store the training dataset required for the deep learning training task. Before the deep learning training begins, the storage device learns the training dataset used for this training through a message. The storage device determines the start and end addresses of the training dataset in the storage space. The storage device obtains the optimal block granularity of the dataset according to the current virtualization strategy. The virtualization strategy includes information such as the disk redundancy array type, stripe size, and block size. If the block granularity is a specified value, the training dataset is blocked according to the data volume of that granularity. For a dataset with a total data volume of a specified size and a block granularity of another specified size, a corresponding number of data blocks can be obtained. In reality, the total data volume of a deep learning training set is often in the TB or PB range, far exceeding the data volume of the example above; this is only used to illustrate the block granularity principle. Alternatively, the data block granularity can be based on the data volume required for each batch of deep learning training. In one implementation, the optimal block granularity is the greatest common divisor of the data volume required for each batch of deep learning and the block granularity of the virtualization strategy. The storage device reads a cryptographically secure pseudo-random number generator from the operating system to obtain random numbers as an entropy source. These random numbers are then encrypted and hashed using a secure hash algorithm to further enhance randomness, resulting in a random sequence seed. This random sequence seed is then passed to a Mersenne spin algorithm pseudo-random number generator to generate random sequences corresponding to each sub-training data block. The sub-training data blocks, originally having consecutive sequence numbers based on their addresses, are accessed in a random order after this processing. The storage device saves this random sequence to a data structure equal to the number of sub-training data blocks and synchronizes this data structure to each controller in the cluster. For the allocation of data blocks handled by different controllers, the storage device introduces a consistent hash ring. When the storage device cluster contains multiple controllers, the hash ring contains the same number of values ​​as the controllers, i.e., the identifiers of each controller. Adding a controller to the cluster adds an identifier to the hash ring; removing a controller decrements an identifier. Each controller determines its assigned data block using the index of the randomized sequence data structure. The determination method is as follows: The data block index is moduloed by the total number of controllers. If the result equals the identifier of a controller, that controller is responsible for that data block. Once the data block index is determined, the original number of the sub-training data block corresponding to that index position in the address sequence can be obtained through a randomized sequential data structure. Each controller calculates the data blocks it is responsible for based on the above logic. Data blocks are prefetched sequentially from front to back in a random order. During the prefetching process, since the random order itself is random, the prefetching order of each data block is also random throughout the entire prefetching process. The storage device monitors the front-end data read bandwidth in real time. The back-end data prefetch bandwidth is generally no higher than a specified multiple of the front-end data read bandwidth to reserve bandwidth resources for other applications.

[0106] For server-side deep learning training, the storage device also supplies data randomly. The storage device controller receives the training request for each batch from the server, which includes the size and quantity of small data blocks. The storage controller divides the prefetched cached sub-training data blocks into multiple small data blocks based on their size. The storage device uses a pseudo-random number generator to randomize these small data blocks, generating a random access order. The storage device then supplies data to the server according to the number of small data blocks required for each training batch, following an average distribution principle. This completes the random small-block hit access function for front-end deep learning training, ensuring both randomness and high bandwidth in data supply. Accessed cached data is immediately released to free up cache space for storing other unaccessed data in subsequent requests.

[0107] As can be seen from the above, this embodiment decentralizes the data selection and scheduling functions to the storage device itself. This allows the storage device to access the storage medium sequentially in large blocks during the data transfer process from the storage medium to the cache, ensuring full utilization of the storage medium's data read bandwidth. Conversely, during the data transfer process from the cache to the training task, data is supplied randomly in small blocks, guaranteeing the data randomness required for the training task. This application decouples the conflicting high bandwidth and data randomness requirements in deep learning training scenarios to different levels within the storage device for separate processing, thereby improving the input / output performance of the storage device in deep learning training scenarios.

[0108] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The systems, computing devices, or apparatuses described herein encompass a wide range of means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0109] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0110] The foregoing has provided a detailed description of the storage device, training data reading method, medium, product, and system provided by the present invention. The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Several improvements and modifications can be made to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A method for reading training data from a storage device, characterized in that, include: Determine the training data corresponding to the target training task, and divide the training data into multiple sub-training data blocks; Randomize each sub-training data block to obtain the first random access order corresponding to each sub-training data block; Before receiving a training data read request generated by the training task, each sub-training data block is prefetched into the cache according to the first random access order; When a training data read request is received, each sub-training data block in the cache is split into sub-data blocks to be read to satisfy the training data read request; Each sub-data block to be read is randomized to obtain a second random access order that is different from the first random access order. The target data corresponding to the training data reading request is determined from each sub-data block to be read according to the second random access order.

2. The method for reading training data from a storage device according to claim 1, characterized in that, Randomization is performed on each sub-training data block, including: Obtain entropy source data; Perform a hash operation on the entropy source data to obtain the first random seed; The first random seed is input into the pseudo-random number generator to generate the first random access order.

3. The method for reading training data from a storage device according to claim 2, characterized in that, Randomization is performed on each sub-data block to be read, including: Obtain a second random seed; the second random seed is different from the first random seed; The second random seed is input into the pseudo-random number generator to generate the second random access order.

4. The method for reading training data from a storage device according to claim 1, characterized in that, Determine the training data corresponding to the target training task, and divide the training data into multiple sub-training data blocks, including: When a training task start notification is received, the start and end addresses of the training data required for the target training task in the storage address space are determined according to the training task start notification. Based on the address range between the start address and the end address, the training data is divided into multiple sub-training data blocks according to the address mapping granularity of the storage medium; the address mapping granularity is determined based on the redundant array type and stripe capacity of the storage medium.

5. The method for reading training data from a storage device according to claim 1, characterized in that, When the address mapping granularity is inconsistent with the training data volume, the training data is divided into multiple sub-training data blocks, including: Obtain the amount of training data required for a single batch in the target training task; The address mapping granularity of the storage medium is compared with the amount of training data, and the data size of the sub-training data block is determined according to the greatest common divisor of the address mapping granularity and the amount of training data, so that the data size of the sub-training data block is equal to the greatest common divisor.

6. The method for reading training data from a storage device according to claim 1, characterized in that, Prefetch each sub-training data block into the cache, including: The data transfer rate between the storage device and the host device that issues the target training task is detected; Based on the data transmission rate, the rate at which each sub-training data block is prefetched from the storage medium is controlled.

7. The method for reading training data from a storage device according to claim 6, characterized in that, Based on the data transmission rate, controlling the rate at which each sub-training data block is prefetched from the storage medium includes: Obtain preset rate adjustment parameters; The data prefetch rate is determined based on the preset rate adjustment parameter and the data transmission rate, so that the ratio of the data prefetch rate to the data transmission rate is not higher than the sum of 1.2 and the preset floating value.

8. The method for reading training data from a storage device according to claim 6, characterized in that, Prefetch each sub-training data block into the cache, including: Continuously monitor the data transmission rate of the storage device sending data to the host device through the communication interface; When the data transmission rate increases relative to the value at the previous moment, the rate of prefetching sub-training data blocks from the storage medium will be increased. When the data transmission rate decreases relative to the value at the previous moment, the rate at which sub-training data blocks are prefetched from the storage medium is reduced.

9. The method for reading training data from a storage device according to claim 1, characterized in that, The training data read request also includes information on the amount of training data in a single run, dividing each sub-training data block in the cache into sub-data blocks to be read to satisfy the training data read request, including: Each sub-training data block in the cache is divided into multiple sub-data blocks to be read according to the amount of training data in a single session. The amount of data in each sub-data block to be read is equal to the amount of data in a single training session.

10. The method for reading training data from a storage device according to claim 1, characterized in that, After determining the target data corresponding to the training data reading request from each sub-block of data to be read, the method further includes: The target data is sent to the host device; Once the target data is successfully sent, the sub-data blocks that make up the target data will be deleted from the cache.

11. The method for reading training data from a storage device according to any one of claims 1 to 10, characterized in that, The storage device includes multiple controllers, which together provide data access services for the training data; According to the first random access order, each sub-training data block is prefetched into the cache, including: Based on the correspondence between each sub-training data block and each controller, each controller prefetches its corresponding sub-training data block from the storage medium to the cache according to the first random access order.

12. The method for reading training data from a storage device according to claim 11, characterized in that, The storage device includes a master controller and at least one slave controller, and before determining the correspondence between each sub-training data block and each controller, it further includes: The main controller constructs a data block query structure based on the number of sub-training data blocks, and each element in the data block query structure corresponds one-to-one with each sub-training data block. The sub-training data block indexes of each sub-training data block are stored sequentially into the data block query structure according to their position numbers in the first random access order, and the data block query structure is synchronized to each slave controller.

13. The method for reading training data from a storage device according to claim 11, characterized in that, Each sub-training data block corresponds to a sub-training data block index, which is used to identify the position of the corresponding sub-training data block in the address order of the training data, and further includes: Get the number of controllers contained in the storage device; Divide the sub-training data block index value of each sub-training data block by the number of controllers, and determine the controller corresponding to the corresponding sub-training data block based on the remainder.

14. The method for reading training data from a storage device according to claim 13, characterized in that, Also includes: When the number of controllers changes, obtain the updated number of controllers; The controllers corresponding to each sub-training data block are re-determined based on the updated number of controllers.

15. The method for reading training data from a storage device according to claim 11, characterized in that, The storage device includes a master controller and at least one slave controller, which determines the training data corresponding to the target training task, including: The main controller acquires the identification information of the target training task and determines the address range of the training data in the storage address space of the storage device based on the identification information.

16. A storage device, characterized in that, include: Memory, used to store training data and computer programs; A cache is used to cache data read from the memory; The communication interface is used to receive training data reading requests and send target data; A controller, connected to the memory, coupled to the buffer and the communication interface, is used to implement the steps of the training data reading method of the storage device as described in any one of claims 1 to 15 when executing the computer program.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by the controller, implements the steps of the training data reading method for the storage device as described in any one of claims 1 to 15.

18. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the controller, it implements the steps of the training data reading method of the storage device according to any one of claims 1 to 15.

19. A training data reading system, characterized in that, Including servers and storage devices; Before the target training task is executed, the server sends a notification message about the training data required for the target training task to the storage device through the communication interface of the storage device. During the execution of the target training task, when a training data read request is generated, the training data read request is sent to the controller of the storage device through the communication interface; The memory of the storage device pre-stores the training data corresponding to the training task of the server. When the controller receives the training data read request and executes the computer program stored in the memory, it implements the steps of the training data read method of the storage device as described in any one of claims 1 to 15, and sends the target data corresponding to the training data read request to the server through the communication interface.

20. The training data reading system according to claim 19, characterized in that, The storage device includes a master controller and multiple slave controllers; The server sends the training data read request to the main controller; The main controller receives the training data reading request and determines the sub-data blocks that each slave controller needs to provide based on the correspondence between each sub-training data block and each controller. Each slave controller determines the sub-data block to be read from its corresponding cached sub-training data block based on the amount of training data in the training data read request, and sends the sub-data block to be read to the master controller. The main controller combines the received sub-data blocks to be read into the target data corresponding to the training data reading request, and sends the target data to the server through the communication interface.

Images