Mechanical hard disk sequential parallel scheduling double random sample high-speed access method and system

Abstract

The invention relates to a mechanical hard disk sequential parallel scheduling double random sample high-speed access method and system, and the method comprises the steps: packaging training samples into a fixed number of data packages, and uniformly distributing the data packages to a plurality of mechanical hard disks through a greedy algorithm; a plurality of data readers are created through the data manager, and all the mechanical hard disks are distributed to the data readers in a balanced mode; circularly accessing the distributed mechanical hard disk data packets through an infinite iterator of each data reader, randomly rearranging a data packet path queue during each cycle, continuously reading training samples from the current data packets in sequence, and storing the training samples into a buffer pool of a data manager; sequential reading operation is executed in parallel through a plurality of data readers, and the throughput of each mechanical hard disk is accumulated; a client request is received based on a high-efficiency communication protocol, and a training sample is randomly selected and returned, so that the purpose of random sample high-speed access which is low in cost, high in throughput and capable of meeting deep learning training requirements is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage and access optimization, and in particular to a method and system for sequentially and parallelly scheduling dual random sample high-speed access to a mechanical hard disk. Background Art

[0002] During deep learning model training, access performance to training data directly determines model iteration efficiency. Traditional solutions typically rely on solid-state drives (SSDs) or memory as storage media to meet the high-speed random access requirements of training tasks. While SSDs offer significantly better random read performance than mechanical hard drives, they are expensive, costing approximately 5-8 times more per unit than HDDs, and have limited lifespans in large-scale data scenarios. While memory can provide nanosecond latency, it is limited by capacity and cannot directly store terabytes of training data.

[0003] Mechanical hard disks (HDDs) offer a potential alternative due to their low cost and high sequential read speeds. However, their random read performance is extremely poor, making them unable to directly meet the random access requirements of samples used in deep learning training. Existing technologies attempt to optimize HDD performance through solutions such as data preloading and cache tiering, but these efforts suffer from issues such as a lack of randomness, throughput bottlenecks, and insufficient scalability. For example, relying solely on data preloading can destroy the sample randomness required for training, while cache tiering solutions can easily cause cache breakdown as data scale increases, leading to a sharp drop in performance. Therefore, achieving a balance between low cost, high throughput, and randomness on mechanical hard disks has become a technical challenge that urgently needs to be overcome. Summary of the Invention

[0004] The main purpose of the present invention is to provide a method and system for high-speed access to dual random samples in sequential parallel scheduling of a mechanical hard disk. Through sequential reading optimization and double-layer randomness design of the mechanical hard disk, the purpose of high-speed access to random samples with low cost and high throughput that meets the needs of deep learning training can be achieved.

[0005] To achieve the above object, the present invention provides a method for high-speed access of mechanical hard disks by sequential parallel scheduling of double random samples, comprising the following steps: Pack the training samples into a fixed number of data packets, and evenly distribute the data packets to multiple mechanical hard drives using a greedy algorithm; Create multiple data readers through the data manager, evenly distribute all mechanical hard drives to each data reader, and bind one or more mechanical hard drives to each data reader; The infinite iterator of each data reader loops through the allocated mechanical hard disk data packets, randomly rearranges the data packet path queue in each loop, and continuously reads training samples from the current data packet in sequence and stores them in the buffer pool of the data manager; The sequential read operation is performed in parallel by multiple data readers, the throughput of each mechanical hard disk is accumulated, and the data filling rate of the buffer pool is maintained. Based on the efficient communication protocol, the client request is received, the sample extraction instruction is sent, the training sample is randomly selected from the buffer pool, and the training sample is returned.

[0006] Further, the step of packing the training sample into a fixed number of data packets and uniformly distributing the data packets to multiple mechanical hard disks by a greedy algorithm comprises: Pack all collected training sample data into multiple data packets, and the number of samples in each data packet is fixed; Traverse the remaining capacity of all mechanical hard disks, sort them in descending order of capacity to generate a hard disk queue; Select the mechanical hard disk with the largest remaining capacity in the hard disk queue in turn, distribute an undistributed data packet to the hard disk, and update the remaining capacity of the hard disk; Repeat the distribution operation, and the difference in the number of data packets between each mechanical hard disk is not more than 1, until all data packets are distributed.

[0007] Further, the step of uniformly distributing the data packets to multiple mechanical hard disks by a greedy algorithm further comprises: When the number of mechanical hard disks is greater than the number of data packets, mark the excess hard disks as to-be-expanded storage nodes; When a new data packet is added, it is preferentially distributed to the to-be-expanded storage node.

[0008] Further, the step of creating multiple data readers by the data manager and evenly distributing all mechanical hard disks to each data reader comprises: According to the ratio of the total number of mechanical hard disks to the number of data readers, at least one mechanical hard disk is allocated to each data reader by a greedy algorithm; The difference in the number of mechanical hard disks allocated to each data reader is not more than 1, and the same mechanical hard disk is only allocated to one data reader.

[0009] Further, the step of creating multiple data readers by the data manager and evenly distributing all mechanical hard disks to each data reader further comprises: Monitor the real-time load data of each data reader, including CPU occupancy, memory usage, and hard disk I / O delay, and set the initial threshold value to CPU 70% and I / O delay 200ms; Based on the load change, the CPU occupancy threshold value is dynamically adjusted by a time window moving average algorithm, and the adjustment amplitude is positively correlated with the load fluctuation rate; When the load exceeds the current threshold value, the mechanical hard disks responsible for the high-load data reader are migrated to the data reader with the lowest load.

[0010] Furthermore, the step of looping through the infinite iterator of each data reader to access the allocated mechanical hard disk data packet, randomly rearranging the data packet path queue in each loop, and continuously reading training samples from the current data packet in sequence and storing them in the buffer pool of the data manager includes: Initialize an infinite iterator for each data reader, and associate the infinite iterator with the corresponding allocated mechanical hard disk data packet path list; Traversing the data packet path list in a circular order, and triggering a random reordering of the paths in the list after the infinite iterator completes a round of data packet traversal; Check the integrity of the data packet path, remove damaged paths, re-queue valid paths, and perform random re-arrangement of paths; Record the random seeds of each re-arrangement and synchronize the seed value to the client; According to the rearranged path sequence, the data packet contents are read sequentially from the mechanical hard disk, and the samples are stored in the buffer pool of the data manager.

[0011] Furthermore, the step of performing sequential read operations in parallel by multiple data readers to accumulate the throughput of each mechanical hard disk and maintain the data filling rate of the buffer pool includes: Each data reader sends a continuous data block read instruction to the assigned mechanical hard disk, triggering the hard disk head to move sequentially along the radial direction of the disk; Based on the parallel scheduling mode, the path of the data packets to be read is arranged according to the physical order of the mechanical hard disk tracks, and the head movement direction is constrained to unidirectional scanning; The difference between the sample filling rate of the buffer pool and the client request rate is calculated in real time. If the difference exceeds the preset tolerance, the number of concurrent threads of the data reader is increased proportionally.

[0012] Furthermore, the steps of receiving a client request based on an efficient communication protocol, sending a sample extraction instruction, randomly selecting a training sample from a buffer pool and returning the sample include: An asynchronous message queue protocol is used to establish a multiplexed channel between the data manager and the client. The asynchronous message queue protocol supports high concurrency requests and low latency transmission; In response to the client request, the asynchronous message queue protocol is returned to the client in real time through the multiplexing channel to complete the sample access.

[0013] Furthermore, the step of establishing a multiplexing channel between the data manager and the client further includes: Establish a many-to-one broadcast channel between the data manager and the client, and periodically send the sample distribution metadata of the buffer pool to all clients; Based on the sample type distribution in the metadata, the request priority is dynamically adjusted to prioritize scarce sample models.

[0014] The present invention also provides a mechanical hard disk sequential parallel scheduling dual random sample high-speed access system, comprising: A sample distribution unit is used to package the training samples into a fixed number of data packets and evenly distribute the data packets to multiple mechanical hard disks using a greedy algorithm; A hard disk dispatching unit is used to create multiple data readers through a data manager, evenly distribute all mechanical hard disks to each data reader, and bind one or more mechanical hard disks to each data reader; The data reading unit is used to loop through the infinite iterator of each data reader to access the allocated mechanical hard disk data packets, randomly rearrange the data packet path queue in each loop, and continuously read the training samples from the current data packet in sequence and store them in the buffer pool of the data manager; The concurrent acceleration unit is used to perform sequential read operations in parallel through multiple data readers, accumulate the throughput of each mechanical hard disk, and maintain the data filling rate of the buffer pool; The sample response unit is used to receive client requests based on an efficient communication protocol, send sample extraction instructions, randomly select training samples from the buffer pool and return them.

[0015] The method and system for high-speed access to dual random samples using sequential parallel scheduling of mechanical hard disks provided by the present invention have the following beneficial effects: the present invention constructs a storage system entirely based on mechanical hard disks, replacing traditional high-cost SSD or memory solutions, significantly reducing hardware procurement and maintenance costs, and is particularly suitable for massive data storage scenarios; through data packaging and multi-hard disk parallel scheduling, the potential for sequential reading of mechanical hard disks is fully exploited, and the overall system throughput scales linearly with the number of hard disks, meeting the needs of high-concurrency data access. A dynamic disorder mechanism is adopted at the data packet access level to break the physical order limitation of storage; at the sample return level, intelligent random sampling is performed through the buffer pool to ensure the unbiasedness and diversity of training data, effectively matching model optimization requirements; the present invention supports dynamic expansion of hard disk and data reader scale, adapting to different training scenarios from single machines to distributed clusters, and combining efficient communication protocols to achieve seamless access and resource collaboration for multiple clients, thereby improving overall system utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of a method for high-speed access of mechanical hard disks in sequential parallel scheduling of double random samples in one embodiment of the present invention; Figure 2 This is a structural block diagram of a mechanical hard disk sequential parallel scheduling dual random sample high-speed access system in one embodiment of the present invention; The object, technical solutions and advantages of the present application will be further described in combination with embodiments and with reference to the drawings. DETAILED DESCRIPTION

[0017] In order to make the object, technical solutions and advantages of the present application clearer, the present application will be further described in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not used to limit the present application.

[0018] Reference Figure 1 A flowchart of a mechanical hard disk sequential parallel scheduling double random sample high-speed access method according to the present application is shown in the figure, which comprises the following steps: S1, packing the training samples into a fixed number of data packets, and distributing the data packets to multiple mechanical hard disks by using a greedy algorithm; S2, creating multiple data readers by using a data manager, and distributing all the mechanical hard disks to the data readers, each of which is bound to one or more mechanical hard disks; S3, cyclically accessing the data packets of the mechanical hard disks by using the unlimited iterator of each data reader, randomly rearranging the data packet path queue at each cycle, and continuously reading the training samples from the current data packet in sequence and storing them in the buffer pool of the data manager; S4, performing the sequential reading operation in parallel by using multiple data readers, accumulating the throughput of each mechanical hard disk, and maintaining the data filling rate of the buffer pool; S5, receiving the client request based on the efficient communication protocol, sending the sample extraction instruction, and randomly selecting the training samples from the buffer pool and returning.

[0019] As described in step S1, the training samples are packed in a fixed number (such as 1000 per packet), ensuring that the size of each data packet is similar, so as to eliminate the small file metadata overhead and improve the efficiency of single sequential reading; the queue is generated according to the remaining capacity of the mechanical hard disk, and the hard disk with the largest remaining capacity is preferentially allocated; a continuous storage area is pre-allocated for each hard disk, and the data packets are stored in physical order, reducing the seek time of the magnetic head.

[0020] As described in step S2, according to the total number of hard disks and the number of readers, the hard disks are distributed according to the greedy algorithm (such as 10 hard disks→3 readers→4-3-3), ensuring that the load difference is ≤1; the CPU occupancy rate (threshold 70%) and the I / O delay (threshold 200ms) are monitored, and the hard disk migration is triggered when the load is high, so as to achieve load balancing and reduce the system throughput fluctuation rate; each hard disk is only bound to one reader, so as to avoid I / O competition and reduce the multi-thread reading conflict rate.

[0021] As described in step S3 above, the iterator is initialized to associate a list of packet paths. After each round of traversal, the Fisher-Yates shuffle algorithm is triggered to achieve packet-level reordering. Paths that fail CRC checks are eliminated to ensure that valid data enters the buffer pool. Data packets are read sequentially from the mechanical hard drive according to the reordered paths, with a single read of 64MB contiguous blocks (matching the mechanical hard drive's optimal throughput). This ensures a throughput of 180MB / s per drive despite packet-level randomness, significantly improving the speed compared to purely random reads.

[0022] As described in step S4 above, the parallel scheduling module forces the head to scan in one direction according to the physical order of the tracks (for example, from outer circle to inner circle), thereby reducing the proportion of seek time. The buffer pool filling rate (R_fill) and the client request rate (R_consume) are calculated. If R_fill < R_consume, the concurrent threads are increased according to the difference ratio (for example, 20% → +2 threads) to ensure that the buffer pool sample ready rate (Ready Rate) is ≥99.5%, thereby avoiding the situation where requests are starved due to lack of available resources, that is, they cannot be processed for a long time.

[0023] As described in step S5 above, the ZeroMQ ROUTER-DEALER model is used, with a single channel supporting 10K QPS and a response delay of <1ms. Compared with the HTTP protocol, this reduces latency while improving throughput. The buffer pool sample distribution metadata (such as category proportions) is broadcast periodically, and clients prioritize requests for scarce samples (such as the 0.1% positive samples in fraud detection).

[0024] In one embodiment, the system configuration includes four mechanical hard drives (each with a capacity of 4TB, a rotation speed of 7200RPM, and a sequential read speed of 180MB / s), one data manager (CPU: 8 cores, memory: 32GB), and three data readers (each with an independent I / O thread). The total number of training samples is 100,000, which are pre-packaged into 1,000 data packets, each containing 100 samples and sized at 64MB. A greedy algorithm is used to evenly distribute the data packets across the four mechanical hard drives, with each drive storing 250 data packets to ensure load balancing. Based on the ratio of hard drives to readers, the data manager allocates the four hard drives to three data readers, with one reader responsible for two drives and the other two for one drive each. This achieves exclusive binding and parallel scheduling of hardware resources.

[0025] Each data reader accesses its assigned mechanical hard drive through an infinite iterator. After each round of traversal, the iterator randomly rearranges the list of data packet paths and removes damaged paths, retaining only valid paths for subsequent reads. The data reader continuously reads data packet contents from the hard drive in the order of the rearranged paths. The sequential read speed of a single hard drive is stable at 180MB / s, and the total system read speed reaches 540MB / s. Read samples are continuously stored in the data manager's buffer pool. The buffer pool capacity is dynamically adjusted to match the client's request rate, ensuring that the sample filling rate always exceeds the consumption demand.

[0026] The client establishes an efficient communication channel with the data manager through the ZeroMQ protocol, supporting high-concurrency requests and low-latency transmission. When the data manager responds to a request, it uses a lock-free queue to randomly extract samples from the buffer pool and returns them in real time through an asynchronous message queue. To optimize training efficiency, the system periodically broadcasts sample distribution metadata in the buffer pool to the client, and the client prioritizes requests for samples in scarce categories based on this, further accelerating model convergence. Tests show that this embodiment achieves a throughput of 540MB / s on four mechanical hard drives, which is 15 times higher than the traditional random read solution, while reducing hardware costs by 75%, fully verifying its feasibility and superiority in low-cost deep learning training scenarios.

[0027] In a real-world test environment, the system throughput reached 540 MB / s, with dual-layer randomness entropy values ​​reaching 4.95 and 4.9 (maximum 5.0) at the packet and sample levels, respectively, meeting the stringent data randomness requirements of model training. Client request response latency was less than 1ms, a 94% reduction compared to traditional HTTP protocols. Hardware costs were only $800 (for four mechanical hard drives), saving $2,400 compared to a similarly performing SSD solution. Furthermore, a dynamic load balancing mechanism kept the I / O latency fluctuation of each drive within ±5%, and the buffer pool sample readiness rate consistently exceeded 99.9%, completely eliminating the risk of data starvation or overflow during training. This embodiment achieves sample access performance nearing SSD-level performance on mechanical hard drives through a combination of greedy allocation, dual-layer randomness design, parallel sequential reads, and an efficient communication protocol. This solution supports large-scale deep learning training tasks at a very low hardware cost, providing a new technical path for reducing costs and increasing efficiency in AI infrastructure.

[0028] Reference Figure 2 , is a structural block diagram of a mechanical hard disk sequential parallel scheduling dual random sample high-speed access system according to an embodiment of the present invention, comprising: A sample distribution unit is used to package the training samples into a fixed number of data packets and evenly distribute the data packets to multiple mechanical hard disks using a greedy algorithm; A hard disk dispatching unit is used to create multiple data readers through a data manager, evenly distribute all mechanical hard disks to each data reader, and bind one or more mechanical hard disks to each data reader; The data reading unit is used to loop through the infinite iterator of each data reader to access the allocated mechanical hard disk data packets, randomly rearrange the data packet path queue in each loop, and continuously read the training samples from the current data packet in sequence and store them in the buffer pool of the data manager; The concurrent acceleration unit is used to perform sequential read operations in parallel through multiple data readers, accumulate the throughput of each mechanical hard disk, and maintain the data filling rate of the buffer pool; The sample response unit is used to receive client requests based on an efficient communication protocol, send sample extraction instructions, randomly select training samples from the buffer pool and return them.

[0029] For the specific implementation of each unit in the above device example, please refer to the above method embodiment, which will not be repeated here.

[0030] In summary, the present invention packages training samples into a fixed number of data packets, and evenly distributes the data packets to multiple mechanical hard disks through a greedy algorithm; creates multiple data readers through a data manager, and evenly distributes all mechanical hard disks to each data reader, and each data reader is bound to one or more mechanical hard disks; loops through the allocated mechanical hard disk data packets through the infinite iterator of each data reader, randomly rearranges the data packet path queue in each loop, and continuously reads training samples from the current data packet in sequence and stores them in the buffer pool of the data manager; executes sequential reading operations in parallel through multiple data readers, accumulates the throughput of each mechanical hard disk, and maintains the data filling rate of the buffer pool; receives client requests based on an efficient communication protocol, sends sample extraction instructions, randomly selects training samples from the buffer pool and returns them, so as to achieve low cost, high throughput and high-speed access to random samples that meets the needs of deep learning training.

[0031] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware using a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media provided herein and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM.

[0032] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0033] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for high-speed access of mechanical hard disks by sequential parallel scheduling of double random samples, characterized in that: The following steps are involved: Pack the training samples into a fixed number of data packets, and evenly distribute the data packets to multiple mechanical hard drives using a greedy algorithm; Create multiple data readers through the data manager, evenly distribute all mechanical hard drives to each data reader, and bind one or more mechanical hard drives to each data reader; The infinite iterator of each data reader loops through the allocated mechanical hard disk data packets, randomly rearranges the data packet path queue in each loop, and continuously reads training samples from the current data packet in sequence and stores them in the buffer pool of the data manager; Multiple data readers execute sequential read operations in parallel, accumulating the throughput of each mechanical hard disk and maintaining the data filling rate of the buffer pool; Based on an efficient communication protocol, it receives client requests, sends sample extraction instructions, randomly selects training samples from the buffer pool and returns them.

2. The method for high-speed access of mechanical hard disks by sequential parallel scheduling of double random samples according to claim 1, characterized in that: The step of packaging the training samples into a fixed number of data packets and evenly distributing the data packets to multiple mechanical hard disks using a greedy algorithm includes: All collected training sample data are packaged into multiple data packets, with a fixed number of samples in each data packet; Traverse the remaining capacity of all mechanical hard disks and sort them from largest to smallest to generate a hard disk queue; Select the mechanical hard disk with the largest remaining capacity in the hard disk queue in turn, assign an unassigned data packet to the hard disk, and update the remaining capacity of the hard disk; Repeat the allocation operation, with the difference in the number of data packets between each mechanical hard disk not exceeding 1, until all data packets are allocated.

3. The method for high-speed access of mechanical hard disks by sequential parallel scheduling of double random samples according to claim 2, characterized in that: The step of evenly distributing the data packets to multiple mechanical hard disks using a greedy algorithm further includes: When the number of mechanical hard disks is greater than the number of data packets, the excess hard disks are marked as storage nodes to be expanded; When a new data packet is added, it is preferentially allocated to the storage node to be expanded.

4. The method for high-speed access of mechanical hard disks by sequential parallel scheduling of double random samples according to claim 1, characterized in that: The step of creating multiple data readers through the data manager and evenly distributing all mechanical hard disks to the data readers includes: According to the ratio of the total number of mechanical hard disks to the number of data readers, at least one mechanical hard disk is allocated to each data reader through a greedy algorithm; The difference in the number of mechanical hard disks assigned to each data reader does not exceed 1, and the same mechanical hard disk is assigned to only one data reader.

5. The method for high-speed access of mechanical hard disks by sequential parallel scheduling of double random samples according to claim 4 is characterized in that: The step of creating multiple data readers through the data manager and evenly distributing all mechanical hard disks to the data readers further includes: Monitor the real-time load data of each data reader, including the CPU usage, memory usage, and disk I / O latency of the data reader, with initial thresholds set at 70% CPU usage and 200ms I / O latency. Based on load changes, the CPU usage threshold is dynamically adjusted using a time window sliding average algorithm. The adjustment range is positively correlated with the load fluctuation rate. When the load exceeds the current threshold, the hard disks that are responsible for the high-load data reader are migrated to the data reader with the lowest load.

6. The method for high-speed access of mechanical hard disks by sequential parallel scheduling of double random samples according to claim 1, characterized in that: The steps of looping through the infinite iterator of each data reader to access the allocated mechanical hard disk data packet, randomly rearranging the data packet path queue in each loop, and continuously reading training samples from the current data packet in sequence and storing them in the buffer pool of the data manager include: Initialize an infinite iterator for each data reader, and associate the infinite iterator with the corresponding allocated mechanical hard disk data packet path list; Traversing the data packet path list in a circular order, and triggering a random reordering of the paths in the list after the infinite iterator completes a round of data packet traversal; Check the integrity of the data packet path, remove damaged paths, re-queue valid paths, and perform random re-arrangement of paths; Record the random seeds of each re-arrangement and synchronize the seed value to the client; According to the rearranged path sequence, the data packet contents are read sequentially from the mechanical hard disk, and the samples are stored in the buffer pool of the data manager.

7. The method for high-speed access of mechanical hard disks by sequential parallel scheduling of double random samples according to claim 1, characterized in that: The step of performing sequential read operations in parallel by multiple data readers, accumulating the throughput of each mechanical hard disk, and maintaining the data filling rate of the buffer pool includes: Each data reader sends a continuous data block read instruction to the assigned mechanical hard disk, triggering the hard disk head to move sequentially along the radial direction of the disk; Based on the parallel scheduling mode, the path of the data packets to be read is arranged according to the physical order of the mechanical hard disk tracks, and the head movement direction is constrained to unidirectional scanning; The difference between the sample filling rate of the buffer pool and the client request rate is calculated in real time. If the difference exceeds the preset tolerance, the number of concurrent threads of the data reader is increased proportionally.

8. The method for high-speed access of mechanical hard disks by sequential parallel scheduling of double random samples according to claim 1, characterized in that: The steps of receiving a client request based on an efficient communication protocol, sending a sample extraction instruction, randomly selecting a training sample from a buffer pool, and returning the sample include: An asynchronous message queue protocol is used to establish a multiplexed channel between the data manager and the client. The asynchronous message queue protocol supports high concurrency requests and low latency transmission; In response to the client request, the asynchronous message queue protocol is returned to the client in real time through the multiplexing channel to complete the sample access.

9. The method for high-speed access of mechanical hard disks by sequential parallel scheduling of double random samples according to claim 8, characterized in that: The step of establishing a multiplexing channel between the data manager and the client further includes: Establish a many-to-one broadcast channel between the data manager and the client, and periodically send the sample distribution metadata of the buffer pool to all clients; Based on the sample type distribution in the metadata, the request priority is dynamically adjusted to prioritize scarce sample models.

10. A mechanical hard disk sequential parallel scheduling double random sample high-speed access system, characterized in that: include: A sample distribution unit is used to package the training samples into a fixed number of data packets and evenly distribute the data packets to multiple mechanical hard disks using a greedy algorithm; A hard disk dispatching unit is used to create multiple data readers through a data manager, evenly distribute all mechanical hard disks to each data reader, and bind one or more mechanical hard disks to each data reader; The data reading unit is used to loop through the infinite iterator of each data reader to access the allocated mechanical hard disk data packets, randomly rearrange the data packet path queue in each loop, and continuously read the training samples from the current data packet in sequence and store them in the buffer pool of the data manager; The concurrent acceleration unit is used to perform sequential read operations in parallel through multiple data readers, accumulate the throughput of each mechanical hard disk, and maintain the data filling rate of the buffer pool; The sample response unit is used to receive client requests based on an efficient communication protocol, send sample extraction instructions, randomly select training samples from the buffer pool and return them.

Images