A comprehensive memory management method and system based on deep learning training

Abstract

The application relates to a comprehensive memory management method and system based on deep learning training, which comprises the following steps: dividing a data set to form a plurality of data blocks; grouping the plurality of data blocks to form a plurality of data block groups, and mapping the data blocks in the same data block group to the same abstract block, so that a plurality of abstract blocks form an abstract memory space in logic; when a data access request is received, determining an abstract block slot corresponding to a data file requested to be accessed; if the abstract block slot is empty or invalid, selecting a target data block from the data block group according to a filling rate of the data block, reloading the target data block, and filling the reloaded content into the empty slot in the abstract block; and marking the actually consumed data file as invalid. The application makes full use of the characteristic that data access has no fixed order in the deep learning training process, and significantly improves the loading efficiency of the training data and the system I / O throughput capacity.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence training technology, and in particular to a comprehensive memory management method and system based on deep learning training. Background Technology

[0002] Deep learning has permeated various fields, including computer vision, speech recognition, and natural language processing, becoming ubiquitous in society. The widespread adoption of deep learning is largely attributed to its superior accuracy, a quality that depends not only on the deep neural network algorithm but also on the training dataset. High-quality, diverse, and large-scale datasets are crucial for achieving high accuracy in deep neural network models. Consequently, datasets used to train deep neural network models are becoming increasingly large, reaching terabytes in size and containing tens of millions of data instances. Such massive datasets place enormous I / O pressure on deep learning training. In particular, the significant advancements in high-performance computing accelerators (such as GPUs and TPUs) have further widened the gap between the speed of processing training data and the speed of retrieving data from disk, making the I / O performance bottleneck even more significant.

[0003] Addressing the I / O bottleneck in deep learning training presents two challenges. The first is the massive and ever-growing volume of training data. The vast gap between the processing speed and I / O performance of computing units like CPUs and GPUs is well-known. A traditional approach to bridging this gap leverages the fundamental principles of memory hierarchies: using memory as a layer between I / O and computing units. However, the sheer scale of training data, crucial for achieving high model accuracy, necessitates holding only a small fraction of the data in memory. Intelligent memory management mechanisms can typically achieve efficient data caching, but this capability is hampered by the second challenge: the access patterns of deep learning training data are entirely random. This renders various replacement strategies, such as Least Recently Used (LRU) or Least Frequently Used (LFU), which aim to capture certain access patterns, obsolete. Deep learning training iterates over the training dataset randomly at each epoch; this randomness is essential for ensuring training efficiency.

[0004] Due to the random access pattern of deep learning training data, the design and optimization space for its memory management is very limited. Without affecting randomness, most existing methods cache a fixed portion of the data in memory, enabling faster access to the cached data, while still incurring high latency through I / O on the uncached data. Based on this static caching strategy, the proportion of data access that can be accelerated depends entirely on the memory capacity. Unfortunately, memory capacity cannot scale with the amount of data. Furthermore, the input data for deep learning training is typically stored in many small data files, which incur higher I / O overhead when accessed randomly than large data files. To minimize overhead, large analytics systems often reorganize these into relatively large data blocks and employ batch reading. However, due to the random access pattern, directly applying batch reading to deep learning training is ineffective. Specifically, when loading data blocks from external storage, there is no way to ensure that all or most of the data contained in the block is utilized by the deep learning framework before being swapped out of memory.

[0005] For example, CN119025281A discloses a method and apparatus for allocating memory resources, a storage medium, and an electronic device. The method includes: allocating a first memory management algorithm to a memory call request sent by a deep learning framework, wherein the memory call request requests the use of target memory for a training task, and the memory call request carries at least memory type information of the target memory and memory size information of the first memory resource; determining a first basic interface group pre-configured for the target memory based on the memory type information, wherein the device including the deep learning framework deploys at least two types of memory, including the target memory, and wherein the first basic interface group is a group of encapsulated interfaces for the target memory's proprietary interfaces, allowing the encapsulated interface group to be called by any memory management algorithm; and allocating a first memory resource with the memory size indicated by the memory size information to the training task by calling the first basic interface group through the first memory management algorithm. However, the core of this technical solution lies in achieving flexible invocation of the memory management algorithm through interface encapsulation and optimizing resource allocation for different memory types. However, its technical logic still relies on the basic assumption of the traditional memory hierarchy, namely, reducing I / O access by caching some data. When the size of the training data exceeds the memory capacity (e.g., the currently common petabyte-scale training sets), even with intelligent allocation strategies, physical memory still cannot cover a sufficient amount of data. The patent does not propose an innovative mechanism for dynamically expanding memory capacity or breaking through the limitations of the memory-storage hierarchy, resulting in its optimization effect being limited to local resource allocation and failing to resolve the fundamental contradiction of the order-of-magnitude difference between memory capacity and data size.

[0006] CN116501696A discloses a method and apparatus for distributed deep learning training prefetch cache management, comprising: initializing a prefetch cache space during deep learning task initialization; generating a shuffling sequence for each round of training before the start of each round, and constructing a cache index according to the shuffling sequence; retrieving data from the prefetch cache space according to the cache index during deep learning training; if a cache miss occurs, performing batch prefetching in distributed storage based on storage nodes; and initiating cache eviction when the used capacity of the prefetch cache space reaches a specified threshold, evictring cache-hit data. This method and apparatus perform batch prefetching based on storage nodes, reducing the number of communications between nodes and improving communication efficiency. Simultaneously, evictring cache-hit data ensures that the cache contains data that is about to be hit, improving the cache hit rate. However, this technical solution only discloses a simple prefetch linked list eviction mechanism and fails to address the objective defect of local failures during block-level loading of variable-length data.

[0007] Therefore, the technical problem of how to significantly improve the efficiency of accessing deep learning training data has not been solved.

[0008] Furthermore, on the one hand, there are differences in understanding among those skilled in the art; on the other hand, the applicant studied a large number of documents and patents when making this invention, but due to space limitations, not all details and contents were listed in detail. However, this does not mean that the present invention does not possess the features of these prior art. On the contrary, the present invention already possesses all the features of the prior art, and the applicant reserves the right to add relevant prior art to the background art. Summary of the Invention

[0009] In view of the shortcomings of the existing technology, the purpose of this invention is to solve the technical problem that I / O becomes a bottleneck in deep learning training performance when the memory capacity is insufficient to cache the training set in memory.

[0010] To address the shortcomings of existing technologies, this invention provides a comprehensive memory management method based on deep learning training from a first aspect. The method includes: dividing a dataset into multiple data blocks; grouping the multiple data blocks into several data block groups, and mapping data blocks within the same data block group to the same abstract block, with multiple abstract blocks constituting a logical abstract memory space; when a data access request is received, determining the abstract block slot corresponding to the requested data file; if the abstract block slot is empty or invalid, selecting a target data block from the data block group based on the data block's fill rate for reloading, and filling the empty slot in the abstract block with the reloaded content; and marking the actually consumed data file as invalid.

[0011] This technical solution fully leverages the characteristic that data access is not in a fixed order during deep learning training. It avoids the problem of traditional cache replacement strategies failing in random access scenarios, thereby significantly improving the loading efficiency of training data and the system I / O throughput.

[0012] According to a preferred embodiment, the steps of partitioning the dataset include: randomly shuffling the complete dataset to generate a randomized data index sequence; partitioning the data index sequence based on a set data block size to form multiple fixed-size data blocks, each data block containing several data files.

[0013] By shuffling the data, this technique ensures sufficient randomness in the training data access process. Fixed-size blocks provide the basic framework for subsequent batch data loading, helping to improve single-cycle I / O efficiency.

[0014] According to a preferred embodiment, before the step of grouping multiple data blocks to form several data block groups, the method further includes the steps of: determining the number of abstract block slots in the abstract memory space based on the memory capacity in bytes and the average size of the data in the dataset; and determining the size of the data block group based on the total number of data files in the dataset and the number of abstract block slots.

[0015] This technical solution can dynamically adapt to the actual memory capacity in bytes. This logical design, built on top of physical memory resources, aims to shield the underlying physical distribution through logical mapping, supporting efficient random access and reuse.

[0016] According to a preferred embodiment, after determining the abstract block slot corresponding to the data file to be accessed when a data access request is received, the method further includes the step of: if the data file pointed to by the abstract block slot is currently valid data, then directly returning the corresponding data file index.

[0017] For data files that have been filled into abstract block slots but have not yet been actually consumed by deep learning training tasks (i.e., valid data), the system can skip the cumbersome disk reload process and respond to data access requests with the shortest path and fastest speed, minimizing access latency.

[0018] According to a preferred embodiment, the step of selecting a target data block from the data block group based on the fill rate for reloading includes: calculating the fill rate of multiple candidate data blocks in the data block group; and determining the candidate data block with the largest fill rate as the target data block for reloading.

[0019] This technical solution can dynamically determine the optimal data block loading path. This ensures that the data files loaded in the data block are consumed by the training task to the maximum extent, greatly reducing invalid data loading and redundant memory storage without compromising the random access characteristics of the data.

[0020] According to a preferred embodiment, in a distributed scenario involving multiple participating nodes, the method further includes: processing remote data requests based on an opportunistic prefetching mechanism; wherein, a globally random but fixed data access order is generated for each participating node; when a participating node, as a requesting node, initiates an access request to another participating node, which is the owner node of the target data file, it attaches its current position index in its local access sequence; the owner node infers a prefetching window based on the current position index attached to the requesting node, and when the target slot of the owner node has data and the corresponding abstract block slot of the requesting node is empty, it sends the data file in the prefetching window to the requesting node in advance.

[0021] This technical solution effectively reduces cross-node communication bandwidth overhead and remote I / O access latency. This mechanism is particularly suitable for distributed deep learning training scenarios where the amount of training data far exceeds the memory capacity of a single node.

[0022] According to a preferred embodiment, the step of mapping data blocks in the same data block group to the same abstract block includes: each abstract block is responsible for maintaining the starting address of the mapped data block and the address offset range of each data file in the data block, so as to support random access to data files of variable length.

[0023] This technical solution successfully supports random access to data files of variable length. Because data files within a single data block can vary in size, maintaining the address offset range completely solves the problem of accurately locating the size of heterogeneous files under block-level management.

[0024] According to a preferred embodiment, when a requesting node receives a data file sent by the owner node, it processes the data according to the priority of on-demand response first, followed by prefetched data; if the abstract block slot mapped to the received prefetched data is currently occupied or is valid data, the requesting node discards the prefetched data.

[0025] This prioritization and conflict resolution mechanism ensures that prefetched data can be successfully received and cached. Timely discarding of prefetched data that could cause conflicts strictly guarantees memory consistency and training accuracy.

[0026] The present invention provides, from a second aspect, a comprehensive memory management system based on deep learning training. The system includes a preprocessing unit, a mapping unit, and a selection unit. The preprocessing unit divides the dataset into multiple data blocks. The mapping unit groups the multiple data blocks into several data block groups and maps data blocks within the same data block group to the same abstract block, with multiple abstract blocks constituting a logical abstract memory space. The selection unit, upon receiving a data access request, determines the abstract block slot corresponding to the requested data file. If the abstract block slot is empty or invalid, it selects the data block with the highest fill rate from its data block group for reloading and fills the empty slot in the abstract block with the reloaded content. It also marks the actually consumed data file as invalid and releases it from the abstract memory space.

[0027] While maintaining fully random access semantics for training data, this system achieves block-level batch data loading and memory reuse. Through efficient collaboration among the units in the system, it fundamentally optimizes I / O scheduling performance in large-scale deep learning training tasks.

[0028] According to a preferred embodiment, the system further includes a network interface component and a prefetching check component; the network interface component is configured to communicate with multiple external computing nodes in a distributed training scenario to build a logically consistent global abstract memory view; the prefetching check component is connected to the network interface component and the selection unit to maintain a global random access trajectory and perform opportunistic prefetching decisions based on the current access location index and the prefetching window; when an access request is received, the prefetching check component predicts future access items according to the prefetching window, and generates a remote data scheduling instruction only when the target slot of the sending node has data and the corresponding abstract block slot of the requesting node is idle, and sends the remote data scheduling instruction to the selection unit to perform cross-node data exchange.

[0029] This invention constructs a logically consistent global abstract block address space and performs fine-grained control over inter-node memory access based on an ownership allocation strategy. This optimizes cross-node I / O performance without compromising randomness, ultimately achieving a dual improvement in memory efficiency and training performance.

[0030] The present invention provides an electronic device from a third aspect, including a processor and a terminal, wherein the terminal interacts with the processor, and the processor is configured to: partition a dataset to form multiple data blocks; establish a data mapping mechanism based on an abstract memory space to support efficient memory management and random access; and implement a data access protocol based on opportunistic block selection and slot filling.

[0031] The present invention provides a processor from a fourth aspect, the processor being configured to: partition a dataset to form multiple data blocks; establish a data mapping mechanism based on an abstract memory space to support efficient memory management and random access; and implement a data access protocol based on opportunistic block selection and slot filling.

[0032] The present invention provides a storage medium in a fifth aspect for storing the encoded program of a comprehensive memory management method based on deep learning training.

[0033] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art: (1) This invention proposes a memory management method based on the mapping of abstract blocks and block groups. This method can achieve block-level batch data loading and memory reuse while maintaining the completely random access semantics of training data. It effectively improves the memory access bandwidth utilization through a dynamic data block scheduling strategy. This mechanism makes full use of the "no fixed order" characteristic of data access during deep learning training, avoiding the problem of traditional cache replacement strategies failing in random access scenarios, thereby significantly improving the loading efficiency of training data and the system I / O throughput.

[0034] (2) This invention further provides an opportunistic block selection and abstract block slot filling algorithm, which dynamically determines the optimal data block loading path based on the already filled bits in the abstract block and the current memory state of the data block group, ensuring that the data files loaded in the data block are consumed by the training task to the maximum extent. This technical solution greatly reduces invalid data loading and redundant memory storage without destroying the random access characteristics of data, and fundamentally optimizes the I / O scheduling performance in large-scale deep learning training tasks.

[0035] (3) This invention proposes a scalable distributed abstract memory management mechanism, constructs a logically consistent global abstract block address space, and performs fine-grained control over memory access between nodes based on an ownership allocation strategy. Combined with remote opportunistic prefetching and dynamic memory release mechanisms, it effectively reduces cross-node communication bandwidth overhead and remote I / O access latency, and is particularly suitable for distributed deep learning training scenarios where the amount of training data far exceeds the memory capacity of a single node, achieving a dual improvement in memory efficiency and training performance. Attached Figure Description

[0036] Figure 1 This is a flowchart of the comprehensive memory management method for deep learning training based on batch random access provided by the present invention; Figure 2 This is a schematic diagram of the data block-abstract memory mapping mechanism provided by the present invention; Figure 3 This is a schematic diagram of the local data access protocol provided by the present invention; Figure 4 This is a schematic diagram of the distributed data access protocol provided by the present invention; Figure 5 This is a schematic diagram illustrating the experimental results of training efficiency on LibriSpeech using 1-5 A10 nodes, as provided by the present invention. Figure 6 This is a schematic diagram illustrating the experimental results of training efficiency on ImageNet-1k using three A10 nodes, as provided by this invention. Figure 7 This is a schematic diagram illustrating the experimental results of training efficiency on ImageNet-1k using 5 A10 nodes, as provided by this invention. Figure 8 This is a schematic diagram illustrating the experimental results of training efficiency on ImageNet-1k on a single P100 node, as provided by this invention. Figure 9 This is a schematic diagram illustrating the experimental results of training efficiency on ImageNet-1k using 3 P100 nodes, as provided by this invention. Figure 10 This is a schematic diagram illustrating the experimental results of training efficiency on ImageNet-21k using 2-3 A100 nodes, as provided by this invention. Figure 11 This is a schematic diagram of the module connections of the integrated memory management system for deep learning training based on batch random access according to the present invention.

[0037] List of reference numerals 100: Processor; 110: Preprocessing unit; 120: Mapping unit; 130: Selection unit; 140: Prefetch check component; 150: Storage unit; 200: Network interface component; 300: Compute node. Detailed Implementation

[0038] The following is a detailed explanation with reference to the accompanying drawings.

[0039] This invention provides explanations and descriptions of core terms and concepts.

[0040] Dataset: Refers to the complete original set of data required for deep learning training, typically consisting of massive data instances, reaching terabytes in size. In this invention, the dataset is the logical source for chunked processing and random access.

[0041] A chunk is a fixed-size logical storage unit formed by dividing a dataset (or its random index sequence). It is the basic unit for loading batch data and improving the efficiency of single I / O operations.

[0042] Data Index Sequence: A sequence generated by randomly shuffling the complete dataset. It is used to represent the order in which data is accessed during training, ensuring that the access process has sufficient random semantics.

[0043] Abstract Memory Space: A logical-level storage space composed of multiple abstract blocks, built on top of physical memory resources. It aims to shield the underlying physical distribution through logical mapping and support efficient random access and reuse.

[0044] Abstract Chunk: A logical mapping unit to which multiple data blocks within the same data block group are mapped. Each abstract chunk is responsible for maintaining the starting address of the mapped block and the address offset of the internal data file, supporting random access to variable-length data.

[0045] Abstract block slots are specific storage locations or mapping items within an abstract block. Their status (such as empty, valid, or invalid) directly determines the logical judgment of data loading and filling.

[0046] A chunk group is a collection of multiple data blocks grouped together based on the number of abstract block slots. It is the basic organizational unit for establishing data mapping relationships, with one chunk group corresponding to one abstract block.

[0047] Data mapping mechanism: A technical means based on the establishment of abstract memory space, used to establish the correspondence between physical data blocks and logical abstract blocks, thereby achieving efficient allocation of memory resources while maintaining random access characteristics.

[0048] Opportunistic Prefetching Mechanism: In distributed training scenarios, the owner node infers the future trajectory (prefetch window) of requesting nodes based on the global random access order. When conditions such as the target slot being available are met, data is sent in advance to reduce remote I / O latency.

[0049] Ownership Allocation Strategy: In distributed abstract memory management, this refers to the fine-grained control rules for access permissions between nodes, ensuring that each abstract block is managed by a specific owner node for its local disk mapping, thus maintaining the consistency of global logic.

[0050] Data File: The basic physical unit of a dataset, typically existing as a large number of small files. In the chunking logic, each data block contains several such files and is the target object actually consumed by the training task.

[0051] Valid Data: refers to the state of data files that have been filled into abstract block slots but have not yet been actually consumed by deep learning training tasks.

[0052] Prefetch Window: A subset of remote data requests that the requesting node is likely to initiate within a certain period of time, inferred by the system based on a globally fixed data access trajectory.

[0053] Example 1 The random access nature of deep learning training data severely limits the optimization potential of memory management solutions. Existing methods generally employ static caching strategies, which fix a portion of data in memory to accelerate access. However, uncached data still needs to be retrieved via I / O, resulting in significant latency. Under this strategy, the proportion of data access that can be accelerated is entirely limited by memory capacity, and the physical scalability of memory clearly cannot keep up with the exponential growth of training data volume. It is worth noting that deep learning training data is typically stored in the form of massive small data files, and their random access generates I / O overhead far exceeding that of large data files. To address this, large analysis systems often reorganize data into larger block structures and implement batch reads to reduce overhead. However, this approach has a fundamental flaw in deep learning scenarios: due to the random access nature of the training process, when loading data blocks from external storage, the system cannot guarantee that most of the data contained within the block will be effectively utilized before memory replacement occurs. This uncertainty in data block utilization makes it difficult for batch read mechanisms to achieve the expected optimization effects in deep learning training.

[0054] In view of the objective shortcomings of the prior art, which only discloses a simple prefetch list eviction mechanism and does not solve the problem of local failure when loading variable-length data at the block level, in order to effectively improve the utilization of memory access bandwidth and support accurate jump addressing of variable-length data files, this invention provides a comprehensive memory management method and system for deep learning training based on batch random access.

[0055] Preferably, the preprocessing unit 110 is used to partition the complete dataset to form multiple fixed-size data blocks and persist them to the external storage unit 150, providing a basis for batch data loading. The mapping unit 120 is used to group multiple data blocks to form several data block groups, and map multiple data blocks in the same data block group to the same abstract block in the abstract memory space. The selection unit 130, located at the core scheduling level, is used to dynamically determine the validity of the corresponding abstract block slot and perform slot filling operation when a data access request is received. Specifically, when the distributed mode is not enabled, the selection unit 130 directly performs addressing and slot determination based on the local abstract memory; when cross-node reading in the distributed mode is enabled, the selection unit 130 of the current requesting node routes the request to the owner node through the network interface component 200, and the selection unit 130 of the owner node performs the addressing and mapping determination. In the case of cross-node scheduling based on opportunistic prefetching, the prefetching check component 140 predicts future access items based on the prefetching window. Only when there is valid data in the target slot of the sending node and the corresponding abstract block slot of the requesting node is idle, a remote data scheduling instruction is generated and handed over to the selection unit 130 to coordinate with the network interface component 200 to execute the on-demand response-first cross-node data exchange and loading relationship.

[0056] The system of the present invention includes a processor 100 for executing the comprehensive memory management method based on deep learning training of the present invention. The present invention also provides an electronic device including a terminal and the processor 100 connected thereto. The terminal and the processor 100 interact. The present invention also provides a storage medium for storing the encoded program of the comprehensive memory management method based on deep learning training. The present invention also provides a processor 100 for running the encoded program of the method of the present invention. The processor 100 is configured to: partition a dataset to form multiple data blocks; establish a data mapping mechanism based on an abstract memory space to support efficient memory management and random access; and implement a data access protocol based on opportunistic block selection and abstract block slot filling.

[0057] Preferably, the physical hardware of the processor 100 can be a central processing unit (CPU), a graphics processing unit (GPU), a dedicated application processor (ASIC / accelerator), a microcontroller (MCU), a system-on-a-chip (SoC), etc. The storage medium can be a semiconductor storage medium, a magnetic storage medium, an optical storage medium, and emerging storage media. Semiconductor storage media include NAND flash memory (SSD) and DRAM (Dynamic Random Access Memory). Magnetic storage media include hard disk drives (HDDs) and magnetic tape. Optical storage media include Blu-ray discs (BDXL). Emerging storage media include, for example, 3D XPoint (Optane memory) and resistive random access memory (ReRAM).

[0058] Preferably, such as Figure 11 As shown, the processor 100 includes a preprocessing unit 110, a mapping unit 120, and a selection unit 130. The preprocessing unit 110, mapping unit 120, and selection unit 130 cooperate with each other through a data path. Furthermore, in a distributed scenario, the system of this invention also includes a network interface component 200 for connecting multiple computing nodes 300 to construct a logically consistent Global Abstract Memory View (GAMP). A GAMP refers to a unified, cross-node logical memory addressing registry (or address space) constructed at the cluster level by the network interface component 200 in a distributed deep learning training scenario, after aggregating cross-node communication data and ownership information. Under the GAMP, all computing nodes 300 in the cluster share the exact same logical abstract blocks and abstract block slot partitioning structure, completely breaking the island limitation of physical memory on a single server.

[0059] The processor 100 also includes a prefetch check component 140. The prefetch check component 140 is connected to the network interface component 200 and the selection unit 130, and is used to maintain a global access trajectory and perform remote data scheduling based on opportunistic prefetching to optimize cross-node I / O performance without compromising randomness.

[0060] The preprocessing unit 110 is connected to the storage unit 150 and is used to randomly shuffle the complete dataset to generate a randomized data index sequence, and partition the dataset based on the user-specified data block size to form multiple data blocks stored in the storage medium.

[0061] The mapping unit 120 is connected to the preprocessing unit 110 and the storage unit 150, and is used to group and map randomized data index sequences and partition information to construct a data mapping mechanism based on abstract memory space and corresponding abstract block slots. The mapping unit 120 establishes a logical space containing multiple abstract memory slots according to the memory capacity and the average data size, and maps multiple data blocks in the same data block group to the corresponding abstract blocks.

[0062] As the core of the data access protocol execution, the selection unit 130, connected to the mapping unit 120, is used to determine the validity of abstract block slots based on random access requests. When a data access request is received, the selection unit 130 determines the abstract block slot corresponding to the requested data file. If the abstract block slot is empty or invalid, the selection unit 130 selects a target data block from its data block group based on the data block's fill rate, reloads it, and fills the empty slot in the abstract block with the reloaded content; the selection unit 130 marks the actually consumed data file as invalid and releases it from the abstract memory space.

[0063] Furthermore, in a distributed scenario, the system of the present invention also includes a network interface component 200 for connecting multiple computing nodes 300 to construct a logically consistent global abstract memory view. The processor 100 also includes a prefetch check component 140. The prefetch check component 140 connects the network interface component 200 and the selection unit 130, and is used to maintain a global access trajectory and perform opportunistic prefetch-based remote data scheduling to optimize cross-node I / O performance without compromising randomness.

[0064] More preferably, the processor 100 is a system-on-a-chip (SoC) with application-specific integrated circuit (ASIC) design or a dedicated AI accelerator, and its internal preprocessing unit 110, mapping unit 120, selection unit 130 and prefetch checking component 140 are composed of the following physical hardware circuits and on-chip interconnect topologies.

[0065] The processor 100 is equipped with a high-speed internal data bus and control bus based on a high-bandwidth on-chip network. Each physical hardware entity is directly electrically coupled to the on-chip network to enable data output and reception. Each physical hardware entity outputs data load or release instructions to the external storage unit 150 through the physical pins of the memory controller integrated on the processor 100, thereby establishing a high-speed serial physical connection. At the same time, each physical hardware entity establishes a cross-node physical electrical signal transmission path with the external network interface component 200 through the PCIe bus controller, thereby outputting cross-node data access requests to the external computing node and receiving cross-node communication data and ownership information from the external computing network.

[0066] The physical entity of preprocessing unit 110 is a data preprocessing engine. The data preprocessing engine integrates DMA control circuitry and hardware random number generation circuitry. The physical entity of mapping unit 120 is a memory management unit. The memory management unit includes a static random access memory array and content-addressable memory, used to implement single-clock-cycle address retrieval. The physical entity of selection unit 130 is a scheduling coprocessor. The scheduling coprocessor includes an arithmetic logic unit for performing ratio calculations, a status register, and a hardwired finite state machine. The physical entity of prefetch checking component 140 is a hardware prefetch engine. The hardware prefetch engine includes a cache with hardwired memory access traces and a hardware comparator array.

[0067] The data output of the data preprocessing engine establishes a physical pipeline branch with the data input of the memory management unit via a hardware first-in-first-out queue. The hit output pin of the memory management unit is connected to the scheduling coprocessor via a dedicated control line. The hardware prefetch engine outputs a trigger strobe pulse to the scheduling coprocessor via an interrupt request line. The processor 100 establishes a serial communication link with the memory unit 150 and the network interface component 200 via on-chip integrated pins.

[0068] The data preprocessing engine partitions the dataset into multiple data blocks using internal circuitry. Specifically, a hardware random number generation circuit executes the partitioning algorithm and sends the generated data blocks to the DMA control circuit via the on-chip network or high-speed bus within the data preprocessing engine. Data within the DMA control circuit is read and accessed by the memory management unit. Simultaneously, the index sequence generated by the DMA control circuit is output to the memory management unit (CPU) through a hardware FIFO queue.

[0069] After receiving the sequence information, the memory management unit executes a grouping and mapping algorithm to establish a mapping table between data block groups and abstract blocks in memory. Multiple data blocks are grouped into several data block groups, and data blocks within the same data block group are mapped to the same abstract block through a physical mapping mechanism, thus forming a logical abstract memory space. The memory management unit then sends the mapping table to the scheduling coprocessor.

[0070] When the scheduling coprocessor receives a data access request, it queries the mapping table to automatically determine the abstract block slot corresponding to the requested data file. If the abstract block slot is determined to be empty or invalid, the scheduling coprocessor selects the data block with the highest fill rate from its data block group through hardware calculations and reloads it; that is, the scheduling coprocessor outputs a load instruction to the storage unit 150. The storage unit 150 then outputs the selected data content to the scheduling coprocessor. The scheduling coprocessor fills the empty abstract block slot with the data content; simultaneously, the scheduling coprocessor marks the consumed data file as invalid and releases the memory space.

[0071] In a distributed scenario, the network interface component 200 connects computing nodes to construct a global abstract memory view. In a cross-node scenario, the network interface component 200 outputs the received global abstract memory view to the hardware prefetch engine. The hardware prefetch engine maintains a global random access trajectory and performs opportunistic prefetch decisions. When an access request is received and the prefetch prediction conditions are met (i.e., the sending node's target slot has data and the requesting node's corresponding slot is free), the hardware prefetch engine generates a remote data scheduling instruction and outputs it to the scheduling coprocessor to complete the cross-node exchange.

[0072] The logical relationships between the various modules are as follows.

[0073] like Figure 11 As shown, the preprocessing unit 110 obtains the complete dataset from the storage unit 150. The preprocessing unit 110 performs random shuffling and partitioning of the complete dataset based on the data block size, thereby generating a randomized data index sequence and multiple data blocks. The preprocessing unit 110 then sends the multiple data blocks to the storage unit 150 for persistent storage, and sends the randomized data index sequence and partition information to the mapping unit 120.

[0074] like Figure 11 As shown, the mapping unit 120 receives a randomized data index sequence and partition information from the preprocessing unit 110. Based on the system memory capacity C and the average data size d, the mapping unit 120 groups and maps the randomized data index sequence and partition information to construct a data mapping mechanism based on abstract memory space and corresponding abstract block slots. During this process, the mapping unit 120 calculates the number of abstract block slots M. The formula for calculating the number of abstract block slots is: M = C / d.

[0075] Mapping unit 120 determines the size of the data block group based on the total number of data files N. The formula for calculating the size of the data block group is: n = N / M.

[0076] After the construction is completed, the mapping unit 120 sends the data mapping mechanism containing the abstract block slot structure information to the selection unit 130 as the logical basis for its execution of the data access protocol.

[0077] The network interface component 200 obtains cross-node communication data and ownership information from multiple computing nodes (cluster network) 300. The network interface component 200 performs logically consistent view construction processing on the cross-node communication data and ownership information to obtain a logically consistent global abstract memory view.

[0078] The network interface component 200 then sends the global abstract memory view to the prefetch check component 140.

[0079] If logical views and physical ownership are not distinguished, all compute nodes 300 will read shared storage out of order, leading to severe network congestion and distributed file system lock conflicts. This will result in training efficiency with multiple compute nodes 300 being lower than with a single compute node 300. Therefore, the network interface component 200 of this invention implements a unified addressing space in a multi-node cluster. The requester only needs to care about the abstract block slot, while the owner is responsible for the underlying physical I / O. This design perfectly integrates cross-node data exchange with single-machine batch loading logic, achieving scalability.

[0080] The prefetch check component 140 receives a logically consistent global abstract memory view from the network interface component 200. The prefetch check component 140 maintains a global random access trajectory and performs opportunistic prefetch decisions based on the current access location index and the prefetch window. Upon receiving an access request, the prefetch check component 140 predicts future access items based on the prefetch window and generates a remote data scheduling instruction only if the target slot of the sending node has data and the corresponding abstract block slot of the requesting node is empty, ensuring the effectiveness of remote data scheduling. Finally, the prefetch check component 140 sends the remote data scheduling instruction to the selection unit 130 to perform cross-node data exchange.

[0081] Selection unit 130 receives a data mapping mechanism based on abstract memory space from mapping unit 120 and a remote data scheduling instruction from prefetch checking component 140. Selection unit 130 performs validity determination and opportunistic block selection processing for abstract block slots based on random access requests. If the abstract block slot is determined to be empty or invalid, selection unit 130 calculates and selects the optimal data block according to the principle of maximizing fill rate and generates a data block reload instruction.

[0082] The fill rate is calculated as the ratio of the number of data file slots that can be filled by the main data block to the size of the data block.

[0083] The formula for calculating the fill rate is: .

[0084] In the above formula, `fillable_number` represents the number of data file slots that can be filled by the candidate data blocks, i.e., the fillable number; `chunk_size` represents the size of the data block; and `fill_rate` represents the fill rate. Here, the size of the data block refers to the total number of fixed file slots contained in that data block.

[0085] Subsequently, selection unit 130 sends a data block reload instruction to storage unit 150 to fill the abstract block slots, and ensures that all data files actually consumed by the training task are marked as invalid after use, so that they can be filled in the next round. Preferably, selection unit 130 clears the abstract block slots occupied by data files marked as invalid after data consumption and releases them from the abstract memory space.

[0086] like Figure 1 As shown, the method executed by the system of the present invention specifically includes the following steps.

[0087] S100: Dataset partitioning and preprocessing.

[0088] The preprocessing unit 110 divides the dataset required for deep learning training into multiple data chunks and sends them to the storage unit 150 for external storage, providing a basis for subsequent batch data loading.

[0089] S110: The preprocessing unit 110 randomly shuffles the complete dataset to generate a randomized data index sequence to ensure that the data access process has sufficient randomness.

[0090] If data is combined directly in its original order or without proper shuffling, simply to achieve batch reading, it would destroy the random sampling features required by the stochastic gradient descent (SGD) algorithm in deep learning training, leading to difficulty in model convergence or a significant drop in training accuracy. Therefore, before performing low-level I / O batch processing, the preprocessing unit 110 locks in and solidifies the global randomness semantics necessary for deep learning training from the logical source. This strategy of first globally shuffling and then solidifying the index allows the system to predict and manage seemingly disordered random access trajectories in subsequent operations.

[0091] S120: The preprocessing unit 110 partitions the randomized data index sequence based on the user-specified data block size to form multiple fixed-size data blocks, each containing several data files.

[0092] Preferably, the data block size needs to be set in a trade-off between I / O throughput and end-to-end execution efficiency: larger data blocks help reduce the frequency of I / O interactions (operations) and improve single-load efficiency, but may increase the number of data block loads within a training epoch. The data block size can be determined after a one-time system performance analysis of the target training platform.

[0093] Figure 2 This demonstrates a block partitioning process where the total sequence length is 18 returned data files (i.e., the 18 data files used in training), and the data block size is set to 3.

[0094] Deep learning datasets contain tens of millions of small files. Directly accessing these small files randomly would cause a surge in disk seek time, resulting in catastrophic I / O overhead. This is the fundamental problem in existing technologies where computing resources are waiting for data. Therefore, the preprocessing unit 110 logically aggregates massive amounts of discrete, heterogeneous small files (such as single images or single audio clips) into coarse-grained fixed data chunks. This effectively transforms inefficient random I / O of massive amounts of small files into efficient batch I / O of large files.

[0095] S200: Establishment of a mapping mechanism from data blocks to abstract memory.

[0096] After receiving the randomized data index sequence and partition information from the preprocessing unit 110, the mapping unit 120 groups the data blocks according to the number of abstract block slots, constructing a data mapping mechanism based on the abstract memory space to support efficient memory management and random access. Specifically, this includes: S210: The mapping unit 120 groups multiple data blocks according to the number of abstract memory slots to form several data block groups.

[0097] S220: Mapping unit 120 maps multiple data blocks in the same data block group to an abstract chunk, and the multiple abstract chunks together constitute a logical abstract memory space.

[0098] Specifically, the mapping unit 120 can roughly select the number of abstract block slots (M) by using the memory capacity (C) in bytes and the average size (d) of the data in the dataset, calculated as follows: .

[0099] Furthermore, the mapping unit 120 can determine the size (n) of the data block group based on the total number of data files (N) and the number of abstract block slots (M) in the dataset, using the following formula: .

[0100] If a fixed mapping ratio or memory partitioning method is hard-coded, the system is prone to memory overflow (OOM) or insufficient physical memory utilization when migrating to scenarios with limited video memory or exponentially increasing datasets, resulting in a lack of universality and practical engineering value. Therefore, the mapping unit 120 of this invention provides an adaptive capability for dynamic decoupling of software and hardware. The system can automatically scale the size of the logical abstraction space based on the actual memory resources (C) of the current physical machine and the characteristics (N, d) of the target dataset. This allows the same algorithm to be seamlessly deployed on servers with different configurations and tasks of different scales.

[0101] like Figure 2 The example shown has two data block groups, each containing three data blocks. The data blocks in group 0 and group 1 are mapped to abstract blocks, respectively. and The paper also describes one method for forming groups of data blocks: selecting n consecutive data blocks and mapping them to the same abstract block. Another method is to interleave consecutive data blocks to consecutive abstract blocks, similar to interleaving memory addresses to memory libraries. Interleaving does not provide additional parallelism because the data files returned from deep learning training are randomized. Therefore, the two methods are similar.

[0102] Existing static caching strategies directly bind data to physical memory. When the dataset is much larger than memory (e.g., 10PB of data to 256GB of memory), the physical cache is prone to invalidation, leading to a significant drop in cache hit rate and system performance degradation to frequent direct disk read / write (I / O). Mapping unit 120 constructs a decoupled logical intermediate layer between physical memory and external storage. By establishing a (many-to-one) mapping from data blocks to abstract blocks, mapping unit 120 can adaptively accommodate TB-level or even PB-level datasets based on the actual capacity of physical memory, breaking through the capacity limitations of physical memory modules.

[0103] Since the data files in a data block have different sizes, the mapping unit 120 controls each abstract block to maintain the starting address of the mapping block and the address offset range of each data file in the data block, so as to support random access to data files of variable length.

[0104] Traditional chunk-level read optimizations typically assume that the data within a chunk is a fixed-length record (e.g., in database systems). Without maintaining file-level offsets, when processing media files of heterogeneous sizes, the system cannot directly jump between memory blocks for addressing; it must instead perform an inefficient linear scan to find the specific file, completely negating the performance gains from batch loading. The mapping unit 120 of this invention successfully solves the problem of inconsistent file sizes (variable length) commonly found in deep learning datasets such as computer vision (images) and speech recognition (audio). By maintaining address offsets, the system can still maintain backward compatibility with fine-grained, precise file-level location and random access even after performing coarse-grained chunk-level loading.

[0105] To further clarify the hierarchical boundaries of the abstract memory space, the adaptive hardware / software decoupling mapping architecture of this invention employs a logically consistent abstract memory space mechanism. When the selection unit 130 determines that the corresponding abstract block slot is empty or invalid, it triggers dynamic scheduling calculation, selects the target data block with the highest fill rate from the data block group, reloads it, and precisely fills the empty slot. When the data file pointed to by the corresponding abstract block slot is currently valid data that has not been consumed, the selection unit 130 skips the cumbersome disk reloading steps at the lower level and directly returns the index of the valid data file with the shortest path, forming a high-speed, direct, low-latency data consumption relationship, enabling the abstract memory space to undergo completely random access reuse with physical data blocks without a fixed order.

[0106] S300: Receives data file access requests, based on a data access protocol using opportunistic block selection and abstract block slot filling.

[0107] For random data file access requests generated during deep learning training, selection unit 130 executes the following efficient data access protocol: S310: When the selection unit 130 receives an access request for a certain data file, it first determines the abstract block slot corresponding to the data file.

[0108] Specifically, such as Figure 1 As shown, when the selection unit 130 receives an access request for a certain data file, it first determines whether the distributed training mode is currently enabled (i.e., determines that distributed = ON).

[0109] If distributed training mode is not enabled (judged as N), then select unit 130 to determine the local abstract block slot corresponding to the requested data file.

[0110] If the distributed training mode is enabled (judged as Y), the selection unit 130 of the current node sends the data access request to the owner node of the target data file through the network interface component 200, and the selection unit 130 of the owner node determines the status information of the corresponding abstract block slot in the owner node to determine the abstract block slot corresponding to the requested data file.

[0111] S320: If selection unit 130 determines that the data file pointed to by the abstract block slot is currently valid data, then the corresponding data file index is returned directly, such as... Figure 1 As shown.

[0112] When a data file within a data block is requested, the system can choose to return any data file in the same abstract block slot within the same data block group. For example... Figure 3In the example, although the data file p is mapped to Chunk 1, Chunk 0 can be repopulated to handle the request. Then, the abstract block... Mapped to data blocks It returns a data file f. f and p are mapped to the same abstract block slot, which is the second data file in the group. Although the returned data file is different from the requested data file, the specific data file is not important for deep learning training, as long as the training consumes random data.

[0113] Existing operating system caching or general memory management logic must adhere to the strict consistency principle of "give A if A is needed." If this principle is upheld, under the extremely random requests of deep learning, it will inevitably lead to endless cache misses and extremely high disk I / O overhead. The selection unit 130 of this invention uses existing, unconsumed valid data within the abstract block slot to replace the currently requested data, satisfying the algorithm's randomness requirements with lower system overhead and effectively avoiding unnecessary disk reloading.

[0114] S330: If the selection unit 130 determines that the abstract block slot is empty or the data file is invalid, the fill rate is calculated, the candidate data block with the largest fill rate is selected from the block group, a data block reload instruction is generated and sent to the storage unit 150, the reloaded content is filled into the empty slot in the abstract block, and then the corresponding data file index is returned.

[0115] Preferably, the fill rate represents the ratio of newly filled abstract block slots after refilling and merging a data block. Let fillable_number represent the number of data file slots that can be filled using candidate data blocks, and chunk_size represent the size of the data block.

[0116] Preferably, after determining that the candidate data block has been reloaded, the number of data files that can be effectively filled into the corresponding free abstract block slot is taken as the fillable number; the ratio of the fillable number to the size of the candidate data block is taken as the fill rate.

[0117] Specifically, the formula for calculating the fill rate is: .

[0118] fill_rate represents the fill rate.

[0119] like Figure 3 In the example above, only the abstract block follows the preceding sequence of actions. The first abstract block slot contains a valid data file. Consider a new request e, which is invalid because it's in the second abstract block slot, so there's no way to return any data file. In this case, a new data block should be refilled and merged with c. At this point, there are two options to refill Chunk 1 or Chunk 2.

[0120] If Chunk 1 is refilled, since data file b in the data block has already been consumed, only data file p can be filled. Therefore, the fillable number is 1, and the fill rate is 33%. If Chunk 2 is refilled, since no data files in the data block have been consumed, the fillable number is 2, meaning that all abstract block slots except the first abstract block slot can be refilled, and the fill rate is 67%. Therefore, selection unit 130 determines that refilling Chunk 2 is the better option.

[0121] If blind or sequential loading is used, the system is highly likely to load a large amount of invalid data files that have already been used in training. This not only wastes valuable I / O bandwidth but also occupies scarce memory slots, causing the recently loaded data to trigger cache misses again due to its uselessness. To overcome this technical shortcoming, selection unit 130 provides a quantitative decision-making mechanism for which data block to load in pure random access. Maximum fill rate means that a single I / O operation can bring the most unconsumed valid data to the memory slot, thereby maximizing disk bandwidth utilization and perfectly meeting the needs of random access.

[0122] To address potential misunderstandings regarding storage structure scheduling and the dimensions of fill rate that may arise for those skilled in the art, embodiments of the present invention clarify the objective physical implementation logic of the processor. The processor 100 integrates a preprocessing unit 110, a mapping unit 120, and a selection unit 130, and interacts collaboratively with an external storage unit 150 via a data path. During decision-making scheduling, the selection unit 130 employs the aforementioned ratio calculation mechanism based on the number of elements of the same dimension. Specifically, the numerator of the calculation formula represents the number of valid files that can be actually filled through the candidate data block, while the denominator represents the total number of fixed file slots contained in the data block, i.e., the data block size. This yields the quantitative indicator fill rate used to evaluate the most efficient allocation path.

[0123] S340: Release used data files.

[0124] To resist the pollution caused by the overwriting of expired prefetched data and to maintain high-frequency reuse, selection unit 130 ensures that all data files actually consumed by the training task should be marked as invalid after use and free up abstract block slots from the abstract memory structure, thereby constructing a locally optimal replacement underlying mechanism that maintains the characteristics of fully random access for the next round of filling.

[0125] Existing systems mostly employ LRU (Least Recently Used) or LFU (Least Frequently Used) replacement strategies. In pure random access deep learning, data that has just been used will absolutely not be used again in the current epoch. Strategies like LRU will instead retain this data in memory, preventing truly needed, unaccessed data from entering memory and causing severe "cache pollution." Selection unit 130 establishes a self-invalidation mechanism. Within each epoch (training round), each piece of data is consumed only once. Once consumed, it is immediately released, freeing up space for loading subsequent new data and achieving high-frequency reuse of abstract block slots.

[0126] Figure 3 The example shows two such self-invalidating operations, one after returning f and the other after returning q.

[0127] Furthermore, this method can be effectively extended to distributed scenarios. In a distributed scenario containing multiple computing nodes 300, each computing node 300 maintains the same abstract memory space view through the network interface component 200. The training data is partitioned across all nodes and stored on the disk of each node. Each abstract block can only be mapped from the data block stored locally on the owner node.

[0128] Specifically, when a computing node 300 accesses a data file, it first sends a request to the owner. If the data file is valid, it responds with the current data file in the corresponding abstract block slot; otherwise, the owner node's selection unit 130 reads a data block from the disk in batches into the abstract block according to the local process and returns the corresponding data index file.

[0129] Furthermore, to improve the efficiency of cross-node data access, based on the aforementioned distributed abstract memory management mechanism, this invention introduces an opportunistic prefetching mechanism to reduce the response latency of remote data requests and improve overall training performance.

[0130] Specifically, in a distributed training scenario with multiple participating nodes, any participating node within the cluster can dynamically act as either a requesting node or an owner node based on the physical location of the target data. Before training, the prefetch check component 140 generates a globally random but fixed data access order for each participating node and replicates and distributes this order to all participating nodes, enabling each participating node to maintain the future data access trajectory of other nodes. During training, when a participating node (acting as a requesting node) initiates an access request to another participating node (acting as the owner node) where the target data file is located, it sends not only the index of the target data file but also its current position index in its local access sequence. Based on this, the owner node's prefetch check component 140 can infer the subset of data (i.e., the prefetch window) that the requesting node will request in the near future, thereby enabling the preloading of data that is about to be accessed.

[0131] Upon receiving a request, the owner node, on the one hand, returns the requested data file as needed; on the other hand, based on the predicted access items in the prefetch window, it pre-sends these data files to the requesting node if certain conditions are met. To ensure that the prefetched data can be successfully received and cached, the requesting node prioritizes processing based on the abstract block slot status during the reception process: on-demand response takes precedence, followed by prefetched data. The prerequisite for receiving prefetched data is that the abstract block slot it maps to is currently idle; if the abstract block slot is occupied or contains valid data, the prefetched data will be discarded to ensure memory consistency and training accuracy.

[0132] In complex distributed networks, unconditionally loading all prefetched data can easily lead to memory overflow and even cause valid data to be overwritten (i.e., cache pollution). This situation will cause the training model to read incorrect data sequences, severely reducing training accuracy and even causing the distributed training task to terminate abnormally. Therefore, the prefetch checking component 140 of this invention prioritizes data to ensure that urgently needed data is utilized as quickly as possible. This method (if the abstract block slot is occupied or contains valid data, the prefetched data will be discarded) prevents expired prefetched data caused by network latency from overwriting currently useful valid data, strictly guaranteeing the correct flow of training data.

[0133] Furthermore, to avoid redundant transmission due to the invisible state of the sending end, during the prefetch decision process, the prefetch check component 140 of the owner node will calculate the occupancy status of the target abstract block based on the access location index carried in the request and the prefetch window content, combined with the maintained global access sequence. Only when it is determined that the abstract block slot mapped by the prefetch data is idle will a remote data scheduling instruction be generated and handed over to the selection unit 130 to perform the sending operation, thereby improving the effectiveness of communication overhead.

[0134] Figure 4 This demonstrates a specific example in a distributed scenario. Based on... Figure 4 The mapping in the upper right corner shows data files i, g, and a mapped to abstract block slot M3; d, r, and n mapped to abstract block slot M4; and m, h, and k mapped to abstract block slot M5. Consider two nodes: node 0 and node 1, where node 0 is the requester, and the current request sequence is i, followed by r, m, k, and a. Assume all requests are mapped to the abstract block in node 1 on the right. The prefetch checklist in the lower right corner is the key component determining whether to perform prefetching on remote requests. It defines the state determination logic: prefetching will only be performed if and only if the sender's corresponding abstract block slot has data (state S is √) and the requester's corresponding bit is empty (state R is √). Assume a remote request is mapped to an abstract block slot. Only when the abstract block slot of the sending node (sender) is... There is data; the abstract block slot of the requesting node (the requester) is available. Prefetching only occurs if the node is empty. Starting with a miss on i, when a request from node 0 arrives at node 1, it finds that g is in the abstract block slot M3 mapped to i. Therefore, it can first return g as a response to node 0, and then attempt to prefetch from r, since r is mapped to an empty abstract block slot. Therefore, no data can be returned. This is the opportunistic prefetching mechanism: if the data is already in memory, the sender only provides the data, instead of loading it from disk. For m, which is mapped to M5, node 1 finds h in the same abstract block slot, that is, node 1 finds h in abstract block slot M5, and node 0's abstract block slot M5 is empty, corresponding to state S being √ and R being √. Therefore, h is sent to node 0 as prefetched data. For k and a, which are mapped to abstract block slots M5 and M3 respectively, since the data h in abstract block slot M5 has been extracted and allocated as prefetched data for m, and the data g in abstract block slot M3 has been extracted and allocated as response data for i, node 1 has no remaining valid data to provide in the corresponding abstract block slots, and the corresponding sender state S is X. Therefore, prefetching cannot be performed on k and a. After receiving the response, node 0, in its abstract block... Fill in g and h, and return g.

[0135] Consider the second request r. Since r was not prefetched, node 0 sends another missed request to node 1. At this point, because node 1 has already provided all remaining data to node 0, the abstract block is currently empty, and it reads a data block from disk in batches. First, n is included in the miss response on r, then a prefetch is attempted. Node 1 remembers that it is already m (mapped to the abstract block slot). The data was provided, and the inspection of k (which is also mapped to the abstract block slot) continued. Since the current access to r precedes m, this means that the prefetched m has not been consumed, and k cannot be prefetched even if data is available; otherwise, node 0 would have nowhere to store k's data. However, since a maps to the abstract block slot... At a different location, node 1 includes it in the prefetch. On the other hand, node 0 receives a and n and returns n.

[0136] Because data was prefetched in the first step, the next access to m will hit on node 0, and h will be returned. Next, node 0 sends a miss request on k because it did not prefetch in the first and second steps. Node 1 did not prefetch data for a; it had already prefetched it in the second step, and the data is not in memory anyway. In the final step, node 0 will hit a when accessing a, which is the previously prefetched data.

[0137] The experiment was conducted in three environments, as shown in Table 1.

[0138] Table 1: Experimental Environment Setup Data Table

[0139] For the A10 and P100 setups, each node is equipped with dual Intel Xeon E52682 v4 quad-core processors, and training data is stored on a 10 PB Apsara File Storage NAS. The A100 setup uses nodes equipped with dual AMD EPYC7402 24-core processors, and training data is stored on a 2 PB Lustre file system. All servers run on CentOS Linux 7 (Core) with CUDA Toolkit version 11.2, and PyTorch version 1.7.0 is used for the experiments.

[0140] The experiments covered two fields: Automatic Speech Recognition (ASR) and Computer Vision (CV), using three datasets of different sizes and six state-of-the-art models. For ASR, the Wav2Vec2.0 model was validated on the LibriSpeech dataset using the A10 setting. For CV, five models—SqueezeNet, MobileNetV3, ResNet50, DenseNet121, and VGG16—were trained on the ImageNet-1k and ImageNet-21k datasets.

[0141] The experiment was compared with three baselines. (1) PyTorch: Using the native PyTorch DataLoader, memory management is based on the operating system's caching strategy. (2) No-I / O: Based on the native PyTorch DataLoader, but replacing I / O reads with randomly simulated training data generation. This baseline does not incur any I / O cost and represents the upper limit of any optimization. (3) CoorDL: Caching fixed data in memory without changing memory, representing the state-of-the-art approach without sacrificing data randomness.

[0142] Training performance was evaluated based on the training time of each training epoch: the average training time over 5 epochs (excluding the first epoch) was calculated. All experiments utilized the maximum available memory possible. Experimental results are as follows: Figures 5 to 10 As shown.

[0143] like Figure 5 As shown, when training on a single node, this method outperforms PyTorch, achieving a 1.77x performance improvement, and surpasses CoorDL with a 1.64x speedup. When training on 3 or 5 nodes, distributed training enables cross-node data exchange, achieving a maximum speedup of 2.18x compared to PyTorch and 1.46x faster than CoorDL. Notably, when training on five A10 GPUs, Brand's performance is very close to the theoretical optimum (No I / O).

[0144] like Figure 6 As shown, this method achieves a speedup of up to 2.26 times compared to PyTorch, and in this case, it is 1.96 times faster than CoorDL.

[0145] like Figure 7 As shown, when training on five A10 nodes where approximately 40% of the data can be cached in global memory, this method outperforms PyTorch by up to 2.47 times and is 1.69 times faster than CoorDL.

[0146] like Figure 8 As shown, when training on a single P100 node without cross-node data sharing, this method outperforms PyTorch by up to 2.13 times and CoorDL by up to 1.45 times.

[0147] like Figure 9 As shown, when experiments were conducted on three P100 nodes of the entire dataset that can be cached in global memory, the performance of this method was up to 4.57 times that of PyTorch, and it also exceeded CoorDL by 1.32 times due to the implementation of the prefetch mechanism.

[0148] like Figure 10 As shown, when trained on 2 A100 nodes, this method is 2.37 times faster than PyTorch and 1.73 times faster than CoorDL; when trained on 3 A100 nodes, this method is 2.41 times faster than PyTorch and 1.70 times faster than CoorDL.

[0149] In the system of this invention, without an opportunistic prefetching mechanism, nodes in distributed training can only wait synchronously for network fetching when encountering a local miss, resulting in idle computing power such as GPUs. If blind prefetching is performed (without checking slot availability), prefetched data will overwrite currently useful data, causing memory conflicts or frequent data discarding, unnecessarily increasing network burden. The prefetching check component 140 of this invention utilizes the predictability provided by generating a random but fixed index sequence in the first step to asynchronously push data in advance in the distributed network, masking the latency of cross-node network transmission. Simultaneously, strictly limiting the target slot to be available ensures that the prefetched data always has a place to be placed.

[0150] It should be noted that the specific embodiments described above are exemplary. Those skilled in the art can devise various solutions inspired by the disclosure of this invention, and these solutions all fall within the scope of this invention and its protection. Those skilled in the art should understand that this specification and its accompanying drawings are illustrative and not intended to limit the scope of the claims. The scope of protection of this invention is defined by the claims and their equivalents. This specification contains multiple inventive concepts; terms such as "preferredly," "according to a preferred embodiment," or "optionally" indicate that the corresponding paragraph discloses an independent concept. The applicant reserves the right to file divisional applications based on each inventive concept.

Claims

1. A comprehensive memory management method based on deep learning training, characterized in that, The method includes: The dataset is divided into multiple data blocks; The multiple data blocks are grouped to form several data block groups, and the data blocks in the same data block group are mapped to the same abstract block. The multiple abstract blocks constitute a logical abstract memory space. When a data access request is received, the abstract block slot corresponding to the requested data file is determined. If the abstract block slot is empty or invalid, the target data block is selected from the data block group based on the data block fill rate and reloaded. The reloaded content is then filled into the empty slot in the abstract block. Mark the data files that have actually been consumed as invalid.

2. The method according to claim 1, characterized in that, The steps for partitioning the dataset include: The complete dataset is randomly shuffled to generate a randomized data index sequence; The data index sequence is partitioned based on a set data block size to form multiple data blocks of fixed size, each data block containing several data files.

3. The method according to claim 1 or 2, characterized in that, Prior to the step of grouping multiple data blocks to form several data block groups, the method further includes the following step: The number of abstract block slots in the abstract memory space is determined based on the memory capacity in bytes and the average size of the data in the dataset. The size of the data block group is determined based on the total number of data files in the dataset and the number of abstract block slots.

4. The method according to any one of claims 1 to 3, characterized in that, After the step of determining the abstract block slot corresponding to the data file to be accessed when a data access request is received, the method further includes the following step: If the data file pointed to by the abstract block slot is currently valid, the corresponding data file index is returned directly.

5. The method according to any one of claims 1 to 4, characterized in that, The step of selecting a target data block from the group of data blocks based on the fill rate for reloading includes: Calculate the fill rate of multiple candidate data blocks in the data block group; The candidate data block with the highest fill rate is selected as the target data block and reloaded.

6. The method according to any one of claims 1 to 5, characterized in that, In a distributed scenario involving multiple participating nodes, the method further includes: Remote data requests are processed based on an opportunistic prefetching mechanism; Specifically, a globally random but fixed data access order is generated for each participating node; When a participating node initiates an access request as a requesting node to another participating node that is the owner node of the target data file, it includes the current position index in its local access sequence. The owner node infers the prefetch window based on the current position index attached to the requesting node, and sends the data file in the prefetch window to the requesting node in advance when there is data in the target slot of the owner node and the corresponding abstract block slot of the requesting node is empty.

7. The method according to any one of claims 1 to 6, characterized in that, The step of mapping data blocks in the same data block group to the same abstract block includes: Each abstract block is responsible for maintaining the starting address of the mapped data block and the address offset range of each data file in the data block, in order to support random access to data files of variable length.

8. The method according to any one of claims 1 to 7, characterized in that, When a requesting node receives a data file sent by the owner node, it processes it according to the priority of on-demand response first, followed by prefetching data. If the abstract block slot mapped to the received prefetch data is currently occupied or contains valid data, the requesting node discards the prefetch data.

9. A comprehensive memory management system based on deep learning training, characterized in that, The system includes: The preprocessing unit (110) is used to divide the dataset into multiple data blocks; The mapping unit (120) is used to group the multiple data blocks to form several data block groups, and map the data blocks in the same data block group to the same abstract block, and the multiple abstract blocks constitute a logical abstract memory space. The selection unit (130) is used to determine the abstract block slot corresponding to the data file to be accessed when a data access request is received. If the abstract block slot is empty or invalid, the data block with the largest fill rate is selected from the data block group to which it belongs and reloaded, and the reloaded content is filled into the empty slot in the abstract block. The data file that has actually been consumed is marked as invalid and released from the abstract memory space.

10. The system according to claim 9, characterized in that, The system also includes: The network interface component (200) is configured to communicate with multiple external computing nodes in a distributed training scenario to build a logically consistent global abstract memory view; The prefetch check component (140), connected to the network interface component (200) and the selection unit (130), is used to maintain a global random access trajectory and perform opportunistic prefetch decisions based on the current access location index and the prefetch window. When an access request is received, the prefetch check component (140) predicts future access items according to the prefetch window. Only when there is data in the target slot of the sending node and the corresponding abstract block slot of the requesting node is idle, a remote data scheduling instruction is generated and sent to the selection unit (130) to perform cross-node data exchange.

Images