A stream data loading method and device, electronic equipment and storage medium

Abstract

The application provides a stream data loading method and device, electronic equipment and storage medium. The method comprises the following steps: obtaining a data file to be processed from a data lake; dynamically determining a dynamic block size of data blocks according to available memory resources of a current node and a computing device scale, and dividing the data file into a plurality of continuous data blocks according to the dynamic block size; establishing a sliding window buffer area, which is divided into a front-end area for storing processed data blocks, a middle area for storing data blocks being processed, and a tail-end area for storing data blocks to be preloaded; when the computing device starts processing the data blocks in the middle area, asynchronously loading a next data block of the data blocks into the tail-end area; and when a new data block is stored in the tail-end area, synchronously releasing the processed data blocks in the front-end area.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a streaming data loading method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, data lakes have become a core infrastructure for processing multimodal, ultra-large-scale datasets. Efficiently loading and managing massive amounts of data is a key challenge when training deep learning models, especially requiring a balance between memory resource consumption, computing device utilization, and the data supply needs of different application scenarios (such as training and inference).

[0003] Currently, mainstream technical solutions mainly include full preloading mode and batch loading strategy. Data lake components, represented by Apache Iceberg, adopt the full loading method, loading all data files into memory during the initialization phase to form a unified columnar storage table, while improved solutions alleviate memory pressure by loading fixed-size data blocks in batches.

[0004] However, these existing technologies suffer from inefficient coordination between memory and computing resources, which severely restricts the efficiency of ultra-large-scale data processing. Summary of the Invention

[0005] This application provides a streaming data loading method, apparatus, electronic device, and storage medium to solve the problem of low efficiency in the coordination of memory and computing resources in the prior art.

[0006] Firstly, this application provides a streaming data loading method, including:

[0007] Obtain the data file to be processed from the data lake;

[0008] Based on the available memory resources and computing device size of the current node, the dynamic block size of the data block is dynamically determined, and the data file is divided into several consecutive data blocks according to the dynamic block size.

[0009] A sliding window-style buffer is established, which is divided into: a front area for storing processed data blocks; a middle area for storing currently processed data blocks; and a back area for storing data blocks to be preloaded.

[0010] When the computing device begins processing the data block in the intermediate region, it asynchronously loads the next data block of the data block into the terminal region, and when the new data block is stored in the terminal region, it synchronously releases the data blocks that have been processed in the front-end region.

[0011] In one possible implementation, dynamically determining the dynamic block size of the data block based on the available memory resources and computing device size of the current node includes:

[0012] Monitor the remaining memory space in the available memory resources that is actually available for data loading;

[0013] Obtain the quantization parameters of the computing device size;

[0014] Calculate the initial block value based on the ratio between the remaining memory space and the quantization parameter;

[0015] The dynamic block size is obtained by applying size constraints to the initial block value.

[0016] In one possible implementation, establishing the sliding window buffer includes:

[0017] Obtain the average processing speed of the computing device for processing a single data block, and monitor the average loading speed of data blocks from storage devices to memory;

[0018] Calculate the initial window capacity based on the average processing speed and the average loading speed;

[0019] The initial window capacity is increased by a preset window capacity adjustment base to determine the target window capacity;

[0020] The sliding window buffer is initialized according to the target window capacity, so that the number of data blocks stored simultaneously in the sliding window buffer is consistent with the target window capacity.

[0021] In one possible implementation, the method further includes:

[0022] Monitor the memory usage of the current node in real time;

[0023] If the memory usage exceeds a preset high water level threshold, the target window capacity will be reduced to a first preset value.

[0024] If the memory usage rate is lower than a preset low water level threshold, the target window capacity will be increased to a second preset value.

[0025] In one possible implementation, the method further includes:

[0026] Real-time monitoring of the processing characteristic parameters of the data blocks, which are used to characterize data access pattern features;

[0027] The data supply mode is determined based on the processing characteristic parameters. The data supply mode includes: training mode: storing the data in blocks into a multi-level cache system, and / or inference mode: establishing a direct transmission channel from the data blocks to the computing device.

[0028] In one possible implementation, the method further includes:

[0029] In the training mode, two types of cache replicas are generated and maintained for the data blocks processed by the sliding window buffer: a memory cache replica: storing the data blocks whose access frequency is higher than a preset threshold; and a persistent cache replica: storing the complete data blocks in the distributed storage system.

[0030] In response to a data request from the training iterator, query the frequently accessed data portion in the memory cache; if the memory cache misses, query the complete data block in the persistent cache; if the persistent cache misses, retrieve the data block from the remote storage system; for the data block retrieved from the remote storage system, write the complete data block to the persistent cache; extract the data in the data block whose access frequency is higher than a preset threshold and write it to the memory cache.

[0031] For each data block, access statistics are collected periodically; a popularity value is calculated based on the access statistics; and a migration strategy for the data block between memory cache and persistent cache is determined based on the popularity value corresponding to the data block.

[0032] In one possible implementation, the method further includes:

[0033] The partitioned data blocks are divided using columnar storage memory mapping technology to establish a direct mapping channel from disk files to memory space;

[0034] The direct mapping channel enables the computing device to directly access the raw binary content of the data blocks;

[0035] Configure a shared memory pool mechanism to enable multiple data blocks to reuse the same memory mapping region, and dynamically manage the lifecycle of the memory mapping region based on reference counting, wherein: when a new data block is loaded, the reference count of the corresponding memory region is increased; when the data block processing is completed, the reference count of the corresponding memory region is decreased; when the reference count is detected to be zero, the memory mapping is automatically released and the resources are released.

[0036] Secondly, this application provides a streaming data loading apparatus, comprising:

[0037] The acquisition module is used to acquire data files to be processed from the data lake;

[0038] The determination module is used to dynamically determine the dynamic block size of the data block based on the available memory resources and computing device scale of the current node, and divide the data file into several consecutive data blocks according to the dynamic block size;

[0039] A module is established to create a sliding window-style buffer, which is divided into: a front area for storing processed data blocks; a middle area for storing currently processed data blocks; and a rear area for storing data blocks to be preloaded.

[0040] The processing module is configured to asynchronously load the next data block of the data block into the end region when the computing device starts processing the data block of the intermediate region, and synchronously release the data blocks that have been processed in the front-end region when the new data block is stored in the end region.

[0041] In one possible implementation, the determining module is specifically used for:

[0042] Monitor the remaining memory space in the available memory resources that is actually available for data loading;

[0043] Obtain the quantization parameters of the computing device size;

[0044] Calculate the initial block value based on the ratio between the remaining memory space and the quantization parameter;

[0045] The dynamic block size is obtained by applying size constraints to the initial block value.

[0046] In one possible implementation, the establishment module is specifically used for:

[0047] Obtain the average processing speed of the computing device for processing a single data block, and monitor the average loading speed of data blocks from storage devices to memory;

[0048] Calculate the initial window capacity based on the average processing speed and the average loading speed;

[0049] The initial window capacity is increased by a preset window capacity adjustment base to determine the target window capacity;

[0050] The sliding window buffer is initialized according to the target window capacity, so that the number of data blocks stored simultaneously in the sliding window buffer is consistent with the target window capacity.

[0051] In one possible implementation, the establishment module is further configured to:

[0052] Monitor the memory usage of the current node in real time;

[0053] If the memory usage exceeds a preset high water level threshold, the target window capacity will be reduced to a first preset value.

[0054] If the memory usage rate is lower than a preset low water level threshold, the target window capacity will be increased to a second preset value.

[0055] In one possible implementation, the device further includes a monitoring module for:

[0056] Real-time monitoring of the processing characteristic parameters of the data blocks, which are used to characterize data access pattern features;

[0057] The data supply mode is determined based on the processing characteristic parameters. The data supply mode includes: training mode: storing the data in blocks into a multi-level cache system, and / or inference mode: establishing a direct transmission channel from the data blocks to the computing device.

[0058] In one possible implementation, the device further includes a maintenance module for:

[0059] In the training mode, two types of cache replicas are generated and maintained for the data blocks processed by the sliding window buffer: a memory cache replica: storing the data blocks whose access frequency is higher than a preset threshold; and a persistent cache replica: storing the complete data blocks in the distributed storage system.

[0060] In response to a data request from the training iterator, query the frequently accessed data portion in the memory cache; if the memory cache misses, query the complete data block in the persistent cache; if the persistent cache misses, retrieve the data block from the remote storage system; for the data block retrieved from the remote storage system, write the complete data block to the persistent cache; extract the data in the data block whose access frequency is higher than a preset threshold and write it to the memory cache.

[0061] For each data block, access statistics are collected periodically; a popularity value is calculated based on the access statistics; and a migration strategy for the data block between memory cache and persistent cache is determined based on the popularity value corresponding to the data block.

[0062] In one possible implementation, the device further includes an access module for:

[0063] The partitioned data blocks are divided using columnar storage memory mapping technology to establish a direct mapping channel from disk files to memory space;

[0064] The direct mapping channel enables the computing device to directly access the raw binary content of the data blocks;

[0065] Configure a shared memory pool mechanism to enable multiple data blocks to reuse the same memory mapping region, and dynamically manage the lifecycle of the memory mapping region based on reference counting, wherein: when a new data block is loaded, the reference count of the corresponding memory region is increased; when the data block processing is completed, the reference count of the corresponding memory region is decreased; when the reference count is detected to be zero, the memory mapping is automatically released and the resources are released.

[0066] Thirdly, this application provides an apparatus comprising: a processor and a memory, the processor being configured to execute a streaming data loading program stored in the memory to implement the streaming data loading method described in any one of the first aspects.

[0067] Fourthly, this application provides a storage medium storing one or more programs that can be executed by one or more processors to implement the streaming data loading method described in any one aspect.

[0068] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application adjusts the data block size according to real-time memory resources and computing power through a dynamic block strategy, which avoids the memory pressure of full loading and ensures the continuous data supply of computing devices through size adaptation; the three-region (front / middle / end) design of the sliding window buffer combined with the asynchronous preloading mechanism enables data processing and data loading to form a pipeline in parallel, completely eliminating the idle waiting period of computing devices; thereby achieving the coordinated optimization of controllable memory usage and continuous saturation of computing resources, so that the system can control memory usage and maintain the continuous saturation of computing devices when processing ultra-large-scale data. Attached Figure Description

[0069] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0070] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0072] Figure 1A flowchart illustrating an embodiment of a streaming data loading method provided in this application;

[0073] Figure 2 A flowchart illustrating an embodiment of another streaming data loading method provided in this application;

[0074] Figure 3 A schematic diagram illustrating a data query process in a training mode provided in this application embodiment;

[0075] Figure 4 This application provides a schematic diagram of the entire process of streaming data loading in an embodiment.

[0076] Figure 5 A block diagram illustrating an embodiment of a streaming data loading device provided in this application;

[0077] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0079] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0080] To address the technical problem of low coordination efficiency between memory and computing resources in existing technologies, this application provides a streaming data loading method that can achieve coordinated optimization of controllable memory usage and continuous saturation of computing resources, enabling the system to control memory usage while maintaining continuous saturation of computing devices when processing ultra-large-scale data.

[0081] Figure 1 This is a flowchart illustrating an embodiment of a streaming data loading method provided in this application. Figure 1 As shown, the method includes the following steps:

[0082] Step 101: Obtain the data file to be processed from the data lake.

[0083] A data lake is a large-scale distributed storage system that supports the storage of multimodal data (including structured, semi-structured, and unstructured data). Its core feature is the unified management of various data files stored in the underlying file system (such as HDFS (Hadoop Distributed File System) or Amazon Simple Storage Service (S3)) through a metadata layer.

[0084] Data files to be processed refer to the collection of data that needs to be loaded into a computing system for machine learning training or inference tasks. These files typically use columnar storage formats (such as Apache Parquet or ORC (Optimized Row Columnar)) to optimize read performance.

[0085] In this embodiment, the system accesses the target data file through a standardized data lake interface (such as the API (Application Programming Interface) provided by Apache Iceberg or Delta Lake). During the access process, the system prioritizes loading the metadata information of the data file (including partition structure, statistics, etc.), and determines the subsequent data block partitioning strategy based on this metadata. This step establishes the entry point for the data loading process, providing raw data input for subsequent dynamic block processing.

[0086] Step 102: Based on the available memory resources and computing device scale of the current node, dynamically determine the dynamic block size of the data block, and divide the data file into several continuous data blocks according to the dynamic block size.

[0087] A current node refers to a physical or virtual computing unit that executes data processing tasks. Its available memory resources include physical memory not occupied by system processes and off-heap memory. The scale of the computing device characterizes the node's parallel processing capability, such as the number of GPUs (Graphics Processing Units) or the number of CPU cores (Central Processing Units). Dynamic block size is the size of the data block determined based on real-time resource conditions.

[0088] In one embodiment, dynamically determining the dynamic block size of data blocks based on the available memory resources of the current node and the scale of the computing device may include the following steps: monitoring the remaining memory space actually available for data loading in the available memory resources; obtaining the quantization parameter of the computing device scale; calculating the initial block value based on the ratio between the remaining memory space and the quantization parameter; and applying size constraints to the initial block value to obtain the dynamic block size.

[0089] Remaining memory space refers to the actual available memory capacity in the current computing node that can be used for data loading, excluding the memory reserved by the system and the memory occupied by other running processes.

[0090] In this implementation, the system first monitors the remaining available memory space in real time and obtains the quantization parameters of the computing device. Then, it calculates the initial block size using the formula "Initial block size = Remaining memory space / (Quantization parameter × Adjustment coefficient)," where the adjustment coefficient is an empirical value (e.g., 4). Finally, the final dynamic block size is obtained through size constraints. For example, if it is less than a preset lower limit (e.g., 256 megabytes), it is set as the lower limit to avoid memory fragmentation; if it is greater than a preset upper limit (e.g., 1024 megabytes), it is limited to the upper limit to ensure parallel processing efficiency. Through this scheme, the system achieves automatic adaptation of the data block size to the real-time resource status of the nodes, optimizing computational efficiency while ensuring memory security.

[0091] In another embodiment, the method may further include the following steps: applying columnar storage memory mapping technology to the divided data blocks to establish a direct mapping channel from disk files to memory space; enabling the computing device to directly access the original binary content of the data blocks through the direct mapping channel; configuring a shared memory pool mechanism to enable multiple data blocks to reuse the same memory mapping region, and dynamically managing the lifecycle of the memory mapping region based on reference counting, wherein: when a new data block is loaded, the reference count of the corresponding memory region is increased; when the data block processing is completed, the reference count of the corresponding memory region is decreased; when the reference count is detected to be zero, the memory mapping is automatically released and resources are released.

[0092] This embodiment achieves efficient data loading through the Apache Arrow inter-process communication protocol (Arrow IPC protocol), specifically including: Memory mapping establishment: Mapping columnar storage files on disk (such as Apache Parquet / optimized row-column storage) to user space memory via the Arrow IPC protocol; the starting position of each data block is strictly aligned to the metadata boundary of the Arrow record batch; maintaining the integrity of the original columnar storage format and avoiding structural damage caused by data splitting. Zero-copy mechanism: The computing device directly accesses the original binary data through virtual address mapping; completely avoiding traditional data deserialization and memory copying operations; supporting cross-process / cross-device shared memory access. Resource management optimization: Multiple data blocks share the same physical memory mapping area; automatic lifecycle management based on reference counting: increasing the reference count of the corresponding memory area when loading a new block; decreasing the reference count after block processing is completed; automatically unmapping and releasing resources when the reference count reaches zero. This technical solution significantly improves data transmission efficiency (eliminating copy overhead), optimizes memory utilization (shared mapping area), and enhances system reliability (automatic resource reclamation) while ensuring data integrity, providing a high-performance, low-overhead memory access solution for ultra-large-scale data processing.

[0093] In another embodiment, the method may further include the following steps: establishing a metadata index for each data block, the metadata index including: the starting offset of the data block in the source file; the byte length of the data block; and the corresponding columnar storage format metadata boundary information; wherein the metadata index is stored and managed in a lightweight structure independent of the data block body.

[0094] Metadata indexes are structured information describing the physical storage characteristics of data chunks. They contain three key fields: the starting offset, indicating the starting byte position of the chunk in the source file; the byte length, recording the storage space occupied by the chunk; and columnar storage format metadata boundary information, ensuring that chunk splitting aligns with the inherent structure of the original data format (such as Parquet's Row Groups or ORC's Stripes). Lightweight structure refers to index storage schemes using efficient encoding formats such as JSON and binary.

[0095] In this embodiment, the system independently constructs a metadata index for each dynamically generated data block. This index is managed in a separate file or dedicated storage area, physically separated from the data block itself. During data access, the system prioritizes loading the lightweight metadata index, quickly locating the physical location and structural characteristics of the target block by parsing the index information, and then loading the actual data content as needed. The index construction process strictly follows the columnar storage format specification, ensuring that the block boundaries are always aligned with the metadata boundaries of the original data. This significantly improves block retrieval efficiency, enabling rapid location through the index and avoiding full file scanning.

[0096] Step 103: Establish a sliding window buffer, which is divided into: front area: for storing processed data blocks; middle area: for storing currently processed data blocks; and end area: for storing data blocks to be preloaded.

[0097] A sliding window buffer is a fixed-capacity circular queue structure used for pipelined data processing. Its core feature is dividing the storage space into three functional areas: a front area caches processed data blocks awaiting release, a middle area stores blocks currently being processed by the computing device, and a rear area stores pre-loaded blocks awaiting processing. Each area is marked by pointers to indicate its boundary position. The read pointer indicates the position of the currently processed data block, and the write pointer indicates the position of the new data block to be written. The movement of these pointers automatically connects the release of data in the front area with the loading of data in the rear area, forming a first-in, first-out (FIFO) data flow pipeline.

[0098] In one embodiment, establishing a sliding window buffer may include the following steps: obtaining the average processing speed of a computing device for processing a single data block, and monitoring the average loading speed of data blocks from storage devices to memory; calculating an initial window capacity based on the average processing speed and the average loading speed; adding a preset window capacity adjustment base to the initial window capacity to determine a target window capacity; and initializing the sliding window buffer according to the target window capacity, so that the number of data blocks simultaneously stored in the sliding window buffer is consistent with the target window capacity.

[0099] Average processing speed refers to the throughput (in MB / s) of a computing device (such as a GPU / CPU) processing a single data block. Average load speed reflects the transfer rate at which a storage device (such as an SSD / HDD) loads data blocks into memory. Window capacity adjustment base is a buffer margin (empirical value) set to cope with speed fluctuations.

[0100] In this embodiment, the system dynamically monitors the real-time processing power and data loading performance of the computing device. First, it calculates the initial window capacity (initial window capacity = average processing speed / average loading speed). Then, it increases the window capacity adjustment base (e.g., +1) to obtain the target window capacity. Finally, it initializes the sliding window buffer according to this capacity. This process ensures that the buffer capacity always matches the system's current processing power: when the computing speed is fast, the window is increased to maintain data supply; when the loading speed is slow, the window is reduced to prevent memory accumulation.

[0101] In another embodiment, the method may further include the following steps: real-time monitoring of the memory usage of the current node; if the memory usage exceeds a preset high watermark threshold, reducing the target window capacity to a first preset value; if the memory usage is below a preset low watermark threshold, increasing the target window capacity to a second preset value.

[0102] Memory utilization rate refers to the percentage of total memory capacity used in the current node; the preset high watermark threshold and preset low watermark threshold are pre-set memory warning lines (e.g., 80% and 50%), used to trigger window capacity adjustment; the first preset value and the second preset value represent the minimum and maximum safe values ​​of window capacity (e.g., 3 and 5 data blocks), respectively.

[0103] This embodiment, based on dynamic window adjustment, adds a real-time feedback mechanism based on memory pressure: the system continuously monitors the memory usage of nodes, and when it exceeds the high-water mark threshold, it immediately reduces the window size to a first preset value (e.g., 3 blocks) to prevent memory overflow; when it falls below the low-water mark threshold, it expands the window size to a second preset value (e.g., 5 blocks) to fully enhance data supply capacity. This dual threshold control ensures that the system always operates within a safe memory range.

[0104] Step 104: When the computing device starts processing the data block in the intermediate region, it asynchronously loads the next data block of the data block into the end region, and when the new data block is stored in the end region, it synchronously releases the data blocks that have been processed in the front-end region.

[0105] Asynchronous loading refers to data prefetching operations performed by a background thread independently of the main computation process; synchronous release means that memory resources are reclaimed immediately after the data blocks in the front-end region are processed. These two operations are completed collaboratively through the three functional areas of the sliding window buffer: the middle area (in computation), the end area (preloading), and the front-end area (to be released).

[0106] This embodiment automates the operation of the data processing pipeline: when the computing device begins processing the current data block in the intermediate region, the system immediately triggers the asynchronous loading of the next data block to the end region; when the new block is loaded, the old blocks that have been processed in the front region are released synchronously. This mechanism maintains the continuous movement of the sliding window through a double-buffering strategy (processing and loading in parallel), ensuring that the computing device always has available data while keeping memory usage constant.

[0107] The technical solution provided in this application adjusts the data block size according to real-time memory resources and computing power through a dynamic block strategy. This avoids the memory pressure of full loading and ensures a continuous data supply to the computing device through size adaptation. The three-region (front / middle / end) design of the sliding window buffer, combined with an asynchronous preloading mechanism, enables data processing and data loading to form a pipeline in parallel, completely eliminating the idle waiting period of the computing device. This achieves coordinated optimization of controllable memory usage and continuous saturation of computing resources, enabling the system to control memory usage and maintain a continuous saturation of computing device operation when processing ultra-large-scale data.

[0108] Figure 2 A flowchart illustrating another embodiment of the streaming data loading method provided in this application. Figure 2 The process shown is in Figure 1 Based on the illustrated process, the following steps are included:

[0109] Step 201: Monitor the processing characteristic parameters of the data blocks in real time. The processing characteristic parameters are used to characterize the data access pattern characteristics.

[0110] Processing feature parameters are dynamic metrics used to quantify data chunk access patterns, such as data reuse frequency (the ratio of repeated access to the same chunk) and calculation iteration count (the number of rounds in which a chunk participates in training iterations). These parameters are collected in real time by monitoring the data access pipeline.

[0111] In this embodiment, the system continuously tracks the processing of each data block, records key feature parameters through a lightweight statistics module, and summarizes and analyzes them based on time windows (e.g., every 5 minutes). These parameters dynamically reflect the characteristics of the workload; for example, frequently reused blocks may belong to hot features, and blocks that have undergone multiple iterations are usually located in the key training sample set.

[0112] This step can provide a quantitative basis for selecting the data supply mode (training / inference).

[0113] Step 202: Determine the data supply mode based on the processing feature parameters. The data supply mode includes: training mode: storing the data in blocks into a multi-level cache system, and / or inference mode: establishing a direct transmission channel from the data blocks to the computing device.

[0114] Data supply mode refers to the differentiated data distribution strategy selected by the system according to data processing needs. It includes two types: training mode and inference mode. The training mode optimizes data reuse through a multi-level caching system (including memory caching and persistent storage layers), while the inference mode uses a direct transmission channel (bypassing the cache and transmitting directly to the computing device) to ensure real-time performance.

[0115] In this embodiment, the system dynamically selects the optimal supply mode based on real-time monitored processing characteristic parameters: when high-frequency reuse or multi-round iteration features are detected (typical training scenario), the training mode is enabled and the data is cached in blocks to a multi-level storage system; when data access presents a one-time or low-latency requirement (typical inference scenario), the system switches to direct connection transmission mode. Mode switching is achieved through a lightweight routing module.

[0116] Figure 2 The process described above involves several key steps. First, based on real-time data reuse frequency and computation iteration count, the system accurately identifies the workload type (training / inference), providing a quantitative basis for subsequent optimization decisions. Second, a multi-level caching system (memory + persistent storage) in training mode enables efficient reuse of hot data, improving data supply efficiency for training tasks. Finally, the direct transmission channel established in inference mode completely avoids cache access overhead, compressing end-to-end latency to milliseconds. This allows a single system architecture to intelligently adapt to the differentiated needs of training and inference, improving resource utilization while reducing deployment complexity, providing a high-throughput, low-latency unified data supply solution for heterogeneous computing scenarios.

[0117] In another embodiment of this application, the method may further include the following steps: In the training mode, for the data blocks processed by the sliding window buffer, generate and maintain two types of cache replicas: a memory cache replica: storing the portion of data in the data block whose access frequency is higher than a preset threshold; and a persistent cache replica: storing the complete data block in the distributed storage system; in response to a data request from the training iterator, query the frequently accessed data portion in the memory cache; if the memory cache misses, query the complete data block in the persistent cache; if the persistent cache misses, obtain the data block from the remote storage system; for the data block obtained from the remote storage system, write the complete data block into the persistent cache; extract the portion of data in the data block whose access frequency is higher than a preset threshold and write it into the memory cache; for each data block, periodically collect access statistics information of the data block; calculate a popularity value based on the access statistics information; and determine the migration strategy of the data block between the memory cache and the persistent cache based on the popularity value corresponding to the data block.

[0118] Memory cache replicas refer to frequently accessed data segments residing in volatile memory (such as DRAM (Dynamic Random Access Memory)); persistent cache replicas are complete data blocks stored in distributed storage systems (such as SSD clusters); and the popularity value is a data priority score derived by quantitatively analyzing access statistics (such as access frequency and timeliness).

[0119] In this scheme, two optimized replicas are first generated simultaneously for the processed data blocks: a memory cache replica stores frequently accessed data fragments to achieve fast response, and a persistent cache replica stores complete blocks to ensure data integrity. Secondly, a hierarchical query mechanism is established to check the memory cache, persistent cache, and remote storage in sequence. If a cache miss occurs, data backfilling and hotspot extraction are automatically triggered. Finally, a dynamic heat assessment system continuously analyzes the access characteristics of each block (such as access frequency and access timeliness), calculates a comprehensive heat value, and intelligently adjusts the distribution of data among cache levels accordingly.

[0120] The calculation of the popularity value includes: collecting the number of accesses and the most recent access timestamp of each data block; calculating the comprehensive popularity value according to the preset weighting coefficient; and implementing cache migration based on the ranking results of the popularity values.

[0121] In this scheme, the system periodically collects access records for each cached data block, normalizes the access count and the most recent access timestamp, and then linearly weights them according to preset weights to generate a comprehensive popularity value in the range of 0-1. For example: Popularity value = 0.7 × normalized access count + 0.3 × normalized time decay value. Based on the calculation results, all blocks are sorted, and the top 20% of high-popularity blocks are upgraded to in-memory cache, while the bottom 10% of low-popularity blocks are downgraded to remote storage, and the remaining blocks remain at the current cache level. In this way, a comprehensive popularity evaluation model based on access frequency and timeliness can make caching decisions more scientific and reasonable, avoiding the limitations of traditional single-indicator evaluation.

[0122] Figure 3 This application provides a schematic diagram of a data query process in a training mode, as illustrated in the embodiments of this application. Figure 3 As shown, in response to data requests from the training iterator, the system prioritizes querying high-frequency data segments in the memory cache; if a memory cache miss occurs, it queries the complete block in the persistent cache; if both caches miss, the original data is retrieved from the remote storage system. During the query process, a "query-backfill" linkage mechanism is implemented: for the data retrieved from the remote cache, a new cache copy is generated synchronously. Additionally, the system continuously monitors and analyzes access characteristics through a heatmap analyzer, calculates a dynamic heatmap score based on these characteristics, and performs intelligent migration: high-frequency hot data is preferentially retained in the memory cache; low-frequency cold data is removed from the memory cache; and expired data is evicted from the persistent cache.

[0123] Figure 2 The process shown achieves triple optimization through the above mechanism: performance optimization: a three-level acceleration system of "memory-persistence-remote" is built to achieve a gradient reduction in data access latency; resource optimization: memory utilization efficiency is improved by filtering hot data and storage load is reduced by automatic migration; management optimization: the fully automated workflow reduces the need for manual intervention and improves system stability.

[0124] In another embodiment of this application, the training mode further includes the following steps: triggering cache evaluation at the beginning and end of the training cycle of the training framework; calculating priority scores based on the access patterns of data blocks in consecutive training cycles; and promoting data blocks to higher-level caches when the priority scores simultaneously meet the following conditions: the score exceeds a preset promotion threshold; the access frequency ranking in the current training cycle enters a preset top percentile range; and the number of accesses shows an increasing trend in multiple consecutive training cycles.

[0125] Training cycle refers to the iterative process of completely traversing the training dataset once during machine learning training; cache evaluation refers to the process of analyzing the storage status and usage efficiency of data blocks in the cache system; priority score is a quantitative indicator calculated by comprehensively considering the access characteristics of data blocks over multiple training cycles, used to evaluate the importance level of the data; upgrade threshold is the minimum score requirement set for cache upgrade operations; top percentage range indicates the relative ranking of the access frequency of data blocks in the current training cycle.

[0126] This embodiment implements intelligent cache optimization management in training mode: the system continuously tracks the access behavior (including access frequency, period growth trend, etc.) of each data block in consecutive training cycles through an evaluation mechanism triggered at the beginning and end of the training cycle, and calculates a priority score based on multi-dimensional features. When a data block simultaneously meets three upgrade conditions—its score exceeds a preset threshold, its access frequency ranks high in the current cycle, and it shows a continuous growth trend—the system automatically migrates it to a higher-level cache (such as upgrading from SSD cache to memory cache). This decision-making mechanism based on multi-cycle behavior analysis can accurately identify truly valuable hot data.

[0127] The technical solution in this embodiment achieves several improvements. First, the evaluation mechanism triggered by the training cycle boundary ensures that caching decisions are synchronized with the training phase, avoiding evaluation interference with the normal training process. Second, multi-cycle access feature analysis effectively distinguishes between temporary and persistent hotspots, improving the accuracy of caching decisions. Finally, joint judgment based on multi-dimensional conditions prevents misjudgments caused by a single indicator, significantly improving the cache hit rate while reducing unnecessary cache migration overhead. These effects collectively optimize the data supply efficiency during the training process.

[0128] In another embodiment of this application, data migration can also be achieved in training mode through the following steps: when loading data, a metadata tag containing a unique identifier, loading time information and initial access status is established for each data block; a periodic event listener is registered in the training framework to track changes in the training period status in real time; a dynamic priority score is calculated by comprehensively analyzing the historical periodic access data recorded in the metadata tag; and intelligent migration of data blocks between different storage levels is performed based on the dynamic priority score result and the current status characteristics recorded in the metadata tag.

[0129] Metadata tags are data structures built for each data chunk, containing a unique identifier, loading timestamp, and initial state; periodic event listeners are triggering mechanisms implemented through the start / end hooks of the training cycle (epoch) of a training framework (such as PyTorch); dynamic priority scores are numerical values ​​calculated using a specific formula that incorporates cross-epoch access features; and storage tiers include storage media with different performance levels, such as memory and SSDs.

[0130] This scheme first establishes metadata tags containing unique identifiers, loading times, and initial states for each data block during data loading. Second, it tracks the training status in real-time using periodic event hooks within the training framework. Then, it calculates a dynamic priority score based on historical access data, with the scoring criteria strictly following these principles: priority is increased when the number of accesses in the current period enters the top 20%, when accesses increase for three consecutive periods, or when data is marked as warm-up data; and priority is downgraded when data is not accessed and the score is too low, when memory limits are exceeded and the score is below the median, or when the lifespan exceeds the limit. Finally, it performs intelligent migration of data blocks between storage tiers based on the scoring results. In this way, through period-aware dynamic cache scheduling, it accurately matches the data access characteristics of training and achieves optimal allocation of storage resources, significantly improving cache hit rate and training efficiency.

[0131] In another embodiment of this application, the training mode further includes the following steps: globally reordering the data blocks using a distributed hash algorithm; and maintaining a virtual sharding mapping table at the iterator interface layer to achieve cross-node data rearrangement.

[0132] Distributed hash algorithms (such as Rendezvous hashing) are used to perform consistent logical sharding of data blocks across the cluster, ensuring that the same blocks are stably allocated to fixed nodes in different training cycles; the virtual sharding mapping table is a lightweight data structure maintained at the iterator interface layer, which records the dynamic mapping relationship between logical shards and physical nodes, and supports the need for cross-node data rearrangement.

[0133] In this embodiment, during training mode, a globally unique logical shard number is first assigned to all data blocks using a distributed hash algorithm, achieving an initial out-of-order distribution. Subsequently, a virtual shard mapping table is dynamically maintained at the iterator interface layer, adjusting the actual storage location of the blocks in real time based on node load or network conditions, while ensuring that the logical shard number remains unchanged. This two-layer mapping mechanism maintains the randomness of data distribution (which is beneficial for model convergence) and provides flexible reordering capabilities (to adapt to cluster changes).

[0134] Figure 4 This application provides a schematic diagram of the entire process of streaming data loading, as shown in the embodiments. Figure 4 As shown, its complete workflow can be divided into two parallel paths:

[0135] Main data flow path:

[0136] Multimodal data lake: The system reads raw data in Parquet / ORC / JSON format from data lakes such as Iceberg.

[0137] Streaming segmented loading engine: Employs Arrow memory format for zero-copy loading; dynamic segmentation strategy (256MB-1GB); maintains a sliding window buffer of 3-5 data segments.

[0138] Memory buffer: Establish a double buffering mechanism (Buffer A / B) to achieve asynchronous preloading.

[0139] Mode selector: Determines the routing of data streams based on control center policy decisions.

[0140] Controlling the decision path:

[0141] Control Center: Real-time monitoring of memory water level (e.g., high water level 80% / low water level 50%); analysis of workload characteristics (e.g., number of iterations / data reuse frequency); output of caching strategy decisions.

[0142] Training mode branches: Alluxio cache cluster: implements epoch-aware caching strategy; establishes a two-level cache of memory and SSD; dynamically migrates hot data (e.g., the top 20% are upgraded to memory and the bottom 10% are downgraded); dynamic shuffler: implements global out-of-order processing based on the Rendezvous hash algorithm; GPU computing unit: receives processed data.

[0143] Inference mode branches: Straight-through pipeline: bypasses the caching system; Sequential processor: ensures data ordering; GPU computing unit: receives processed data.

[0144] This solution achieves zero-copy data transfer through a dynamic segmented loading engine and Arrow memory format, significantly improving data loading efficiency. It employs an epoch-aware intelligent caching strategy and the Alluxio distributed caching system, significantly reducing epoch switching time in training scenarios through a multi-level cache migration mechanism. The innovative dual-mode data supply interface intelligently switches between training and inference modes according to different task requirements. In training mode, a dynamic shuffler achieves global data disorder, while in inference mode, sequential processing is maintained, ensuring optimal data supply performance for various deep learning tasks. Based on a dynamic memory level adjustment mechanism and intelligent heat analysis algorithm, it achieves efficient utilization of storage resources while avoiding the risk of memory overflow.

[0145] Figure 5 This is a block diagram illustrating an embodiment of a streaming data loading device provided in this application. Figure 5 As shown, the device includes:

[0146] Module 51 is used to obtain the data file to be processed from the data lake;

[0147] The determination module 52 is used to dynamically determine the dynamic block size of the data block based on the available memory resources and computing device scale of the current node, and divide the data file into several consecutive data blocks according to the dynamic block size;

[0148] Module 53 is used to establish a sliding window-type buffer, which is divided into: a front area for storing processed data blocks; a middle area for storing currently processed data blocks; and a rear area for storing data blocks to be preloaded.

[0149] Processing module 54 is configured to asynchronously load the next data block of the data block into the end region when the computing device starts processing the data block of the intermediate region, and synchronously release the data block that has been processed in the front-end region when the new data block is stored in the end region.

[0150] In one possible implementation, the determining module is specifically used for:

[0151] Monitor the remaining memory space in the available memory resources that is actually available for data loading;

[0152] Obtain the quantization parameters of the computing device size;

[0153] Calculate the initial block value based on the ratio between the remaining memory space and the quantization parameter;

[0154] The dynamic block size is obtained by applying size constraints to the initial block value.

[0155] In one possible implementation, the establishment module is specifically used for:

[0156] Obtain the average processing speed of the computing device for processing a single data block, and monitor the average loading speed of data blocks from storage devices to memory;

[0157] Calculate the initial window capacity based on the average processing speed and the average loading speed;

[0158] The initial window capacity is increased by a preset window capacity adjustment base to determine the target window capacity;

[0159] The sliding window buffer is initialized according to the target window capacity, so that the number of data blocks stored simultaneously in the sliding window buffer is consistent with the target window capacity.

[0160] In one possible implementation, the establishment module is further configured to:

[0161] Monitor the memory usage of the current node in real time;

[0162] If the memory usage exceeds a preset high water level threshold, the target window capacity will be reduced to a first preset value.

[0163] If the memory usage rate is lower than a preset low water level threshold, the target window capacity will be increased to a second preset value.

[0164] In one possible implementation, the device further includes a monitoring module for:

[0165] Real-time monitoring of the processing characteristic parameters of the data blocks, which are used to characterize data access pattern features;

[0166] The data supply mode is determined based on the processing characteristic parameters. The data supply mode includes: training mode: storing the data in blocks into a multi-level cache system, and / or inference mode: establishing a direct transmission channel from the data blocks to the computing device.

[0167] In one possible implementation, the device further includes a maintenance module for:

[0168] In the training mode, two types of cache replicas are generated and maintained for the data blocks processed by the sliding window buffer: a memory cache replica: storing the data blocks whose access frequency is higher than a preset threshold; and a persistent cache replica: storing the complete data blocks in the distributed storage system.

[0169] In response to a data request from the training iterator, query the frequently accessed data portion in the memory cache; if the memory cache misses, query the complete data block in the persistent cache; if the persistent cache misses, retrieve the data block from the remote storage system; for the data block retrieved from the remote storage system, write the complete data block to the persistent cache; extract the data in the data block whose access frequency is higher than a preset threshold and write it to the memory cache.

[0170] For each data block, access statistics are collected periodically; a popularity value is calculated based on the access statistics; and a migration strategy for the data block between memory cache and persistent cache is determined based on the popularity value corresponding to the data block.

[0171] In one possible implementation, the device further includes an access module for:

[0172] The partitioned data blocks are divided using columnar storage memory mapping technology to establish a direct mapping channel from disk files to memory space;

[0173] The direct mapping channel enables the computing device to directly access the raw binary content of the data blocks;

[0174] Configure a shared memory pool mechanism to enable multiple data blocks to reuse the same memory mapping region, and dynamically manage the lifecycle of the memory mapping region based on reference counting, wherein: when a new data block is loaded, the reference count of the corresponding memory region is increased; when the data block processing is completed, the reference count of the corresponding memory region is decreased; when the reference count is detected to be zero, the memory mapping is automatically released and the resources are released.

[0175] like Figure 6 As shown in the figure, this application provides a device including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0176] Memory 113 is used to store computer programs;

[0177] In one embodiment of this application, when the processor 111 executes a program stored in the memory 113, it implements the streaming data loading method provided in any of the foregoing method embodiments, including:

[0178] Obtain the data file to be processed from the data lake;

[0179] Based on the available memory resources and computing device size of the current node, the dynamic block size of the data block is dynamically determined, and the data file is divided into several consecutive data blocks according to the dynamic block size.

[0180] A sliding window-style buffer is established, which is divided into: a front area for storing processed data blocks; a middle area for storing currently processed data blocks; and a back area for storing data blocks to be preloaded.

[0181] When the computing device begins processing the data block in the intermediate region, it asynchronously loads the next data block of the data block into the terminal region, and when the new data block is stored in the terminal region, it synchronously releases the data blocks that have been processed in the front-end region.

[0182] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the streaming data loading method provided in any of the foregoing method embodiments.

[0183] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0184] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0185] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0186] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for loading streaming data, characterized in that, The method includes: Obtain the data file to be processed from the data lake; Based on the available memory resources and computing device size of the current node, the dynamic block size of the data block is dynamically determined, and the data file is divided into several consecutive data blocks according to the dynamic block size. A sliding window-style buffer is established, which is divided into: a front area for storing processed data blocks; a middle area for storing currently processed data blocks; and a back area for storing data blocks to be preloaded. When the computing device starts processing the data block in the intermediate region, it asynchronously loads the next data block of the data block into the terminal region, and when the new data block is stored in the terminal region, it synchronously releases the data blocks that have been processed in the front-end region. The establishment of the sliding window buffer includes: Obtain the average processing speed of the computing device for processing a single data block, and monitor the average loading speed of data blocks from storage devices to memory; Calculate the initial window capacity based on the average processing speed and the average loading speed; The initial window capacity is increased by a preset window capacity adjustment base to determine the target window capacity; The sliding window buffer is initialized according to the target window capacity, so that the number of data blocks stored simultaneously in the sliding window buffer is consistent with the target window capacity.

2. The method according to claim 1, characterized in that, The step of dynamically determining the dynamic block size of data blocks based on the available memory resources and computing device scale of the current node includes: Monitor the remaining memory space in the available memory resources that is actually available for data loading; Obtain the quantization parameters of the computing device size; Calculate the initial block value based on the ratio between the remaining memory space and the quantization parameter; The dynamic block size is obtained by applying size constraints to the initial block value.

3. The method according to claim 1, characterized in that, The method further includes: Monitor the memory usage of the current node in real time; If the memory usage exceeds a preset high water level threshold, the target window capacity will be reduced to a first preset value. If the memory usage rate is lower than a preset low water level threshold, the target window capacity will be increased to a second preset value.

4. The method according to claim 1, characterized in that, The method further includes: Real-time monitoring of the processing characteristic parameters of the data blocks, which are used to characterize data access pattern features; The data supply mode is determined based on the processing characteristic parameters. The data supply mode includes: training mode: storing the data in blocks into a multi-level cache system, and / or inference mode: establishing a direct transmission channel from the data blocks to the computing device.

5. The method according to claim 4, characterized in that, The method further includes: In the training mode, two types of cache replicas are generated and maintained for the data blocks processed by the sliding window buffer: a memory cache replica: storing the data blocks whose access frequency is higher than a preset threshold; and a persistent cache replica: storing the complete data blocks in the distributed storage system. In response to a data request from the training iterator, query the frequently accessed data portion in the memory cache; if the memory cache misses, query the complete data block in the persistent cache; if the persistent cache misses, retrieve the data block from the remote storage system; for the data block retrieved from the remote storage system, write the complete data block to the persistent cache; extract the data in the data block whose access frequency is higher than a preset threshold and write it to the memory cache. For each data block, access statistics are collected periodically; a popularity value is calculated based on the access statistics; and a migration strategy for the data block between memory cache and persistent cache is determined based on the popularity value corresponding to the data block.

6. The method according to claim 1, characterized in that, The method further includes: The partitioned data blocks are divided using columnar storage memory mapping technology to establish a direct mapping channel from disk files to memory space; The direct mapping channel enables the computing device to directly access the raw binary content of the data blocks; Configure a shared memory pool mechanism to enable multiple data blocks to reuse the same memory mapping region, and dynamically manage the lifecycle of the memory mapping region based on reference counting, wherein: when a new data block is loaded, the reference count of the corresponding memory region is increased; when the data block processing is completed, the reference count of the corresponding memory region is decreased; when the reference count is detected to be zero, the memory mapping is automatically released and the resources are released.

7. A streaming data loading device, characterized in that, The device includes: The acquisition module is used to acquire data files to be processed from the data lake; The determination module is used to dynamically determine the dynamic block size of the data block based on the available memory resources and computing device scale of the current node, and divide the data file into several consecutive data blocks according to the dynamic block size; A module is established to create a sliding window-style buffer, which is divided into: a front area for storing processed data blocks; a middle area for storing currently processed data blocks; and a rear area for storing data blocks to be preloaded. The processing module is configured to asynchronously load the next data block of the data block into the end region when the computing device starts processing the data block of the intermediate region, and synchronously release the data blocks that have been processed in the front-end region when the new data block is stored in the end region. Specifically, the establishment module is used for: Obtain the average processing speed of the computing device for processing a single data block, and monitor the average loading speed of data blocks from storage devices to memory; Calculate the initial window capacity based on the average processing speed and the average loading speed; The initial window capacity is increased by a preset window capacity adjustment base to determine the target window capacity; The sliding window buffer is initialized according to the target window capacity, so that the number of data blocks stored simultaneously in the sliding window buffer is consistent with the target window capacity.

8. A device, characterized in that, include: A processor and a memory, the processor being configured to execute a streaming data loading program stored in the memory to implement the streaming data loading method according to any one of claims 1-6.

9. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the streaming data loading method according to any one of claims 1-6.

Images