Data acquisition method and system with evolution prediction mechanism, and program product

Abstract

The invention discloses a data acquisition method and system with an evolution prediction mechanism, and a program product. The method comprises the following steps: acquiring a data request of a target storage space and a request feature corresponding to the data request; generating reasoning feature parameters for prediction based on the request features, and obtaining a data prefetching strategy based on the reasoning feature parameters; the data prefetching strategy corresponds to a prediction result of the to-be-read file; copying a big data object corresponding to the to-be-read file to a local cache medium from a far-end storage space according to a data prefetching strategy; wherein the big data object is a data aggregation result of the to-be-read file which is pre-aggregated and stored in the far-end storage space. According to the embodiment of the invention, on the basis that the data aggregation results corresponding to the multiple to-be-read files are generated in advance, the waiting time of the GPU in intelligent model training is shortened through predictive reasoning on the prefetched data, and the data reading efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of training data acquisition for artificial intelligence model training, specifically to a data acquisition method, system, and program product with an evolutionary prediction mechanism. Background Technology

[0002] In recent years, large-scale artificial intelligence model training has been widely used in various fields such as industry and academia. The scale of its training data has developed from the early single machine GB level to the single training TB level and even PB level. The data form has also evolved from a single large file (e.g., the entire library packaged) to a large number of small files (e.g., each image in the commonly used image inference model YOLOv5 corresponds to an independent file).

[0003] Specifically, during the training process, the aforementioned data files are typically read repeatedly in batches by the dataset loader to form an I / O pattern of "multi-epoch (i.e., a complete traversal of the data level in AI model training), with random or sequential reading within each epoch). As GPU computing power (i.e., an important indicator of the performance of a graphics processing unit when performing computational tasks) continues to improve, the requirements for I / O bandwidth and latency during the training process are also increasing. This leads to the GPU becoming frequently idle due to waiting for data if the storage layer cannot deliver the data required for training in a timely manner, resulting in a significant decrease in training efficiency and resource utilization.

[0004] Therefore, storage and I / O optimization for AI training has become a key technical problem to be solved. Summary of the Invention

[0005] In view of the above-mentioned defects or deficiencies in the existing technology, it is desirable to provide a data acquisition method, system and program product with an evolution prediction mechanism. Based on the pre-generated data aggregation results corresponding to multiple files to be read, this method reduces the waiting time of the GPU in the training of intelligent models and improves the data reading efficiency by predictive inference of the pre-fetched data.

[0006] In a first aspect, the present invention provides a data acquisition method with an evolutionary prediction mechanism, applied to intelligent model training, the method comprising: Obtain the data request for the target storage space and the corresponding request characteristics. The request characteristics include at least the correlation information of the data request across multiple read dimensions. Inference feature parameters for prediction are generated based on request features, and a data prefetching strategy is obtained based on the inference feature parameters; the data prefetching strategy corresponds to the prediction result of the file to be read. According to the data prefetching strategy, the big data object corresponding to the file to be read is copied from the remote storage space to the local cache medium; whereby the big data object is the data aggregation result of the file to be read that is pre-aggregated and stored in the remote storage space.

[0007] In one possible implementation, a data prefetching strategy is obtained based on inference feature parameters, including: The inference feature parameters are input into the read prediction model, and the read prediction model calculates the read probability of multiple file objects in the target storage space; Based on the read probabilities of multiple file objects, file objects with read probabilities greater than a preset read probability are selected, and a data prefetching strategy is generated based on the file objects with read probabilities greater than the preset read probability.

[0008] In one possible implementation, the method also includes: The relevant data of the historical data prefetching strategy is input into the strategy evolution model, and the strategy evolution model outputs the update logic weights. The read prediction model outputs the read probability of different file objects based on the update logic weights. Among them, the relevant data of the historical data prefetching strategy includes at least the historical data requests corresponding to the historical data prefetching strategy, the request characteristics corresponding to the historical data requests, and the deviation degree of the historical data prefetching strategy.

[0009] In one possible implementation, the remote storage space also includes other data aggregation results from multiple file objects, and the method further includes: The system monitors the extended attribute status of file directories in the remote storage space in real time. When the extended attribute status changes to the target extended attribute status, it performs an aggregation operation on multiple file objects in the remote storage space. The change in the target extended attribute status is in response to user-triggered operations or automatic operations triggered by background threads.

[0010] In one possible implementation, the method also includes: Based on the aggregation operation of multiple file objects in the remote storage space, a mapping relationship between multiple file objects and big data objects is generated, and the mapping relationship is updated based on the changes in the state characteristics of each file object.

[0011] In one possible implementation, the method also includes: Based on the existing mapping relationship, the discrete small files in the remote storage space that do not have a mapping relationship are sorted in ascending order of file size to obtain the sorting result of the discrete small files; Based on the sorting results, the discrete small files are aggregated into a large data object with data gaps to complete the aggregation operation.

[0012] In one possible implementation, the data request for the target storage space and the corresponding request characteristics are obtained, including: Based on the preset probe function, the kernel data structure corresponding to each data request is obtained. When the device object pointer corresponding to any kernel data structure is consistent with the device object address corresponding to the target storage space, the data request corresponding to the kernel data structure is determined to be the data request of the target storage space.

[0013] Secondly, a data acquisition system with an evolution prediction mechanism is provided for use in intelligent model training. The system includes a data sampling module, a strategy engine module, a core control module, and a data management module. The data sampling module is used to obtain the data request of the target storage space and the request characteristics corresponding to the data request. The request characteristics include at least the correlation information of the data request in multiple reading dimensions. The strategy engine module is used to generate inference feature parameters for prediction based on request features, and to obtain a data prefetching strategy based on the inference feature parameters; the data prefetching strategy corresponds to the prediction result of the file to be read, and the file to be read includes at least the file object corresponding to the data request; The core control module is used to copy the big data object corresponding to the file to be read from the remote storage space to the local cache medium through the data management module, according to the data prefetching strategy; whereby the big data object is the data aggregation result of the file to be read that is pre-aggregated and stored in the remote storage space.

[0014] In one possible implementation, the system also includes a decision evolution module. The decision evolution module is used to input update logic weights into the policy engine module, so that the policy engine module can generate a data prefetching policy based on the update logic weights.

[0015] Thirdly, a computer program product is provided, which includes instructions that, when executed, perform the method described in any one of the first aspects.

[0016] Fourthly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the method described in any one of the first aspects above.

[0017] Fifthly, a computer-readable storage medium is provided having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the method described in any one of the first aspects above.

[0018] The data acquisition method, system, and program product with evolution prediction mechanism provided in this application, on the one hand, solves the problem of insufficient storage capacity of existing computing nodes by storing a large number of small files (i.e., corresponding files to be read) used for intelligent model training in a remote storage space; on the other hand, based on the acquired data request, through prediction inference of the files to be read (i.e., corresponding data prefetching strategy) and aggregation and merging of file objects in the remote storage space, the data used for intelligent model training can be quickly read into the local cache medium to adapt to the multi-epoch periodicity of intelligent model training, thereby reducing the waiting time of the GPU in intelligent model training and improving data reading efficiency. Attached Figure Description

[0019] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the architecture of a data acquisition system 10 with an evolution prediction mechanism provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a data acquisition method with an evolution prediction mechanism provided in an embodiment of this application. Figure 3 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0020] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present application will now be described in detail with reference to the accompanying drawings and embodiments. Furthermore, the term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The terms "first" and "second," etc., in the specification and claims of the embodiments of this application are used to distinguish different objects, not to describe a specific order of objects.

[0022] Currently, existing storage and file systems generally have the following gaps or deficiencies in the dimension of "multi-epoch periodic I / O for AI training": First, general-purpose caches struggle to translate periodicity and predictability into advantages in prefetching and scheduling; that is, they are not designed for "multi-epoch periodicity". Specifically, most general-purpose operating systems and kernel caches (such as the Linux Page Cache) employ LRU or its variants to replace data based on the "least recently used" strategy, which leads to insufficient utilization of the global periodicity presented by "multiple rounds of traversal of the entire dataset".

[0023] Correspondingly, when the training set size exceeds the memory capacity, the data already read in the previous stage may be discarded before a single epoch is completed, and the data needs to be read again from the slow storage during the next epoch training. This results in a low cache hit rate and the GPU being idle due to I / O waiting.

[0024] Secondly, the small file aggregation and general caching of traditional massive small file systems cannot solve problems such as "low hit rate in the next epoch" and "GPU waiting for I / O". That is, it cannot use the periodic information of "reading again in the next round" for targeted prefetching and scheduling. Specifically, existing massive small file systems (such as Taobao TFS and Facebook Haystack) designed for e-commerce, logs, and image CDN scenarios are mainly geared towards random, high-concurrency, and non-periodic access patterns. Their optimization focuses on small file aggregation, metadata shrinkage, and high throughput.

[0025] Correspondingly, traditional massive small file systems are typically characterized by single or small-scale repeated access to the same file, and the data popularity also changes with business needs; while the access characteristics during AI training are typically that the same dataset is repeatedly and completely traversed within multiple epochs, and within a single epoch, it is read in batch order or shuffled (i.e., multi-epoch periodicity, and randomness within a batch within a single epoch).

[0026] Based on this, due to the different time structures, there is a fundamental difference between the access model of traditional massive small file systems and the access model trained by AI.

[0027] In conclusion, storage and I / O optimization for AI training remain technical shortcomings of existing technologies.

[0028] In response, this application provides a data acquisition method with an evolution prediction mechanism. Based on the pre-generated aggregated data results of multiple files to be read, this method reduces the waiting time of the GPU during intelligent model training and improves data reading efficiency through predictive inference of the pre-fetched data.

[0029] In one embodiment of this application, a data acquisition system with an evolution prediction mechanism is provided. This system explicitly utilizes the Epoch periodicity to improve the hit rate of the next Epoch read by predicting and moving the data to be read between the Fast Path (i.e., the path for reading data from the local NVMe disk) and the Slow Path (the path for reading data from the remote storage system), thereby reducing GPU waiting time and improving data reading efficiency.

[0030] In one possible implementation, Figure 1 This is a schematic diagram of the architecture of a data acquisition system 10 with an evolution prediction mechanism provided in an embodiment of this application. The data acquisition system 10 with an evolution prediction mechanism is used to execute the data acquisition method with an evolution prediction mechanism provided in this application, such as... Figure 1 As shown, the data acquisition system 10 with an evolution prediction mechanism includes a data sampling module 11, a strategy engine module 12, a core control module 13, a data management module 14, and a decision evolution module 15.

[0031] For example, the data sampling module 11, as the core module of the system 10 for sensing the system's operating status and capturing the characteristics of business I / O behavior, is used to obtain various data indicators and write the request characteristics of the application I / O data requests corresponding to the data indicators into a circular queue, so that the reading thread can send them to the strategy engine module 12 and expose them to the user space for decision evolution.

[0032] In this embodiment, by deploying the data sampling module 11 in the kernel to achieve kernel-mode data acquisition, it is possible to detect and obtain key information of application I / O, thereby making the data acquisition process have fine perception granularity and high real-time performance.

[0033] For example, the strategy engine module 12 serves as the intelligent analysis hub of the system 10. It receives I / O feature data written by the data sampling module 11, infers and generates a data prefetching strategy for a specific file or directory based on the I / O feature data, and sends the data prefetching strategy to the core control module 13.

[0034] In this embodiment, based on the light inference of the policy engine module 12 in the kernel, the passive response of the traditional file system is transformed into active prediction, so as to prepare the data required by the application in advance and reduce the complex distributed reading across network nodes to direct reading of local high-speed media, thereby improving the processing efficiency of application data.

[0035] For example, the core control module 13, as the core hub and scheduling execution module of the system 10, is used to listen to and parse the data prefetching strategy issued by the strategy engine module 12, so as to issue specific execution logic (e.g., small file aggregation request, file / directory prefetching instruction, etc.) to the data management module 14.

[0036] For example, refer to Figure 1 The data management module 14 includes a metadata manager 141 and a data manager 142. The metadata manager 141 is used to manage file metadata, and the data manager 142 is used to realize the efficient transfer and lifecycle management of underlying physical data between local and remote storage media.

[0037] Specifically, the core functions of the metadata manager 141 include metadata management functions that meet the needs of integrating and invalidating small files or data areas, caching files or data areas, and invalidating caches; the data manager 142 can directly interface with local and remote storage media to implement true physical prefetching and aggregation operations.

[0038] For example, the decision evolution module 15 is used to listen to the I / O feature data reported by the kernel, perform offline deep analysis based on the I / O feature data, optimize the logical weights, and then send the updated logical weights to the policy engine module 12 to complete the policy evolution; correspondingly, the policy engine module 12 provides an interface for updating logical weights to receive updated logical weights.

[0039] It should be noted that the aforementioned Logic Weight refers to a set of parameters used to guide the policy engine module 12 in reasoning, in order to determine the strength of the judgment of the prediction results for different I / O features.

[0040] In another embodiment of this application, a data acquisition method with an evolution prediction mechanism is provided.

[0041] In one possible implementation, Figure 2 This is a flowchart illustrating a data acquisition method with an evolution prediction mechanism provided in an embodiment of this application, as shown below. Figure 2 As shown, the method specifically includes the following steps: Step S201: Obtain the data request for the target storage space and the request characteristics corresponding to the data request. The request characteristics include at least the correlation information of the data request in multiple reading dimensions.

[0042] In one possible implementation, the data requests to the target storage space and the corresponding request characteristics can be obtained by implanting an interception probe; wherein, the data request to the target storage space is a read request for a small file object within the target storage space.

[0043] For example, the kernel data structure corresponding to each data request can be obtained based on a preset probe function. When the device object pointer corresponding to any kernel data structure is consistent with the device object address corresponding to the target storage space, the data request corresponding to the kernel data structure is determined to be the data request of the target storage space.

[0044] In one example, an intercept probe can be implanted at the core I / O entry point of the Virtual File System (VFS) to capture all data requests to the target storage space.

[0045] Correspondingly, when an I / O request triggers an interception probe, the correlation information of the data request in multiple reading dimensions can be extracted in real time. This correlation information includes, but is not limited to, the target file to be read, the read offset, and the read length.

[0046] In this embodiment, by embedding data sampling deep into the kernel mode of the operating system in a low-overhead, non-blocking manner, it is possible to continuously and fully collect multi-dimensional access features, thereby providing real-time and high-precision data support for the subsequent reasoning and decision evolution of the policy engine.

[0047] Step S202: Generate inference feature parameters for prediction based on request features, and obtain data prefetching strategy based on inference feature parameters; the data prefetching strategy corresponds to the prediction result of the file to be read.

[0048] For example, the request features can be preprocessed to generate inference feature parameters for prediction; wherein, the inference feature parameters are, for example, represented as a multidimensional matrix.

[0049] In one possible implementation, after obtaining the inference feature parameters, the inference feature parameters can be inferred and predicted using the read prediction model deployed in the policy engine module 12 to obtain the data prefetching strategy.

[0050] For example, inference feature parameters can be input into a read prediction model (e.g., a built-in MLP model), and the read prediction model can calculate the read probability of multiple file objects in the target storage space; then, based on the read probabilities of multiple file objects, file objects with read probabilities greater than a preset read probability are selected, and a data prefetching strategy is generated based on the file objects with read probabilities greater than the preset read probability; wherein, the preset read probability is, for example, 99%.

[0051] It should be noted that when the output of the reading prediction model indicates that the reading probability of multiple file objects is less than the reading probability, the generated data prefetching strategy is empty; that is, there are no file objects that meet the prefetching conditions.

[0052] In one example, when a data request for the target storage space corresponds to reading small files A and B within the target storage space, the inference feature parameters corresponding to small files A and B are input into the read prediction model so that the read prediction model can calculate the read probability of small files C, D, E... within the target storage space; then, D and E with read probabilities greater than the preset read probabilities are selected, and a data prefetching strategy is generated based on small files D and E; where the data prefetching strategy can also be called a candidate dataset, that is, the file / directory that needs to be prefetched.

[0053] Step S203: According to the data prefetching strategy, the big data object corresponding to the file to be read is copied from the remote storage space to the local cache medium; wherein, the big data object is the data aggregation result of the file to be read that is pre-aggregated and stored in the remote storage space.

[0054] For example, the core control module 13 can trigger a multi-threaded asynchronous prefetching mechanism to call the data manager 142 to copy the big data objects corresponding to the files / directories included in the data prefetching strategy from the remote slow storage to the local cache medium in batches.

[0055] Correspondingly, after the data manager 142 completes the data prefetching operation (i.e., completes the above-mentioned batch copy operation), the metadata manager 141 can register the mapping relationship table between the virtual file and the high-speed storage physical block in the high-speed hash table in memory, and update the mapping relationship in the file metadata for maintenance in system memory and metadata.

[0056] Based on this, when an upper-layer application subsequently requests to read the file, the system 10 can use a lazy addressing mechanism to compare timestamps for consistency verification (that is, during the data reading phase, consistency verification is completed by comparing extremely lightweight timestamps, thus eliminating the need for a complete metadata handshake process). After the verification passes, the request is redirected to the file corresponding to the local cache, thereby achieving extremely fast local read return.

[0057] In this embodiment, when implementing the data prefetching process, the strategy engine module 12 mainly performs light inference and generates candidate datasets. The core control module 13 calls the data manager 142 to perform data preloading based on the candidate datasets. I / O acceleration that is completely transparent to upper-layer services is achieved through intelligent routing and diversion at the underlying Virtual File System (VFS).

[0058] In another embodiment of this application, a specific method for updating the logical weights applied by the prediction model is also provided.

[0059] In one possible implementation, based on the strategy evolution model deployed by the decision evolution module 15, relevant data of the historical data prefetching strategy is input into the strategy evolution model, and the strategy evolution model outputs update logic weights, so that the read prediction model outputs the read probability of different file objects based on the update logic weights; wherein, the relevant data of the historical data prefetching strategy includes at least the historical data requests corresponding to the historical data prefetching strategy, the request characteristics corresponding to the historical data requests, and the deviation degree of the historical data prefetching strategy.

[0060] For example, historical data can be used as a training set to pre-fetch relevant data of the strategy and optimize the training of the strategy evolution model to generate new logical weights.

[0061] Correspondingly, the new logical weights can be updated to the kernel MLP model (i.e., the multilayer perceptron model, specifically the reading prediction model in this invention) through the update logical weights interface exposed by the strategy engine module 12, so as to complete the decision evolution.

[0062] The decision evolution process provided in this embodiment is an offline update process. It can receive real-time feature data from the data sampling module 11, optimize and generate update logic weights through AI training, and directly send the update logic weights to the MLP model of the policy engine module 12 to realize the update and optimization of the kernel inference model.

[0063] Based on this, the data acquisition method provided in this application has the following advantages: First, it has self-correcting capabilities: it can adjust the prefetching progress according to business switching; second, it ensures zero business interruption: it adopts atomic switching technology to hot update weight parameters without restarting the system or suspending any I / O requests, thus ensuring the continuity of core business; third, it decouples computation and storage: it transfers the deep analysis that consumes a lot of computing power to user space for execution, while keeping the lightweight table lookup / linear calculation in kernel space, thus achieving absolute optimization of resource allocation.

[0064] In another embodiment of this application, a specific aggregation method for file objects in remote storage space is also provided.

[0065] In this embodiment of the application, in response to the storage fragmentation and metadata storm problems caused by a large number of small files, an efficient offline aggregation and garbage collection stream designed in collaboration between the metadata manager 141 and the data manager 142 is used to transform discrete underlying data into continuous large blocks of objects through asynchronous batch processing.

[0066] In one possible implementation, the extended attribute status of file directories in the remote storage space can be monitored in real time. When the extended attribute status changes to the target extended attribute status, an aggregation operation on multiple file objects in the remote storage space is performed. Here, the file directory in the remote storage space corresponds to the directory in the remote storage space used to store small files, and the change in the target extended attribute status responds to user-triggered operations or automatic triggering operations of background threads.

[0067] For example, the core control module 13 can monitor the extended attribute status of the file directory in real time through periodic polling and event listening mechanisms. When the target extended attribute status is identified, the background multi-threaded scanning and aggregation tasks are automatically triggered.

[0068] In one example, the target extended attribute status is a custom attribute, and no specific restrictions are imposed here.

[0069] In one possible implementation, a mapping relationship between multiple file objects and big data objects can be generated based on the aggregation operation of multiple file objects in the remote storage space, and the mapping relationship can be updated based on the changes in the state characteristics of each file object.

[0070] For example, before performing the aggregation operation, the existing logical mapping relationship of the target file object (i.e., the file object to be aggregated) can be validated, and the mapping relationship can be updated according to the validation result.

[0071] In one example, the latest status characteristics (e.g., timestamps) of the original file can be compared. If the comparison result indicates that the original file has been modified, the metadata controller is called to mark the old mapping relationship corresponding to the original file as "invalid".

[0072] In another example, the underlying integrated big data objects can be monitored. If it is determined that all small file mappings within a big data object have become invalid, the underlying storage interface is directly triggered to physically release the storage space occupied by the big data object, thereby achieving efficient recycling of storage resources.

[0073] In one possible implementation, discrete small files that do not have a mapping relationship in the remote storage space can be sorted in ascending order of file size based on the existing mapping relationship to obtain the sorting result of the discrete small files; then, according to the sorting result, each discrete small file is aggregated into a big data object with data gaps to complete the aggregation operation.

[0074] For example, the system 10 can start a concurrent processing thread to obtain discrete small files that are smaller than a preset size and currently have no valid mapping relationship.

[0075] Correspondingly, when performing a physical merge operation, based on the sorting results of the discrete small files, the appropriately sized discrete small files can be merged into the data gaps in the existing big data objects caused by the deletion or invalidation of historical files. When the existing data gaps are insufficient to merge the existing discrete small files, a new large-capacity data object can be created for additional storage.

[0076] It should be noted that once all eligible small files in the file directory within the remote storage space have been processed, System 10 marks the progress attribute as complete and simultaneously removes the pending scan attribute from that file directory. This avoids redundant scanning and unnecessary computational waste by System 10.

[0077] The merging operation in this embodiment achieves several advantages. First, it improves prefetching and cache loading efficiency: Based on deep collaboration with the strategy engine, when a data retrieval command is issued, instead of fetching multiple small files sporadically over the network, it directly packages and fetches large-capacity aggregated objects into the local high-speed cache. This large-block streaming loading method improves the execution efficiency of the prefetching pipeline, ensuring uninterrupted data supply to front-end computing resources. Second, it maximizes storage bandwidth: Fragmented data is physically reorganized into continuous large data blocks in the background beforehand. When upper-layer applications initiate intensive reads, the underlying storage only needs to perform efficient large-block sequential reads, replacing the traditional inefficient massive random reads and releasing the maximum throughput potential of the underlying storage medium. Third, it avoids storage fragmentation: The introduction of a space management strategy of "failure marking and gap priority merging and filling" prioritizes the use of data gaps in existing large data objects caused by file updates / deletions to store new small files, avoiding fragmentation of the underlying storage and achieving efficient and seamless reuse of storage space.

[0078] In another embodiment of this application, further technical details of a data acquisition method with an evolution prediction mechanism are provided.

[0079] In one possible implementation, the data acquisition method with an evolution prediction mechanism can be divided into a file merging process, a data sampling process, a data prefetching process, and a decision evolution process.

[0080] For example, the file merging process includes the following steps: S1. Event-driven and scan-triggered; S2. Mapping verification and intelligent waste recycling; S3. Spatial void reuse and ordered aggregation; S4. State closure and attribute cleanup.

[0081] For example, the data sampling process includes the following steps: S1. Capture data request; S2. Extract request features; S3. Provide request features to external parties.

[0082] For example, the data prefetching process includes the following steps: S1. Data preprocessing; S2. The strategy engine generates strategies; S3. Asynchronous prefetching; S4. Router activation.

[0083] For example, the decision evolution process includes the following steps: S1. Feature data collection; S2. Offline training; S3. Update kernel weights.

[0084] Based on the above embodiments and the above execution steps, the technical comparison results between the file prefetching system provided in this application embodiment and the existing storage file system can be obtained as shown in Table 1 below.

[0085] Table 1. Technical Comparison between the File Prefetching System Provided in the Embodiments of this Application and Existing Storage File Systems

[0086] In summary, the data acquisition method and system with an evolution prediction mechanism provided in this application, while maintaining the standard file system interface and compatibility with the distributed file system backend, can provide predictive reading, small file merging, and multi-level storage scheduling (Fast Path / Slow Path) for I / O features such as multi-epoch in AI training, forming an efficient I / O pipeline from storage to GPU memory to maximize GPU utilization and minimize I / O latency. Secondly, through I / O feature collection, lightweight inference and prediction, and multi-level storage and prefetch scheduling, the cache hit rate and data availability rate in training scenarios can be improved. Thus, focusing on read path and cache / scheduling optimization during the training phase, and through file system and kernel-side capabilities, existing training programs can achieve I / O acceleration without modifying business logic.

[0087] The following is for reference. Figure 3 , Figure 3 A schematic diagram of a communication device suitable for implementing embodiments of this application is shown, such as... Figure 3As shown, the communication device 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 302 or a program loaded from a storage section 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the system's operating instructions. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0088] The following components are connected to the input / output (I / O) interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the input / output (I / O) interface 305 as needed. A removable medium 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 310 as needed so that computer programs read from it can be installed into the storage section 308 as needed.

[0089] Specifically, according to embodiments of this application, the flowchart above refers to... Figure 2 The described process can be implemented as a computer software program. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program contains program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the functions defined in the system of this application.

[0090] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium compatible with computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0091] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operational instructions of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two connected blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified functions or operational instructions, or using a combination of dedicated hardware and computer instructions.

[0092] The modules described in the embodiments of this application can be implemented in software or hardware. These modules can also be located within a processor.

[0093] On the other hand, this application also provides a computer-readable storage medium, which may be included in the communication device described in the above embodiments, or may exist independently and not assembled into the communication device. The aforementioned computer-readable storage medium stores one or more programs that, when used by one or more processors, execute the methods described in this application. For example, it may execute... Figure 2 The steps of the method shown.

[0094] This application provides a computer program product including instructions that, when executed, cause the method described in this application to be performed. For example, it can execute... Figure 2 The steps of the method shown.

[0095] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A data acquisition method with an evolution prediction mechanism, characterized in that, The method, applied to intelligent model training, includes: Obtain the data request for the target storage space and the request characteristics corresponding to the data request, wherein the request characteristics include at least the association information of the data request in multiple reading dimensions; Based on the request features, inference feature parameters for prediction are generated, and a data prefetching strategy is obtained based on the inference feature parameters; the data prefetching strategy corresponds to the prediction result of the file to be read. According to the data prefetching strategy, the big data object corresponding to the file to be read is copied from the remote storage space to the local cache medium; wherein, the big data object is the data aggregation result of the file to be read that is pre-aggregated and stored in the remote storage space.

2. The data acquisition method with an evolution prediction mechanism according to claim 1, characterized in that, The data prefetching strategy based on the inference feature parameters includes: The inference feature parameters are input into the read prediction model, and the read prediction model calculates the read probability of multiple file objects in the target storage space; Based on the read probabilities of the multiple file objects, file objects with read probabilities greater than a preset read probability are selected, and the data prefetching strategy is generated based on the file objects with read probabilities greater than the preset read probability.

3. The data acquisition method with an evolution prediction mechanism according to claim 2, characterized in that, The method further includes: The relevant data of the historical data prefetching strategy is input into the strategy evolution model, and the strategy evolution model outputs the update logic weights, so that the reading prediction model outputs the reading probability of different file objects based on the update logic weights; The relevant data of the historical data prefetching strategy includes at least the historical data requests corresponding to the historical data prefetching strategy, the request characteristics corresponding to the historical data requests, and the deviation degree of the historical data prefetching strategy.

4. The data acquisition method with an evolution prediction mechanism according to claim 1, characterized in that, The remote storage space also includes other data aggregation results of multiple file objects, and the method further includes: The extended attribute status of file directories in the remote storage space is monitored in real time. When the extended attribute status changes to the target extended attribute status, an aggregation operation is performed on multiple file objects in the remote storage space. The change in the target extended attribute status is in response to a user-triggered operation or an automatic triggering operation by a background thread.

5. The data acquisition method with an evolution prediction mechanism according to claim 4, characterized in that, The method further includes: Based on the aggregation operation on multiple file objects in the remote storage space, a mapping relationship between the multiple file objects and the big data object is generated, and the mapping relationship is updated based on the state feature changes of each file object.

6. The data acquisition method with an evolution prediction mechanism according to claim 5, characterized in that, The method further includes: Based on the existing mapping relationship, the discrete small files in the remote storage space that do not have a mapping relationship are sorted in ascending order of file size to obtain the sorting result of the discrete small files; Based on the sorting results, the discrete small files are aggregated into a large data object with data gaps to complete the aggregation operation.

7. The data acquisition method with an evolution prediction mechanism according to claim 1, characterized in that, The data request for obtaining the target storage space and the request characteristics corresponding to the data request include: Based on the preset probe function, the kernel data structure corresponding to each data request is obtained. When the device object pointer corresponding to any kernel data structure is consistent with the device object address corresponding to the target storage space, the data request corresponding to the kernel data structure is determined to be the data request of the target storage space.

8. A data acquisition system with an evolution prediction mechanism, characterized in that, The system, used for intelligent model training, includes a data sampling module, a policy engine module, a core control module, and a data management module. The data sampling module is used to obtain the data request of the target storage space and the request features corresponding to the data request. The request features include at least the association information of the data request in multiple reading dimensions. The strategy engine module is used to generate inference feature parameters for prediction based on the request features, and to obtain a data prefetching strategy based on the inference feature parameters; the data prefetching strategy corresponds to the prediction result of the file to be read. The core control module is used to copy the big data object corresponding to the file to be read from the remote storage space to the local cache medium through the data management module according to the data prefetching strategy; wherein, the big data object is the data aggregation result of the file to be read that is pre-aggregated and stored in the remote storage space.

9. The data acquisition system with an evolution prediction mechanism according to claim 8, characterized in that, The system also includes a decision evolution module. The decision evolution module is used to input update logic weights into the strategy engine module, so that the strategy engine module can generate the data prefetching strategy based on the update logic weights.

10. A computer program product, characterized in that, The computer program product includes instructions that, when executed, cause the method as described in any one of claims 1-7 to be implemented.

Images