Request processing method and device in distributed system, equipment, medium and product

Abstract

The invention discloses a request processing method and device in a distributed system, equipment, a medium and a product, and relates to the field of distributed systems. The method comprises the following steps: sending a target IO request; inserting the target IO request into a request queue based on the priority information of the target IO request; processing at least one IO request in the request queue in sequence; wherein the at least one IO request is ranked in the request queue according to the priority from high to low. The IO requests which can cause the real-time influence on the training tasks are processed preferentially, and the IO requests which do not influence the training real-time performance are processed in parallel while the training tasks are ensured to be continuously carried out. The waiting time for the computing nodes to carry out the training tasks is shortened, and the overall resource utilization rate and the task execution efficiency of the distributed system are improved.

Description

Technical Field

[0001] This application relates to the field of distributed system technology, and in particular to a request processing method, apparatus, device, medium and product in a distributed system. Background Technology

[0002] Distributed training is a machine learning model training technique that breaks down training tasks and distributes them across multiple computing nodes for parallel execution. Computing and storage nodes work together to complete parallel model training. While this architecture provides the necessary computational scale, it also introduces higher data input / output (IO) requirements.

[0003] In related technologies, when computing nodes in a distributed system execute training tasks, they need to acquire training data and save the generated model data. The input / output of data is typically managed by a distributed file system. During training, the distributed file system receives various types of I / O requests and performs data read or write operations according to a preset processing flow, completing data transfer between computing nodes and storage nodes.

[0004] However, distributed file systems cannot distinguish which I / O requests directly affect the current training task and which can be processed later. This means that I / O requests that directly affect the training task may be queued after other I / O requests, causing the training task to be interrupted while waiting for training data, thus reducing training efficiency. Summary of the Invention

[0005] This application provides a request processing method, apparatus, device, medium, and product in a distributed system. The technical solution provided by this application includes the following aspects.

[0006] According to one aspect of the embodiments of this application, a request processing method in a distributed system is provided, the method comprising: The compute node sends a target I / O request to the storage node; The storage node receives the target I / O request; based on the priority information of the target I / O request, it inserts the target I / O request into the request queue; and processes at least one I / O request in the request queue in sequence. The at least one IO request is sorted in the request queue according to its priority from high to low.

[0007] According to one aspect of the embodiments of this application, a computing device in a distributed system is provided, the computing device comprising: The computing process module is configured to generate target I / O requests, which correspond to priority information. The sending module is configured to send the target IO request to the storage node, the storage node being used to receive the target IO request; insert the target IO request into a request queue based on the priority information of the target IO request; and process at least one IO request in the request queue in sequence.

[0008] According to one aspect of the embodiments of this application, a storage device in a distributed system is also provided, the storage device comprising: The receiving module is configured to receive target I / O requests sent by the computing node; The IO priority scheduling module is configured to insert the target IO request into the request queue based on the priority information of the target IO request; The IO processing module is configured to process at least one IO request in the request queue in sequence.

[0009] According to one aspect of the embodiments of this application, a distributed system is provided, the distributed system including computing nodes and storage nodes, the computing nodes being configured to send target IO requests to the storage nodes; the storage nodes being configured to receive the target IO requests; insert the target IO requests into a request queue based on priority information of the target IO requests; and process at least one IO request in the request queue in sequence.

[0010] According to one aspect of the embodiments of this application, a computing node is provided, the computing node comprising: a processor and a memory, the memory storing at least one program, the at least one program being loaded and executed by the processor to implement the request processing method in the distributed system described above.

[0011] According to one aspect of the embodiments of this application, a storage node is provided, the storage node comprising: a processor and a memory, the memory storing at least one program, the at least one program being loaded and executed by the processor to implement the request processing method in the distributed system described above.

[0012] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the request processing method in the distributed system described above.

[0013] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium, and a processor reading from the computer-readable storage medium and executing the computer program to implement the request processing method in the above-described distributed system.

[0014] The technical solution provided in this application can bring the following beneficial effects: An IO priority scheduling mechanism was designed. The distributed file system inserts IO requests into a request queue based on their priority information, thus implementing priority division and scheduling of IO requests. At least one IO request in the request queue is processed sequentially. Combined with the IO processing module's decomposition of large-granularity IO requests, this prioritizes IO requests that may affect the real-time performance of the training task. While ensuring the continuous execution of the training task, background IO requests that do not affect the real-time performance of training are processed in parallel. This reduces the waiting time of computing devices for training tasks and improves the overall resource utilization and task execution efficiency of the distributed system. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the structure of a distributed system provided in an exemplary embodiment of this application; Figure 2 This is a schematic diagram of the structure of a distributed system provided in another exemplary embodiment of this application; Figure 3 This is a flowchart of a request processing method in a distributed system provided by an exemplary embodiment of this application; Figure 4 This is a schematic diagram of a request queue in a distributed file system provided in an exemplary embodiment of this application; Figure 5 This is a flowchart of a request processing method in a distributed system provided by another exemplary embodiment of this application; Figure 6 This is a schematic diagram illustrating the setting of extended attributes of a target file in a distributed system according to an exemplary embodiment of this application; Figure 7 This is a schematic diagram illustrating the arrangement of IO requests in a distributed file system provided by an exemplary embodiment of this application; Figure 8 This is a flowchart of a request processing method in a distributed system provided by another exemplary embodiment of this application; Figure 9 This is a structural block diagram of a request processing apparatus provided in an exemplary embodiment of this application; Figure 10This is a structural block diagram of a request processing apparatus provided in another exemplary embodiment of this application; Figure 11 This is a structural block diagram of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0018] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0019] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0020] It should be understood that although the terms “first,” “second,” etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word “if” as used herein may be interpreted as “when…” or “in response to determination.”

[0021] It should be noted that this application may display a prompt interface, pop-up window, or output voice prompt information before collecting user, processor, computer device, and other related data, and during the process of collecting user-related data. This prompt interface, pop-up window, or voice prompt information is used to inform the user that their related data is being collected. This ensures that the application only begins executing the steps related to collecting user-related data after receiving confirmation from the user regarding the prompt interface or pop-up window; otherwise (i.e., without receiving confirmation from the user), the steps related to collecting user-related data end, meaning no user-related data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of related user data must comply with the laws, regulations, and standards of relevant countries and regions.

[0022] First, the terms used in the embodiments of this application will be introduced.

[0023] An I / O request, in a distributed system, is a request instruction initiated by a computing process on a compute node to read training data (such as datasets or model checkpoints) from a storage node or to write data generated during training (such as model checkpoints) to a storage node. I / O requests include input and output requests. Input requests are requests initiated by the compute node's operating system when data is transferred from the compute node's memory to other external devices; output requests are requests initiated by the compute node's operating system when data is transferred from the compute node's memory to other external devices. In this embodiment, based on whether the compute node blocks the compute node's processor during the processing of the target I / O request, I / O requests can be divided into blocking I / O requests and non-blocking I / O requests. When a compute node initiates and processes a blocking I / O request, the compute node's processor is blocked, and the processor cannot perform other operations.

[0024] A model refers to the structure of a machine learning model that is continuously optimized through distributed training. Its training process is executed by computing nodes, specifically including: performing forward and backward propagation based on the dataset read from storage nodes to calculate gradients, and updating model parameters by synchronizing gradients among computing nodes. During training, the complete state of the model (including parameters, optimizer state, etc.) is saved as a model checkpoint file and written to the storage nodes.

[0025] Model checkpoints are datasets that periodically save the entire training state during model training. They typically contain key information such as model parameters, optimizer status, and the current training epoch. Saving model checkpoints provides fault tolerance and recovery capabilities for the training process: when the training task unexpectedly stops due to hardware failure or human interruption, training can be quickly resumed directly from the most recent checkpoint, avoiding the huge waste of time and resources caused by starting from scratch.

[0026] A dataset is a set of related data samples used to train, validate, or test machine learning models. Data sets can be in the form of numbers, text, images, audio, or video, and are typically used to train and test machine learning models.

[0027] Distributed training refers to a machine learning model training method that utilizes multiple computing nodes working collaboratively. By decomposing the training task of a machine learning model into multiple sub-tasks and distributing them across multiple computing nodes for parallel computation, it supports training large machine learning models that exceed the memory capacity of a single computing node. In some embodiments, a distributed system typically includes two core components: computing nodes and storage nodes. Computing nodes are responsible for performing numerical computation tasks such as forward and backward propagation of the model. Each computing node is typically equipped with a high-performance processor. After each computing node independently completes forward and backward propagation computations based on its assigned subset of data, it obtains a set of local data-based update directions for the model parameters, i.e., gradients. To obtain a consistent model update based on the global data distribution, these gradients scattered across different nodes must be efficiently collected, synchronized, and aggregated before being used to update the global model parameters. Storage nodes provide ample data access space to store the datasets and model checkpoints required for large-scale training.

[0028] Distributed training completes the model training process in the following ways: 1. Data Parallelism: In data parallelism, the same training dataset is divided into multiple subsets, each computing node processes a different subset, and each computing node maintains a complete copy of the model. After each computing node completes forward and backward propagation, it calculates the local gradients; then, through ensemble communication, the gradients of all computing nodes are synchronized and aggregated, and the model parameters of all copies are updated based on this. 2. Model Parallelism: Model parallelism is typically suitable when the model parameter size exceeds the memory capacity of a single node. This method divides the model structure into multiple parts and distributes them across different computing nodes. During training, data needs to be passed sequentially between computing nodes, with the output of the previous computing node serving as the input of the next computing node, thus forming a pipelined computation.

[0029] Storage nodes typically exist in a distributed manner, generally employing a distributed file system (DFS) as a unified management layer. The DFS aggregates multiple independent storage nodes into a logically single shared storage pool. When compute nodes need to read or write data during machine learning model training, it manages and schedules the data, providing a unified global namespace. In some embodiments, the DFS runs on all storage nodes used to store data needed for machine learning model training. This DFS manages and schedules storage resources through data sharding, replication, and multi-node collaboration mechanisms, providing data storage and access services to compute nodes in the distributed system. Specifically, data sharding divides files into multiple data blocks, distributing them across different nodes to improve read / write parallelism; replication copies the same data block to multiple nodes to prevent data loss.

[0030] To improve the training efficiency of machine learning models, a common approach is to accelerate the read and write speed of data during distributed training. Remote Direct Memory Access (RDMA) network technology is frequently used to access data. RDMA is a network technology that allows one computer on a network to directly access the memory of another computer without the intervention of the other computer's operating system. Data transfer via RDMA speeds up data read and write operations and reduces processor blocking time.

[0031] Optionally, the related technology also employs setting up a cache space in the computing node. This cache space typically uses a flash drive with high read and write speeds. During data transmission, the data is first pulled from the storage node to the cache space and then stored in the memory of the computing node, thereby accelerating the data read and write speed and reducing the blocking time of the processor in the computing node.

[0032] Optionally, during the training of machine learning models on computing nodes, to cope with potential hardware failures, network fluctuations, and other unforeseen circumstances, and to avoid the loss of previous computational results due to training interruptions, it is necessary to periodically save model checkpoints to record the current training parameters, ensuring that training can be quickly resumed from the most recent checkpoint. For this periodic model checkpoint saving I / O request, related technologies also employ an asynchronous model checkpoint saving scheme: first, the checkpoint data is quickly copied synchronously to the high-speed storage medium on the computing node. After completion, the processor can immediately continue the training task. Subsequently, while the processor is training, the locally stored checkpoint data is asynchronously written to the storage node through a background process, accelerating data read and write speeds and reducing processor blocking time in the computer.

[0033] However, the aforementioned methods for accelerating data read and write speeds suffer from at least one or more of the following problems: 1. Compute nodes continuously read multiple training sets from storage nodes. Simultaneously, compute nodes need to periodically write model checkpoints to storage. These two types of I / O requests highly overlap in time and resources, leading to storage bandwidth contention. 2. Existing mainstream distributed file systems such as Ceph File System (CephFS) and Lustre storage systems are typically designed to be inefficient in handling both the large-granularity I / O requests represented by "checkpoint writes" and the small-granularity I / O requests represented by "dataset loading" concurrently. When both occur concurrently, the allocation of resources within the distributed file system is prone to imbalance. Large-granularity I / O requests severely interfere with the processing speed of small-granularity I / O requests, resulting in increased data loading latency and slowing down the entire training process.

[0034] In this embodiment, IO requests are prioritized and scheduled through an IO request priority scheduling mechanism. When multiple IO requests need to be processed simultaneously, the IO requests that block the processor are divided into high-priority IO requests, and the high-priority IO requests are processed first to complete the data transmission and speed up the processor's training process.

[0035] Figure 1 This is a schematic diagram of a distributed system 100 provided in an exemplary embodiment of this application. The distributed system 100 includes: multiple computing nodes 110, multiple storage nodes 120, and a distributed file system 130. Arrows between different computing nodes 110, between computing nodes 110 and storage nodes 120, and between different storage nodes 120 indicate the data flow relationships between the nodes. Figure 1 The distributed system architecture shown introduces the various nodes and modules in distributed system 100: Compute Node 110: This is the basic working unit participating in collaborative training within the distributed system 100, typically corresponding to a standalone physical server or a virtual machine with a complete operating system environment. Each compute node 110 includes a Central Processing Unit (CPU), one or more Graphics Processing Units (GPUs), or a dedicated AI acceleration unit such as a Neural Processing Unit (NPU). These processors collectively initiate and host one or more training processes, specifically executing all numerical computation tasks such as forward propagation, back propagation, and parameter updates of the machine learning model. The training process is the business process running on the processor of compute node 110 that initiates training tasks, fulfilling file transfer requirements during the training process by issuing various I / O requests. Furthermore, the CPU is also responsible for performing data preprocessing, controlling communication between GPUs, and managing other control logic within the training tasks. The GPU or NPU is used to perform forward computation and back propagation operations during model training.

[0036] Storage node 120: Primarily used to store various training data for training machine learning models, including training datasets, model checkpoints, and training logs. In some embodiments, each storage node 120 is uniformly managed and scheduled by a distributed file system 130.

[0037] Distributed file system 130: A software system for unified management and integration of storage resources across multiple storage nodes 120. In some embodiments, distributed file system 130 runs on all storage nodes 120 used to store data required for training models, aggregating dispersed physical storage resources into unified logical storage resources. This distributed file system 130 manages and schedules storage resources through data sharding, replication, and multi-node collaboration mechanisms, and provides data storage and access services to the computing nodes 110 in the distributed system 100. Specifically, data sharding divides files into multiple data blocks, distributing them across different storage nodes 120 to improve read / write parallelism; replication copies the same data block to multiple storage nodes 120 to prevent data loss.

[0038] In this distributed system, different nodes can connect to each other via a network protocol. The network protocol supports TCP / IP. Different computing nodes 110 are interconnected to work collaboratively to complete the machine learning model training task. Each computing node 110 independently completes forward and backward propagation calculations based on its assigned subset of data, and then obtains a set of update directions for the model parameters based on local data, i.e., gradients. All computing nodes 110 calculate the model gradients, and synchronize and aggregate their respective gradients through the network protocol to update the local model. Different storage nodes 120 are interconnected to store data securely and efficiently. The storage nodes 120 coordinate through a communication protocol to distribute complete files that are automatically divided into multiple data shards by the distributed file system to multiple distributed storage nodes 120 according to a preset data distribution strategy. In order to ensure data reliability, each data shard is copied multiple times and stored on different storage nodes 120. In different storage nodes 120, the writes to all replicas are synchronous and consistent, ensuring data availability and no loss even in the event of a failure of one or more storage nodes 120. Compute nodes 110 and storage nodes 120 are interconnected to meet the data read / write requirements of the training task, and are connected via a network protocol. Compute node 110 is responsible for running the computational tasks of model training; storage node 120 is responsible for storing the data required for model training. The interaction between the two is essentially a data read / write operation: compute node 110 reads the input data required for training from storage node 120 according to the needs of the training process; simultaneously, compute node 110 also writes various state data generated during training (such as intermediate variables and final parameters of the model) to storage node 120 for storage.

[0039] Optionally, the network protocol also supports RDMA, a networking technology that allows one computer on a network to directly access the memory of another computer without the intervention of the other computer's operating system. This technology significantly reduces data transmission latency, enabling low-latency, high-bandwidth data transmission between nodes, and is suitable for the frequent data exchange needs in large-scale distributed training scenarios.

[0040] Figure 1 The distributed system architecture shown may also optionally include an IO priority scheduling module 132 and an IO processing module 134. IO priority scheduling module 132: This module inserts IO requests into the request queue in the distributed file system. Based on the priority information of the data to be read or written, it places different IO requests in different positions within the request queue, thus achieving IO request priority scheduling. Specifically, reading or writing data corresponds to I / O requests for the target file.

[0041] IO processing module 134: This module breaks down large-granularity IO requests (e.g., 16MB) into smaller-granularity sub-IO requests (e.g., 64KB). This is done to prevent the reading or writing of large files from blocking the processing of new, high-priority IO requests. By segmenting the reading or writing of large files, IO processing module 134 ensures that new, high-priority IO requests are processed promptly while a large file is being read or written.

[0042] In this embodiment, the computing node 110 sends an IO request to the storage node 120 via a network protocol. The distributed file system 130 receives and processes the IO request. The IO priority scheduling module 132 divides and schedules the IO request according to the priority information, thereby completing the processing of the IO request and the transmission of data between the computing node 110 and the storage node 120.

[0043] exist Figure 1 In the distributed system architecture shown and described, the IO priority scheduling module 132 and the IO processing module 134 are deployed on each storage node 120. In this mode, each storage node 120 has complete IO request processing capabilities. When a storage node receives a read or write request, the local IO priority scheduling module inserts the IO request into the corresponding position in the local request queue according to its priority characteristics. At the same time, the IO processing module 134 on the node breaks down large-granularity IO requests into smaller-granularity sub-IO requests. This architecture enables all storage nodes to independently complete the priority scheduling of IO requests and data processing.

[0044] In addition to the above-described pattern, distributed system 100 also has another architecture for centralized handling of IO requests. In this architecture, distributed system 100 also includes a master storage node 122. Figure 2 This is a schematic diagram of another distributed system architecture provided in this application embodiment. In this architecture, the distributed file system 130 also includes various storage nodes 120, but the core IO request processing function is completed by the master storage node 122. In a distributed system 100 consisting of multiple storage nodes 120, one storage node 120 is designated as the master storage node 122; the IO priority scheduling module 132 and the IO processing module 134 mainly run on this master storage node 122, rather than running on all storage nodes 120.

[0045] In this architecture, when storage node 120 receives an I / O request, it forwards these I / O requests to the primary storage node 122. The I / O priority scheduling module 132 on the primary storage node 122 prioritizes and schedules all I / O requests from the compute node 110. The I / O processing module on the primary storage node 122 is responsible for breaking down large-granularity I / O requests into multiple smaller-granularity sub-I / O requests. After completing these processes, the primary storage node 122 returns a response containing operation instructions to the corresponding storage node 120. The storage node 120 inserts the processed I / O request into its local request queue and executes the specific read / write operations.

[0046] This master storage node coordination mechanism achieves unified execution of IO priority management across the entire distributed system through centralized request scheduling and processing. These two architectural patterns provide different design options for distributed systems of different scales and application scenarios.

[0047] Figure 3 This is a flowchart of a request processing method provided in an exemplary embodiment of this application. This method can be executed by a distributed system or by a combination of computing nodes and storage nodes. Figure 3 As shown, the method includes the following steps.

[0048] Step 320: The compute node sends a target I / O request to the storage node.

[0049] In some embodiments, the computing process within a compute node serves as the basic unit for executing specific tasks, and its operation is accompanied by continuous data access requirements. The compute process determines the target file to be operated on and its access type (read or write) based on its current execution stage, and generates a corresponding target I / O request for the target file. The compute node sends these target I / O requests to the storage node via a network protocol, where they are uniformly received and processed by the distributed file system. Optionally, in the scenario of machine learning model training, this compute process can be implemented as a training process.

[0050] In some embodiments, the training process in a compute node determines the target file to be accessed next and its operation type (read or write) based on the current training state of the machine learning model, and sends the corresponding target I / O request. The compute node sends the target I / O request to the storage node via a network protocol, where it is processed by the distributed file system running on the storage node. If the target I / O request is a read request, it typically needs to obtain the dataset required for training or restore the model checkpoints required for training; if the target I / O request is a write request, it is mainly used to save model checkpoints or log data generated during training, or other files. The compute node sends the target I / O request to the storage node via a network protocol, where it is received by the distributed file system running on that storage node. Subsequently, the target I / O request is processed by the distributed file system and enters the priority scheduling process.

[0051] In this embodiment, the training of the machine learning model includes four common types of I / O requests that significantly impact training performance: dataset read, model checkpoint read, dataset prefetch, and model checkpoint write. These four types of I / O requests are divided into three priorities: P0, P1, and P2, with P0 having the highest priority and P2 having the lowest priority.

[0052] For I / O requests such as dataset reading, since they directly provide the GPU with the data needed for the next training iteration, if the data is not ready, the GPU needs to stop training and wait for the dataset to be loaded before it can continue training. During the time the dataset is loaded, the GPU is in a blocked state, so the priority is P0.

[0053] For I / O requests such as model checkpoint reads, they usually occur when the previous training task ends and the next training task begins. The compute node needs to read the model checkpoint at the start of training to obtain the model state saved during the training process at the end of the last training session, and continue training from the last training session. If this operation is not completed, the GPU cannot start or continue to perform computation, and the GPU is in a blocked state. Therefore, the priority is P0.

[0054] For I / O requests like dataset prefetching, which are used to facilitate subsequent GPU training, the dataset is prefetched and transmitted to the local compute node. The GPU can directly access the data locally, reducing waiting time and preventing the GPU from being blocked. Therefore, the priority is P1.

[0055] For I / O requests such as model checkpoint writes, the current model state is usually saved to the storage node periodically during training. This operation is generally designed to be executed asynchronously in the background, which means that the GPU does not need to wait for the write to be completed before continuing subsequent computation tasks. The GPU is not in a blocked state, so the priority is P2.

[0056] Dataset and checkpoint read operations, which directly impact the training process, are given the highest priority to ensure the GPU can continuously acquire data. Dataset prefetching, as an optimization technique for the training process, is classified as medium priority, ensuring data preparation for the training pipeline while avoiding excessive prefetching that consumes critical resources. Checkpoint writes, as a fault-tolerance mechanism, are categorized as the lowest priority to ensure that storage tasks are completed without interfering with the normal training process. This approach optimizes storage resource allocation efficiency while ensuring the continuity of training tasks.

[0057] I / O requests are prioritized based on their impact on the GPU training process: P0 priority I / O requests directly affect GPU training, such as data reading; incomplete data reading will cause the GPU to temporarily stop training; P1 priority I / O requests will not block the current GPU training, but will affect the efficiency of subsequent training; P2 priority I / O requests will not affect GPU training. Therefore, although data prefetching and model checkpoint writing do not block the current GPU training, they are still prioritized.

[0058] By establishing a three-tiered priority classification system, the impact of I / O requests can be finely differentiated. This hierarchical mechanism not only ensures that critical I / O requests that directly cause processor blocking receive immediate responses, but also effectively identifies and handles medium-priority I / O requests with potential blocking risks, while allowing background tasks that do not block the processor to be scheduled appropriately. This priority division method enables the distributed file system to adopt differentiated scheduling strategies based on the actual impact of different types of I / O requests on the training process, reducing training blocking problems caused by overly simplistic priority division and improving the overall performance of the distributed system.

[0059] Step 340: The storage node inserts the target IO request into the request queue.

[0060] Upon receiving a target I / O request, the distributed file system (DFS) first parses the request content to determine the type and characteristics of the target file. The DFS then identifies the priority of the target I / O request by obtaining the priority information associated with the target file. When a predefined priority (such as P0, P1, or P2) is identified, the DFS inserts the target I / O request into the appropriate position in the request queue based on that priority information. If the target I / O request does not carry explicit priority information, it is defaulted to the highest priority P0 for processing.

[0061] To illustrate, when the distributed file system receives a target I / O request to read a model checkpoint, it first parses the target I / O request to obtain its priority information and classifies it as P0 priority. This priority indicates that the I / O request needs to be processed immediately to avoid blocking the training tasks of the computing nodes.

[0062] When a preset priority information is detected, the distributed file system inserts the target I / O request into the corresponding position in the request queue according to that priority information. If the model checkpoint reads a priority of P0, it will be inserted by the distributed file system before all I / O requests with a priority of P1 in the request queue; for example... Figure 4 As shown, Figure 4 This is a diagram illustrating the request queue in a distributed file system. For this type of IO request, it will be inserted into the request queue at position P0, ensuring that it is processed before existing requests in the queue with priority P1 (dataset prefetch IO requests) and priority P2 (checkpoint write IO requests).

[0063] By establishing a default priority mechanism, the system fault tolerance problem in scenarios where priority information is missing is effectively solved. This prevents critical I / O requests from being unexpectedly delayed due to missing priority information, and in particular avoids GPU resource idleness caused by training data supply interruptions. This design ensures that when an I / O request lacks a priority identifier due to reasons such as abnormal distributed file system configuration, version incompatibility, or transmission errors, the distributed file system can treat it as a highest-priority I / O request according to the most conservative principle.

[0064] In this embodiment, the training process within the compute node does not prioritize system management I / O requests. These I / O requests include periodically recorded system metric data (such as CPU, memory, GPU, and disk usage), and their generation frequency and data volume are much smaller than the dataset read and checkpoint read / write operations directly involved in model training. Since these system management requests do not block GPU training activities and do not directly participate in training computation, the distributed file system treats them uniformly as default P0 priority I / O requests.

[0065] Step 360: The storage node processes at least one IO request in the request queue in sequence.

[0066] The distributed file system executes read and write operations on I / O requests in the request queue in order of priority, from high priority to low priority. The distributed file system prioritizes high-priority I / O requests to ensure that critical operations such as reading training data can be completed in a timely manner. Low-priority I / O requests, such as writing model checkpoints, are executed during the intervals between high-priority I / O request processing or when the request queue is idle.

[0067] During processing, the distributed file system (DFS) employs different handling methods for read and write requests. For read requests, the DFS retrieves the target file from the storage node based on parameters such as the file path, data offset, and read length specified in the read request. These target files may be fragments of the dataset required for training or portions of model checkpoints. For write requests, the DFS saves the received data to the corresponding storage node according to the specified storage path and distribution strategy. Regardless of whether it's a read or write operation, after each IO request is processed, the DFS generates a response containing the operation status (success or failure) and necessary data content, and returns it to the compute node that initiated the IO request via the network protocol.

[0068] In summary, an IO priority scheduling mechanism was designed. The distributed file system uses the IO priority scheduling module to prioritize and schedule IO requests, processing at least one IO request in the request queue sequentially. Combined with the IO processing module's decomposition of large-granularity IO requests, this prioritizes IO requests that could impact the real-time performance of the training task. While ensuring the continuous execution of the training task, it also processes IO requests that do not affect real-time performance in parallel. This reduces the waiting time for computing devices to perform training tasks and improves the overall resource utilization and task execution efficiency of the distributed system.

[0069] In this embodiment, the priority of I / O requests is determined based on their impact on the GPU training process. A priority allocation mechanism based on processor running state is established, directly linking the priority of I / O requests to the actual training state of the computing node. This design enables the distributed file system to accurately identify critical I / O requests that could block the training process (such as real-time data reading) and assign them the highest scheduling priority. At the same time, the distributed file system can continue to process non-blocking I / O operations (such as model checkpoint writing), thereby effectively reducing the processor's idle waiting time.

[0070] In an optional embodiment, the distributed file system obtains the priority information of the target IO request through the extended attributes of the target file. Figure 5 This is a flowchart of a method for processing requests in a distributed system according to another exemplary embodiment of this application. This method can be executed by the distributed system itself, or by other terminals, servers, or a combination of terminals and servers. Figure 5 As shown, step 320 above can also be implemented as steps 310 to 322 below; step 362 is added after the above steps.

[0071] Step 310: The compute node provides the storage node with the priority information of the target I / O request.

[0072] In some embodiments, the computing process running on the compute node processes different types of target files based on task attributes when executing a task. The computing process analyzes the data access requirements of the current task execution stage to determine the impact of various file access operations on the task, and accordingly assigns appropriate priorities to upcoming target I / O requests. In the specific scenario of machine learning model training, this computing process is specifically implemented as a training process.

[0073] In some embodiments, when a single compute node starts a training process, the training process accesses the target file to be read or written. Optionally, the target file includes one or more types of files. When the compute node needs to read or write different files, the training process initiates different I / O requests and determines whether each type of I / O request will block the processor's training execution based on the internal logic of the different I / O requests, and assigns priority to the files to be accessed accordingly.

[0074] In this embodiment, the training process prioritizes four types of I / O requests—dataset read, model checkpoint read, dataset prefetch, and model checkpoint write—based on whether they will block the processor. After prioritizing these I / O requests, extended attributes (xattr) need to be set for the target file corresponding to the target I / O request. These extended attributes are independent of the file itself; they are optional, custom-defined, and named data added by the user or application. The priority information of the target I / O request can be stored in the extended attributes of the target file. Figure 6 As shown, Figure 6 This is a diagram illustrating the setting of extended attributes for a target file in a distributed system. This data is "optional," meaning it does not affect the core content or basic access to the file, but rather provides additional contextual information for user or application needs (such as IO request priority handling in this application). In this application, the priority information of the target IO request is stored in the extended attributes corresponding to the target file. Extended attributes are typically associated with the file's inode, which is the core metadata structure of a distributed file system that describes a file. Its built-in attributes usually include fixed information such as file size, permissions, owner, timestamps (creation, modification, access), and pointers to the locations of the data that make up the file on the disk.

[0075] The training process calls the extended attributes of the target file, uses the priority information as an extended attribute of the target file, and transmits it to the distributed file system via network protocol.

[0076] The storage method based on extended attributes makes priority information a built-in attribute of the target file. This mechanism maintains the flexibility and scalability of storage scheduling without changing the application code, and provides convenience for policy optimization in different training scenarios.

[0077] To illustrate, the training process Job1 is reading the dataset dataset.data. For this dataset read I / O request, the GPU needs to stop training and wait for the dataset to finish loading before training can continue. At this time, the GPU is in a blocked state, therefore its priority is P0. The training process Job1 sets extended attributes for the dataset read I / O request (dataset.dat) and transmits the request command via the network protocol: setxattr(“dataset.dat”, “io_priority”, “P0”, …) to inform the distributed file system that the upcoming I / O has a priority of P0. The training process Job2 is writing the model checkpoint `checkpoint.pt`. For this I / O request to read the model checkpoint, the compute node needs to read the model checkpoint at the start of training to obtain the model state saved during the last training session and continue training from the last session. At this time, the GPU is in a blocked state, so its priority is P2. The training process Job2 sets extended attributes for the I / O request to write the model checkpoint `checkpoint.pt` and transmits the request command via the network protocol: `setxattr(“checkpoint.pt”, “io_priority”, “P2”, …)` to inform the distributed file system that the priority of the I / O to be accessed is P2.

[0078] Step 312: The storage node writes the priority information of the target IO request into the extended attributes of the target file.

[0079] In some embodiments, after determining the target I / O request and its priority information, the computing process sends the priority information of the target file corresponding to the target I / O request to the distributed file system via a network protocol. Upon receiving the priority information sent by the computing process, the distributed file system saves it as an extended attribute to the target file. This priority information includes the attribute name (e.g., "io_priority") that the target file needs to set and its corresponding specific value (e.g., "P0"). In this way, the computing process can set corresponding priority information for different file access operations according to its task attributes, and the distributed file system can optimize the scheduling order of I / O requests by reading this preset priority information. In the specific scenario of machine learning model training, this computing process is specifically implemented as a training process.

[0080] After determining the target I / O request and its priority information, the training process in the compute node sends the priority information of the target I / O request to the distributed file system via a network protocol. The distributed file system receives the priority information of the target I / O request and saves the priority information to the extended attributes of the target I / O request. The priority information includes the name of the priority attribute to be set for the target file (such as "io_priority") and its specific value (such as "P0").

[0081] After receiving a priority information setting request, the distributed file system needs to save the priority information to the metadata area corresponding to the target file. Specifically, the priority information (e.g., named "io_priority") is saved to the extended attribute space of the target file's inode.

[0082] Continuing the previous example, after receiving multiple `setxattr` requests, the distributed file system saves the priority of the upcoming I / O access in the corresponding inode attribute. This priority information is then saved for use by the subsequent I / O priority scheduling module.

[0083] Step 322: The storage node obtains priority information from the extended attributes.

[0084] When a distributed file system running on a storage node receives a target I / O request, it first parses the request to identify the type and path information of the target file. The distributed file system then queries the inode metadata structure corresponding to the target file, accessing its extended attribute area to find the preset I / O priority attribute information (such as "io_priority"). If valid priority information (such as P0, P1, or P2) is detected in this area, the distributed file system inserts the I / O request into the appropriate position in the request queue based on this priority information: for example, a P0 priority I / O request will be inserted before all P1 priority requests in the request queue; if no preset priority information is detected, the I / O request is defaulted to the highest priority P0 for processing.

[0085] Specifically, the training process also supports directly setting the priority information of the target file in the extended attribute area of ​​the target file's inode, and the distributed file system also supports a priority management mechanism based on a multi-request queue. The distributed file system parses the priority information in the target file and inserts I / O requests of different priorities into their respective dedicated processing request queues.

[0086] Indicatively, such as Figure 7 As shown, Figure 7This is a schematic diagram illustrating the arrangement of IO requests in a distributed file system provided by an exemplary embodiment of this application. The training process sequentially sets extended attributes for file 1, file 2, and file 3, and assigns priorities to them based on their file types. After prioritizing the target files, the training process sends the target IO requests corresponding to the target files to the distributed file system for IO queuing. The distributed file system inserts IO requests of different priorities into their respective dedicated processing request queues based on the priority information of the target files corresponding to the IO requests. IO requests with priority P0 are inserted into the dedicated P0 request queue, IO requests with priority P1 are inserted into the dedicated P1 request queue, and IO requests with priority P2 are inserted into the dedicated P2 request queue. When the distributed file system completes the processing of any IO request, it checks and processes the IO requests in the request queues corresponding to higher priority IO requests first, according to a strict priority order, to ensure that critical tasks receive timely responses.

[0087] Priority information is directly obtained from the extended attributes of the target file, establishing an efficient and stable priority determination mechanism. Obtaining priority information through the standard extended attribute interface enables the distributed file system to set differentiated scheduling strategies for data files at different training stages, achieving more granular resource control.

[0088] Step 362: If the currently processed IO request meets the splitting condition, the storage node splits the IO request into at least two sub-IO requests.

[0089] When a target I / O request meets the splitting criteria, the distributed file system in the storage node will split the target I / O request into multiple target sub-I / O requests. Specifically, after identifying such large-granularity target I / O requests, the distributed file system will split such large-granularity target I / O requests into multiple small-granularity target sub-I / O requests according to a preset fixed size (e.g., 64KB).

[0090] The conditions for splitting a target IO request include one or more of the following: the amount of data read or written is greater than a threshold, the type of the target IO request conforms to the preset splitting type (e.g., splitting only IO requests for model checkpoint writes, or splitting only write requests), and the priority belongs to the preset priority (e.g., it can be only for P1 and P2).

[0091] After splitting, the distributed file system (DFS) sequentially sends these sub-IO requests into a request queue. The DFS then processes the target IO requests and target sub-IO requests in the queue. For read requests and sub-read requests, the DFS retrieves the target file from the storage node based on parameters such as the file path, data offset, and read length specified in the read and sub-read requests. These target files may be fragments of the dataset required for training or portions of model checkpoints. For write requests and sub-write requests, the DFS saves the received data to the corresponding storage nodes according to the specified storage path and distribution strategy. During execution, the DFS immediately checks the request queue for any newly arrived high-priority IO requests after processing each sub-IO request. After all target sub-IO requests have been processed, the DFS collects the execution results of each target sub-IO request and returns the complete processing results to the training process that initiated the target IO request via a network protocol. For target IO requests that were not split, the DFS directly returns the complete processing results to the training process that initiated the target IO request via a network protocol after processing.

[0092] To illustrate, when a compute node sends an I / O request for a model checkpoint write, and the distributed file system detects that the data volume of this write request exceeds a preset threshold, it initiates a splitting process. The distributed file system divides the complete checkpoint file data into 64KB blocks, generating multiple sub-write requests. These sub-write requests are added to a request queue sequentially, and the distributed file system processes them according to the priority of the request queue. After each 64KB data block is written, the current operation is paused, and the system checks and processes any high-priority I / O requests (such as dataset reads) that may exist in the request queue. Only after these high-priority I / O requests are processed can the writing of the next data block continue.

[0093] A dynamic request splitting mechanism is introduced. When the data volume of a target IO request is detected to be greater than or equal to a preset threshold, the distributed file system splits the target IO request into multiple sub-IO requests. This approach ensures both the continuous progress of large-scale data writing tasks and timely responses to critical training IO requests, thus maintaining high throughput while ensuring low latency. It effectively balances resource competition between batch data processing and real-time request response, enabling the distributed file system to dynamically adjust resource allocation strategies based on real-time load, ultimately improving the overall resource utilization and task execution efficiency of the distributed system. In summary, an IO priority scheduling mechanism is designed. The distributed file system uses an IO priority scheduling module to prioritize and schedule IO requests, processing at least one IO request in the request queue sequentially. Combined with the IO processing module's splitting of large-granularity IO requests, it prioritizes IO requests that may affect the real-time performance of the training task, ensuring the continuous execution of the training task while processing IO requests that do not affect real-time performance in parallel. This reduces the waiting time for computing devices to perform training tasks, improving the overall resource utilization and task execution efficiency of the distributed system.

[0094] In this embodiment, the storage node acquires and saves this priority information to the file's extended attribute area, thus associating the priority information with the file itself. This design ensures that all subsequent IO requests for the same file automatically obtain the correct priority information, reducing system overhead.

[0095] In particular, the implementation based on extended attributes offers excellent flexibility and scalability. The training process or the distributed file system can adjust the priority of specific types of I / O requests by modifying the extended attributes of files. Through this priority preservation method, the distributed file system optimizes the resource utilization efficiency of the entire distributed system while ensuring low-latency processing of high-priority I / O requests.

[0096] In an optional embodiment, the above request processing method can be completed collaboratively by sub-modules within the storage node. This method can be executed by a distributed system, or by other terminals, servers, or a combination of terminals and servers. Figure 5 This is a schematic diagram of a request processing procedure based on computing nodes and storage nodes in a distributed system, provided by an exemplary embodiment of this application. Figure 8 This is a schematic diagram illustrating a request processing procedure based on sub-modules in compute nodes and storage nodes, provided in an exemplary embodiment of this application. For example... Figure 8 As shown, the storage node contains an IO priority scheduling module and an IO processing module. Step 340 can also be implemented as step 342; step 362 can also be implemented as step 364.

[0097] Step 342: The IO priority scheduling module in the storage node inserts the target IO request into the request queue.

[0098] After the distributed file system receives a target I / O request, the I / O priority scheduling module first parses the request to determine the type and characteristics of the accessed file. This module identifies the priority level of the target I / O request by obtaining the priority information associated with the file. When a preset priority (such as P0, P1, or P2) is identified, the I / O priority scheduling module inserts the target I / O request into the corresponding position in the request queue according to this priority information. Specifically, this module maintains the priority order of requests in the request queue; for example, it inserts a detected I / O request with priority P0 before all I / O requests with priority P1 in the request queue. If the target I / O request does not carry explicit priority information, the I / O priority scheduling module will default to classifying it as the highest priority P0 for processing.

[0099] In this embodiment, the training process within the compute node does not prioritize system management I / O requests. These I / O requests include periodically recorded system metric data (such as CPU, memory, GPU, and disk usage), and their generation frequency and data volume are much smaller than the dataset read and checkpoint read / write operations directly involved in model training. Since these system management requests do not block GPU training activities and do not directly participate in training computation, the I / O priority scheduling module treats them all as the default P0 priority.

[0100] Specifically, the IO priority scheduling module is also responsible for implementing a priority management mechanism based on multiple request queues. This module parses the priority information in IO requests and inserts IO requests of different priorities into their corresponding dedicated processing request queues. For example, IO requests with priority P0 are inserted into the dedicated P0 request queue, IO requests with priority P1 are inserted into the dedicated P1 request queue, and IO requests with priority P2 are inserted into the dedicated P2 request queue. After the distributed system completes the processing of any IO request, the IO priority scheduling module will check and process IO requests in the high-priority IO request queues first, according to a strict priority order, ensuring that critical tasks receive timely responses.

[0101] Step 364: If the currently processed IO request meets the splitting condition, the IO processing module in the storage node splits the IO request into at least two sub-IO requests.

[0102] When a target I / O request meets the splitting criteria, the storage node will split the target I / O request into multiple target sub-I / O requests. Specifically, this splitting operation is performed by the I / O processing module in the distributed file system. After identifying such large-granularity target I / O requests, this module will split such large-granularity target I / O requests into multiple small-granularity target sub-I / O requests according to a preset fixed size (e.g., 64KB).

[0103] The conditions for splitting a target IO request include one or more of the following: the amount of data read or written is greater than a threshold, the type of the target IO request conforms to the preset splitting type (e.g., splitting only IO requests for model checkpoint writes, or splitting only write requests), and the priority belongs to the preset priority (e.g., it can be only for P1 and P2).

[0104] After splitting, the IO processing module sequentially sends these sub-IO requests into the request queue. The IO processing module then processes the target IO requests and target sub-IO requests in the request queue. For read requests and sub-read requests, the IO processing module retrieves the target file from the storage node based on parameters such as the file path, data offset, and read length specified in the read request and sub-read request. These target files may be fragments of the dataset required for training or parts of the model checkpoints. For write requests and sub-write requests, the IO processing module saves the received data to the corresponding storage node according to the specified storage path and distribution strategy. During execution, the IO processing module immediately checks the request queue for any newly arrived high-priority IO requests after processing each sub-IO request. After all target sub-IO requests have been processed, the IO processing module collects the execution results of each target sub-IO request and returns the complete processing results to the training process that initiated the target IO request via the network protocol. After processing unsplit target IO requests, the IO processing module directly returns the complete processing results to the training process that initiated the target IO request via the network protocol.

[0105] To illustrate, when a compute node sends an I / O request for model checkpoint write, the I / O processing module initiates a splitting process when it detects that the data volume of the write request exceeds a preset threshold. This module divides the complete checkpoint file data into 64KB blocks, generating multiple target sub-write requests. These target sub-write requests are added to the request queue sequentially, and the I / O processing module processes them according to the priority order of the request queue: after each 64KB data block is written, the current operation is paused, and the module checks and processes any high-priority I / O requests (such as dataset read requests) that may exist in the request queue. Only after these high-priority I / O requests are processed can the writing of the next data block continue. In summary, an I / O priority scheduling mechanism is designed. The distributed file system implements priority division and scheduling of I / O requests through the I / O priority scheduling module, processing at least one I / O request in the request queue sequentially. Combined with the I / O processing module's splitting of large-granularity I / O requests, this achieves priority processing of I / O requests that may affect the real-time performance of the training task, ensuring the continuous execution of the training task while processing background I / O requests that do not affect the real-time performance of training in parallel. This reduces the waiting time for computing devices to perform training tasks, and improves the overall resource utilization and task execution efficiency of the distributed system.

[0106] In this embodiment, an efficient IO request processing mechanism is constructed through the collaborative work of the IO priority scheduling module and the IO processing module. The IO priority scheduling module is responsible for identifying the priority of requests and inserting them into the corresponding request queue, ensuring that critical IO requests are processed first. The IO processing module is responsible for breaking down large-granularity IO requests into smaller-granularity sub-IO requests, enabling the distributed file system to flexibly switch execution between different IO requests. The two modules work together to ensure timely response to high-priority IO requests while maintaining the distributed file system's ability to process batch data, thereby effectively improving the overall performance of the distributed system.

[0107] Figure 9 This is a structural block diagram of a computing device in a distributed system provided by an exemplary embodiment of this application, such as... Figure 9 As shown, the device includes: The computing process module 920 is configured to generate target I / O requests, and each target I / O request has priority information. The sending module 940 is configured to send a target I / O request to a storage device, the storage device being used to receive the target I / O request; insert the target I / O request into a request queue based on the priority information of the target I / O request; and process at least one I / O request in the request queue in sequence.

[0108] In an optional embodiment, the computing process module is configured to obtain priority information of the target I / O request based on whether the processor in the computing device is blocked during the processing of the target I / O request, and to prioritize the target I / O request.

[0109] In an optional embodiment, the computing process module is further configured to divide the priority information of the target I / O request into at least two of a first priority, a second priority, and a third priority based on whether the processor in the computing device is blocked during the processing of the target I / O request. During the processing of I / O requests corresponding to the first priority, the processor in the computing device may be blocked; during the processing of I / O requests corresponding to the second priority, the processor in the computing device may be blocked; during the processing of I / O requests corresponding to the third priority, the processor in the computing device will not be blocked.

[0110] The first priority P0 IO requests include IO requests for reading datasets or checkpoints. A dataset is a collection of sample data used for training an AI model, and a checkpoint is a collection of files containing key state information of the model saved during training. The second priority P2 IO requests include IO requests for pre-reading datasets. The third priority P3 IO requests include IO requests for writing checkpoints.

[0111] In an optional embodiment, the sending module is also configured to provide priority information of the target IO request to the storage node; The compute process module is also configured to write the priority information of the target I / O request into the extended attributes of the target file within the storage node; where the target file is the file accessed by the target I / O request.

[0112] Figure 10 This is a structural block diagram of a storage device in a distributed system provided in an exemplary embodiment of this application, such as... Figure 10 As shown, the device includes: The receiving module 1020 is configured to receive a target I / O request sent by the computing device; The IO priority scheduling module 1040 is configured to insert the target IO request into the request queue based on the priority information of the target IO request; The IO processing module 1060 is configured to process at least one IO request in the request queue in sequence.

[0113] In an optional embodiment, the IO priority scheduling module is further configured to insert the target IO request into the request queue according to a first priority if the target IO request has no priority information.

[0114] In an optional embodiment, the IO priority scheduling module is further configured to obtain priority information of the target IO request, including: The IO priority scheduling module is also configured to obtain the priority information of the target IO request from the extended attributes of the target file accessed by the target IO request.

[0115] In an optional embodiment, the IO processing module is further configured to split the target IO request into at least two sub-IO requests if the amount of data written in the target IO request is greater than or equal to a preset threshold, wherein each of the at least two sub-IO requests is less than the preset threshold.

[0116] In summary, this embodiment employs an IO priority scheduling mechanism. The distributed file system prioritizes and schedules IO requests through its IO priority scheduling module, processing at least one IO request in the request queue sequentially. Combined with the IO processing module's segmentation of large-granularity IO requests, this prioritizes IO requests that could impact the real-time performance of the training task. While ensuring the training task continues, it also processes background IO requests that do not affect real-time training in parallel. This reduces the waiting time for computing devices to perform training tasks and improves the overall resource utilization and task execution efficiency of the distributed system.

[0117] It should be noted that the IO request processing device provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the IO request processing device and the IO request processing method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0118] This application also provides a distributed system comprising computing nodes and storage nodes. The computing nodes are configured to send target I / O requests to the storage nodes; the storage nodes are configured to receive the target I / O requests; insert the target I / O requests into a request queue based on priority information of the target I / O requests; and process at least one I / O request in the request queue sequentially. It should be noted that the computer device can be as follows: Figure 11 The computer equipment provided.

[0119] Please refer to Figure 11This illustration shows a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Specifically, the computer device 1100 includes a Central Processing Unit (CPU) 1101, a system memory 1104 including Random Access Memory (RAM) 1102 and Read-Only Memory (ROM) 1103, and a system bus 1105 connecting the system memory 1104 and the CPU 1101. The computer device 1100 also includes a basic input / output system (I / O system) 1106 that facilitates the transfer of information between various devices within the computer, and a mass storage device 1107 for storing the operating system 1113, application programs 1114, and other program modules 1115.

[0120] The basic input / output system 1106 includes a display 1108 for displaying information and an input device 1109 for user input, such as a mouse or keyboard. Both the display 1108 and the input device 1109 are connected to the central processing unit 1101 via an input / output controller 1110 connected to the system bus 1105. The basic input / output system 1106 may also include the input / output controller 1110 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1110 also provides output to a display screen, printer, or other types of output devices.

[0121] The mass storage device 1107 is connected to the central processing unit 1101 via a mass storage controller (not shown) connected to the system bus 1105. The mass storage device 1107 and its associated computer-readable media provide non-volatile storage for the computer device 1100. That is, the mass storage device 1107 may include computer-readable media (not shown) such as a hard disk or a compact disc-read-only memory (CD-ROM) drive.

[0122] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, flash memory or other solid-state storage technologies, CD-ROM or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 1104 and mass storage device 1107 described above can be collectively referred to as memory.

[0123] The memory stores one or more programs, which are configured to be executed by one or more central processing units 1101. The one or more programs contain instructions for implementing the above-described model prediction task or the machine translation model-based translation method. The central processing unit 1101 executes the one or more programs to implement the model prediction task or the machine translation model-based translation method provided in the above-described method embodiments.

[0124] According to various embodiments of this application, the computer device 1100 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1100 can be connected to the network 1112 via the network interface unit 1111 connected to the system bus 1105, or the network interface unit 1111 can be used to connect to other types of networks or remote computer systems (not shown).

[0125] The memory further includes one or more programs stored in the memory, and the one or more programs include steps executed by a computer device in the running method for performing the model prediction task provided in the embodiments of this application.

[0126] On the other hand, embodiments of this application provide a computing node, which includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the request processing method in the distributed system provided in the embodiments of this application.

[0127] On the other hand, embodiments of this application provide a storage node, which includes a processor and a memory, wherein the memory stores at least one program, which is loaded and executed by the processor to implement the request processing method in the distributed system provided in the embodiments of this application.

[0128] On the other hand, embodiments of this application provide a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the IO request processing method in a distributed system as described in the embodiments of this application above.

[0129] On the other hand, embodiments of this application provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the IO request processing method in a distributed system of a chip as described in the embodiments of this application above.

[0130] On the other hand, embodiments of this application provide a computer device, which includes the processor described above. Optionally, the processor is a GPU. The computer device can be at least one of a portable computer, a desktop computer, a server, a server cluster, an artificial intelligence (AI) computing cluster, and a cloud computing cluster. The AI ​​computing cluster can also be simply referred to as an intelligent computing cluster or a smart computing cluster.

[0131] It should be noted that the data collection and processing in this application should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0132] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0133] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A request processing method in a distributed system, characterized in that, The distributed system includes computing nodes and storage nodes, and the method includes: The compute node sends a target input / output (I / O) request to the storage node; The storage node receives the target I / O request; based on the priority information of the target I / O request, inserts the target I / O request into the request queue; and processes at least one I / O request in the request queue in sequence. The at least one IO request is sorted in the request queue according to its priority from high to low.

2. The method according to claim 1, characterized in that, The priority information of the target I / O request is determined by the computing node based on whether the processor in the computing node is blocked during the processing of the target I / O request.

3. The method according to claim 2, characterized in that, The priority information includes at least two of the following: first priority, second priority, and third priority; During the processing of the IO request corresponding to the first priority, the processor in the computing node will be blocked; During the processing of the IO request corresponding to the second priority, the processor in the computing node may be blocked; During the processing of the IO request corresponding to the third priority, the processor in the computing node will not be blocked.

4. The method according to claim 3, characterized in that, The first priority IO request includes IO requests for reading datasets or checkpoints, wherein the dataset is a collection of sample data used for training an artificial intelligence model, and the checkpoint is a collection of files containing key state information of the model saved during the training process of the artificial intelligence model. The second priority IO requests include IO requests for pre-reading the dataset; The third priority IO requests include IO requests for writing checkpoints.

5. The method according to any one of claims 1 to 4, characterized in that, The storage node inserts the target IO request into the request queue based on the priority information of the target IO request, including: If the target IO request has no priority information, the storage node inserts the target IO request into the request queue according to the first priority.

6. The method according to any one of claims 1 to 4, characterized in that, Before the compute node sends a target input / output (I / O) request to the storage node, the following steps are also included: The compute node provides the storage node with the priority information of the target I / O request; The storage node obtains the priority information of the target IO request.

7. The method according to claim 6, characterized in that, The compute node provides the storage node with priority information for the target I / O request, including: The compute node writes the priority information of the target I / O request into the extended attributes of the target file within the storage node; The storage node obtains the priority information of the target IO request, including: The storage node obtains the priority information of the target I / O request from the extended attributes of the target file accessed by the target I / O request; The target file is the file accessed by the target I / O request.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: If the amount of data written in the target IO request is greater than or equal to a preset threshold, the storage node splits the target IO request into at least two sub-IO requests, and each of the at least two sub-IO requests is less than the preset threshold.

9. A request processing method in a distributed system, characterized in that, The distributed system includes computing nodes and storage nodes, and the method is executed by the computing nodes. The method includes: Generate a target I / O request, which has priority information. The target I / O request is sent to the storage node, which receives the target I / O request; based on the priority information of the target I / O request, the target I / O request is inserted into the request queue; at least one I / O request in the request queue is processed in sequence.

10. A request processing method in a distributed system, characterized in that, The distributed system includes computing nodes and storage nodes, and the method is executed by the storage nodes. The method includes: Receive the target I / O request sent by the computing node; Based on the priority information of the target I / O request, the target I / O request is inserted into the request queue; At least one I / O request in the request queue is processed sequentially.

11. A distributed system, characterized in that, The distributed system includes computing nodes and storage nodes; The computing node is used to send a target I / O request to the storage node; The storage node is used to receive the target IO request; Based on the priority information of the target I / O request, the target I / O request is inserted into the request queue; At least one I / O request in the request queue is processed sequentially.

12. A computing node, characterized in that, The computing node includes a processor and a memory, wherein the memory stores at least one program, which is loaded and executed by the processor to implement the request processing method in the distributed system as claimed in claim 9.

13. A storage node, characterized in that, The storage node includes a processor and a memory, wherein the memory stores at least one program, which is loaded and executed by the processor to implement the request processing method in the distributed system as claimed in claim 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one program, which is loaded and executed by a processor to implement the request processing method in the distributed system as described in any one of claims 9 to 10.

15. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the request processing method in a distributed system as described in any one of claims 9 to 10.

Images