File access processing method and system and electronic equipment

Abstract

The invention provides a file access processing method and system and electronic equipment, and belongs to the technical field of computers, the method is applied to a host end, and the method comprises the steps that a request data packet in a target data channel is received through a target working thread, and the target data channel is determined from initial data channels by a computing node; and analyzing a file access operation indicated by the request data packet through the target working thread, obtaining target data and returning the target data to the computing node. According to the method, the request of the specific channel is specially processed through the target working thread at the host end, a thread-level parallel-based file request processing mechanism is achieved, blocking caused by single-thread processing is avoided, and therefore the throughput and efficiency of the host end responding to the high-concurrency file access request of the computing node are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a file access processing method, system, and electronic device. Background Technology

[0002] With the rapid growth in the size of large language model parameters, the size of model files and training datasets often increases rapidly as well, thereby increasing the requirements for data loading and access efficiency of computing clusters.

[0003] To meet the above requirements, existing technologies typically employ a host relay approach: the host first reads the complete model file from the hard drive into its own memory, and then transmits the entire file to the local storage or memory of the computing node via a communication link; subsequently, the computing node performs inference or training tasks based on the complete file in its local storage.

[0004] However, the above-mentioned existing technical solutions have obvious drawbacks: since computing nodes must wait for the complete file transfer to be completed and stored locally before they can start processing, and file transfer tasks are usually processed serially by a single process on the host side, the data transmission and processing latency is high when facing large model file access requests, which cannot meet the computing nodes' high-concurrency data reading requirements and seriously restricts the overall execution efficiency of computing tasks. Summary of the Invention

[0005] This invention provides a file access processing method, system, and electronic device to overcome the deficiencies in the prior art, thereby significantly improving the throughput and efficiency of the host side in responding to high-concurrency file access requests from computing nodes.

[0006] This invention provides a file access processing method, applied on a host side, comprising: The target worker thread receives request data packets from the target data channel; wherein, the request data packets are sent by the computing node through the target data channel, and the target data channel is determined by the computing node from the initial data channel; The target worker thread parses the file access operation indicated by the request data packet to obtain the target data; Return a response data packet carrying the target data to the computing node.

[0007] According to a file access processing method provided by the present invention, the initial data channel corresponds one-to-one with the initial working thread, and the target working thread is the working thread corresponding to the target data channel.

[0008] A file access processing method provided by the present invention further includes: Assign a unique initial list of accessible directories to each of the initial worker threads; Determine the list of target accessible directories corresponding to the target worker thread.

[0009] According to a file access processing method provided by the present invention, the step of parsing the file access operation indicated by the request data packet through the target worker thread to obtain target data includes: Parse the request data packet to obtain the target file path corresponding to the file access operation; When it is determined that the target file path is in the target accessible directory list, the target data is read from the storage medium according to the file access operation.

[0010] According to a file access processing method provided by the present invention, when the target file path is not in the target accessible directory list, the method further includes: Return a null value to the computing node.

[0011] This invention also provides a file access processing method, applied to a computing node, comprising: Obtain the file access request initiated by the computing program, and encapsulate the file access request into a request data packet; A target data channel is determined from multiple initial data channels, and the request data packet is sent to the corresponding target worker thread on the host through the target data channel; wherein, the target worker thread is determined by the host from the initial worker threads, and the initial data channel corresponds one-to-one with the initial worker thread; Receive the response data packet carrying the target data returned by the host; The target data is returned to the calculation program.

[0012] According to a file access processing method provided by the present invention, determining the target data channel from the plurality of initial data channels includes: Obtain the load occupancy parameters for each of the initial data channels; The target data channel is determined based on the load occupancy parameters.

[0013] According to a file access processing method provided by the present invention, the step of obtaining a file access request initiated by a computing program includes: The calculation program initiates a file access request to the FUSE module; The file access request is received through the FUSE module.

[0014] The invention also provides a file access processing system, including the following modules: A data receiving module is used to receive request data packets from a target data channel through a target worker thread; wherein the request data packets are sent by the computing node through the target data channel, and the target data channel is determined by the computing node from an initial data channel; The request processing module is used to parse the file access operation indicated by the request data packet through the target worker thread and obtain the target data; The response feedback module is used to return a response data packet carrying the target data to the computing node.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the file access processing methods described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the file access processing method as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the file access processing method as described above.

[0018] In summary, one or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: By utilizing a target worker thread on the host side to receive request data packets sent by the compute nodes through the target data channel, and having this target worker thread specifically responsible for parsing file access operations, obtaining target data, and returning response data packets, a thread-level parallel file request processing mechanism is implemented on the host side. This avoids the blocking and queuing delays that may be caused by single-threaded processing of multi-channel requests, enabling the host side to fully utilize multi-core computing resources to efficiently respond to high-concurrency data read demands from compute nodes, significantly improving the throughput and efficiency of the host side in responding to high-concurrency file access requests from compute nodes. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the architecture of a file access system based on FUSE provided by the present invention.

[0021] Figure 2 This is one of the flowcharts illustrating the file access processing method provided by the present invention.

[0022] Figure 3 This is the second flowchart of the file access processing method provided by the present invention.

[0023] Figure 4 This is the third flowchart of the file access processing method provided by the present invention.

[0024] Figure 5 This is the fourth flowchart of the file access processing method provided by the present invention.

[0025] Figure 6 This is the fifth flowchart of the file access processing method provided by the present invention.

[0026] Figure 7 This is the sixth flowchart of the file access processing method provided by the present invention.

[0027] Figure 8 This is one of the structural schematic diagrams of the file access processing system provided by the present invention.

[0028] Figure 9 This is the second schematic diagram of the file access processing system provided by the present invention.

[0029] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0031] It should be noted that in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships according to the accompanying drawings, are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the system or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Those skilled in the art will understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0032] The terms "first," "second," etc., used in this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0033] The following is combined with Figures 1 to 10 This invention describes the file access processing method, system, electronic device, storage medium, and computer program product provided by the present invention.

[0034] To facilitate understanding of the plan, firstly, refer to... Figure 1 , Figure 1 This is a schematic diagram of the architecture of a file access system based on FUSE provided by the present invention. Figure 1 As shown, the system mainly consists of a host, computing nodes, and a large-capacity storage medium. The host is connected to one or more computing nodes via physical communication links. The computing nodes can be NPU (Neural Network Processing Unit) computing nodes. The physical communication links include, but are not limited to, PCIe (Peripheral Component Interconnect Express), Ethernet, USB (Universal Serial Bus), etc.

[0035] The host computer can be a personal computer (PC), server, or workstation, running a workservice program whose core function is to proxy file access operations on behalf of the compute nodes. A large-capacity storage medium is connected to the host computer for persistent data storage. This medium includes, but is not limited to, hard disk arrays and solid-state drive (SSD) clusters, and stores deep learning model files, training datasets, or inference resource files. The compute nodes are primarily responsible for executing inference or training tasks for the deep learning models. While each compute node has its own memory, this memory is precious and typically only a few tens of gigabytes. Therefore, when processing extremely large-scale parameter models, frequent access to data on the large-capacity storage medium is necessary.

[0036] Based on the above system architecture, in a first aspect, this application provides a file access processing method, wherein the execution subject of the method is the host (or specifically the control system, processor, etc. running on the host).

[0037] In the specific process of file access, the establishment of communication channels and the initialization of worker threads are involved first. As an optional implementation, the host responds to the communication connection request initiated by the computing node, determines multiple initial data channels connected to the computing node, and creates an initial worker thread associated with each initial data channel.

[0038] Specifically, when a compute node powers on or an application initializes, it sends a connection signal to the host. Upon receiving this request, the workService on the host logically establishes multiple channels for transferring file data between the host and the compute node to meet the demands of high-concurrency data transmission; these are the initial data channels.

[0039] Meanwhile, to achieve parallel processing capabilities, the host computer creates an independent execution unit, or initial worker thread, for each initial data channel within its operating system. A one-to-one binding relationship is maintained between the initial worker thread and the initial data channel to ensure that data subsequently transmitted through a particular channel can be processed immediately by the dedicated worker thread, without having to queue in a single thread.

[0040] During the process of establishing a communication connection between the host and the compute node, it's important to note that the compute node will also establish a control channel with the host. This control channel is used to transmit connection control commands. Specifically, the communication architecture between the host and the compute node can adopt a "1+N" model, where "N" represents the multiple initial data channels used for concurrent file data transfer, and "1" represents this independent control channel. This control channel, distinct from the data channel, is specifically used to transmit system-level control signaling, such as connection handshake signals, heartbeat keep-alive packets, configuration update commands, or disconnection requests. By establishing an independent control channel, the control flow and data flow are decoupled, preventing the transmission of critical control commands from being blocked when the data channel is under full-load file transfer, thereby ensuring the stability and timeliness of the communication link management between the host and the compute node.

[0041] Based on this, refer to Figure 2 , Figure 2 This is one of the flowcharts illustrating the file access processing method provided by this invention. For example... Figure 2 As shown, the file access processing method includes the following steps: Step 101: Receive request data packets from the target data channel through the target worker thread; wherein, the request data packets are sent by the computing node through the target data channel, and the target data channel is determined by the computing node from the initial data channel.

[0042] The request data packet is the data carrier sent by the compute node when it needs to access file data. In specific application scenarios, when a deep learning computation program on the compute node initiates file access, such as reading large model parameters, the FUSE module on the compute node side intercepts the file access behavior and encapsulates it into a standard format request data packet. This request data packet can be in the form of a fusework data packet. It should be noted that before receiving the request data packet from the target data channel through the target worker thread, the host responds to the communication connection request initiated by the compute node, determines multiple initial data channels connected to the compute node, and creates an initial worker thread associated with each initial data channel.

[0043] The target data channel is determined by the computing node from the aforementioned multiple initial data channels. Specifically, before sending a data packet, the computing node selects one of the multiple available initial data channels as the target data channel for the current transmission according to a preset rule. The preset rule can be a channel load balancing strategy. The preset rule will be described in detail in subsequent embodiments.

[0044] Correspondingly, the target worker thread is the initial worker thread associated with the target data channel on the host side. It should be noted that the host side does not need to intervene in the selection process of the target data channel. Instead, it monitors each data channel, and once data arrives on the target data channel, the associated target worker thread will be activated and receive the request data packet.

[0045] Step 102: Obtain the target data by parsing the file access operation indicated by the request data packet through the target worker thread.

[0046] The host machine unpacks and analyzes the received request data packets through the target worker thread. File access operations include, but are not limited to, file reading, file attribute querying, directory traversal, and other operation instructions. Based on the parsed operation type and parameters such as the requested file path, offset, and read length, the target worker thread performs the actual file I / O operation on the mass storage medium connected to the host machine, where the mass storage medium can be a hard disk array.

[0047] For example, if a file access operation instructs the reading of a specific data segment from a large model file, the target worker thread calls the host operating system's file system interface to read the corresponding data segment from the local disk as the target data. This process is transparent to the compute node, which does not need to directly handle the complex logic of the underlying storage.

[0048] Step 103: Return a response data packet carrying the target data to the computing node.

[0049] After obtaining the target data, the target worker thread encapsulates the operation result and the read data content into a response data packet, which can be in the form of a fusesult data packet.

[0050] Subsequently, the target worker thread sends the response packet back to the compute node. Typically, the response packet returns along the original path, i.e., along the target data channel, or through a dedicated control channel. Ultimately, it is received by the FUSE module on the compute node side and submitted to the compute program, completing a full file proxy access process.

[0051] The file access processing method provided in this application involves setting up a target worker thread associated with a data channel on the host side to respond to request data packets sent by the computing node and obtain the target data in a proxy manner. This solution not only decouples the computing node from the underlying storage medium, allowing the computing node to obtain data on demand without storing the complete large model file, but also supports high-concurrency file access requests through a multi-channel and multi-threaded mechanism. This significantly improves the throughput of the host side in handling large-scale data I / O, effectively solving the technical problems of difficult deployment of large models and low access efficiency caused by limited NPU memory in the prior art.

[0052] As a further implementation of the above embodiments, the embodiments of this application further clarify the correspondence between the initial data channel and the initial working thread.

[0053] Specifically, during the stage where the host establishes a communication connection with the computing node and completes initialization, a one-to-one mapping relationship is maintained between the initial data channels determined by the host and the initial worker threads created. That is, each initial data channel is bound to a unique and independent initial worker thread, which is used specifically to handle the data transmission and reception tasks on that channel.

[0054] Accordingly, when a computing node determines a specific channel from multiple initial data channels as the target data channel for sending request packets, the target worker thread invoked by the host is the worker thread with which a pre-established correspondence has been established with that target data channel. This binding relationship can be statically configured or dynamically generated when the connection is established, for example, by maintaining a mapping table of channel IDs and thread IDs in memory.

[0055] By adopting an architecture design that pairs an initial data channel with an initial worker thread, this embodiment achieves true end-to-end parallel processing. Each data transmission physical path is supported by dedicated computing resources (i.e., worker threads) on the host side, avoiding thread blocking and resource contention issues that may occur when multiple channels reuse the same thread. When faced with intensive file access requests initiated by compute nodes, this one-to-one architecture ensures that the transmission bandwidth and processing power of each channel are maximized, thereby significantly reducing the overall latency of file access and guaranteeing the I / O stability of compute nodes running deep learning tasks.

[0056] As a further implementation of the above embodiments, this application embodiment introduces a permission configuration mechanism for worker threads before the host processes the request data packet. (See also...) Figure 3 , Figure 3 This is the second flowchart illustrating the file access processing method provided by this invention. For example... Figure 3 As shown, the method also includes the following steps: Step 201: Assign a unique initial list of accessible directories to each initial worker thread.

[0057] During the initialization phase of creating the initial worker thread on the host side, the host allocates an initial list of accessible directories for each initial worker thread based on a pre-defined configuration file or security policy. This initial list of accessible directories contains one or more file system paths used to limit the range of files that the worker thread is allowed to access. For example, the initial list of accessible directories can specify directories that only allow access to specific large model files or training datasets, such as " / data / models / gpt4" or " / data / datasets / image_net".

[0058] It should be noted that different initial worker threads can be assigned the same directory list to achieve concurrent access to the same set of resources; different initial worker threads can also be assigned different directory lists to achieve resource isolation, and this embodiment does not impose any restrictions on this.

[0059] Step 202: Determine the list of target accessible directories corresponding to the target worker thread.

[0060] Once the host identifies the target worker thread for processing the current request data packet, it further searches for and determines a list of target accessible directories associated with that target worker thread. This list of target accessible directories is the initial list of accessible directories allocated to the target worker thread in the preceding steps. This step ensures that the target worker thread's subsequent file access operations are subject to explicit permission constraints, thus improving file access security.

[0061] This application embodiment effectively prevents malicious programs or erroneous instructions on the computing node side from unauthorized access to sensitive system files or other unauthorized data on the host side by allocating and determining an accessible directory list for each worker thread and restricting the access scope of the worker threads. This ensures that the computing node can obtain the required model resources normally while enhancing the security of the host file system and the confidentiality of data.

[0062] As a further implementation of the above embodiments, this application embodiment specifically describes the process of performing permission verification using a target accessible directory list. (Refer to...) Figure 4 , Figure 4 This is the third flowchart illustrating the file access processing method provided by this invention. For example... Figure 4 As shown, the process of parsing the file access operation indicated by the request data packet and obtaining the target data through the target worker thread specifically includes the following steps: Step 301: Parse the request data packet to obtain the target file path corresponding to the file access operation.

[0063] The target worker thread first unpacks the received request data packet to extract the specific file access parameters requested by the compute node. These parameters include the target file path of the file to be accessed. For example, the compute node might request to read the file " / data / models / gpt4 / config.json", which is the target file path.

[0064] Step 302: When it is determined that the target file path is in the target accessible directory list, read the target data from the storage medium according to the file access operation.

[0065] After obtaining the target file path, the target worker thread does not immediately perform a read operation. Instead, it first compares the target file path with a pre-determined list of accessible directories. The comparison logic can determine whether the target file path is a subpath of a directory in the list, or whether it completely matches a specific file path in the list.

[0066] The target worker thread will only be allowed to continue execution if the determination result is yes, meaning the target file path is indeed within the authorized range of the target's accessible directory list. At this point, the target worker thread, according to the specific instructions for the file access operation (e.g., reading a data segment at a specified offset), reads the corresponding data content from the mass storage medium through the host-side file system interface and uses this data content as the target data.

[0067] This application embodiment adds a path verification step before actually reading data, ensuring that even if the computing node issues an access request for a sensitive area on the host (such as the system configuration directory), the access request will be blocked due to path verification failure, thereby fundamentally eliminating the risk of unauthorized access and ensuring the security of data in the host storage medium.

[0068] As a further implementation of the above embodiments, this application also provides a handling mechanism when permission verification fails.

[0069] Specifically, after the target worker thread compares and determines the target file path, if it determines that the target file path is not in the target accessible directory list, it means that the file access request initiated by the compute node is an unauthorized or illegal access. At this time, the target worker thread will refuse to perform the read operation on the storage medium and will perform the step of returning a null value result to the compute node.

[0070] The target worker thread constructs a response data packet indicating operation failure or empty data (i.e., a null result) and sends it back to the compute node. The null result notifies the compute node that the requested file path is unreachable or access is denied. Upon receiving this null result, the FUSE module on the compute node side returns an error code or empty data to the requesting compute program accordingly.

[0071] This application embodiment, by directly returning a null value when path verification fails, not only effectively intercepts illegal requests and protects the data security of the host, but also provides clear feedback to the computing node, avoiding deadlock or crashes caused by the computing program waiting indefinitely for data, thereby improving the robustness and controllability of the entire file access system.

[0072] Secondly, embodiments of this application also provide a file access processing method applied to a computing node. The execution subject of this method is a computing node, which can be a computing device or computing card equipped with an NPU. The computing node is connected to the host via a physical communication link to obtain file data accessed by the host.

[0073] Reference Figure 5 , Figure 5 This is the fourth flowchart illustrating the file access processing method provided by this invention. Figure 5 As shown, the file access processing method provided in this application includes the following steps: As a possible sequential implementation, a communication link with the host needs to be established before the compute node initiates any actual file access. Specifically, during the power-on or service initialization phase, the compute node initiates a communication connection request to the host and establishes multiple initial data channels for communication with the host.

[0074] In the above implementation, the compute node sends a connection command to the host by loading the user-space file system module (FUSE module). Subsequently, the compute node establishes N parallel data transmission channels between the local machine and the host, where N is a positive integer. These initial data channels lay the foundation for subsequent file data flow, enabling the compute node to send requests concurrently.

[0075] Based on this, the compute node executes step 401: obtain the file access request initiated by the compute program, and encapsulate the file access request into a request data packet.

[0076] When a deep learning computation program running on a compute node attempts to read a file, it issues a standard file access request. The compute node uses a pre-deployed FUSE module to capture this file access request and converts it into a request data packet with a specific format. This request data packet contains key information such as the file path, read offset, and read length, and is formatted into a data frame suitable for transmission over the physical link.

[0077] Step 402: Determine the target data channel from multiple initial data channels, and send the request data packet to the corresponding target worker thread on the host side through the target data channel.

[0078] Before sending a request data packet, the compute node needs to select one of the multiple initial data channels established above as the target data channel. The selection strategy can be based on a random algorithm, a round-robin algorithm, or a load balancing algorithm. After determining the channel, the compute node sends the request data packet through that target data channel.

[0079] The target worker thread is determined from the initial worker threads by the host, and there is a one-to-one correspondence between the initial data channel and the initial worker thread. Based on this one-to-one mapping relationship, when a computing node selects a target data channel to transmit a request data packet, the request data packet is routed to a specific target worker thread that has been pre-associated with the target data channel on the host for processing, thereby achieving precise matching between the data transmission path and the data processing thread on the link.

[0080] Step 403: Receive the response data packet carrying the target data returned by the host.

[0081] After a request is sent, the compute node waits for and receives a response data packet from the host. As one possible implementation, the target data carried in the response data packet is obtained by the host through the target worker thread associated with the target data channel, after processing the request data packet. Specifically, the corresponding thread on the host parses the request, reads data from the storage medium, and packages it back.

[0082] Step 404: Return the target data to the calculation program.

[0083] After receiving the response data packet, the compute node parses it to extract the valid target data, i.e., the file content. Then, the compute node returns this data to the requesting compute program through the FUSE framework. During this process, the compute program initiates calls and receives returned data through the standard file system interface, without needing to know whether the data is actually stored locally or on a remote host. This achieves a transparent remote file access mechanism for the compute program, ensuring the continuity of its execution logic.

[0084] The file access processing method provided in this application intercepts and encapsulates file requests on the computing node side, and concurrently sends request data packets to the host using multiple data channels. This method eliminates the need for the computing node to store complete large files locally; instead, it retrieves data from the host on demand, effectively alleviating the storage pressure on the computing node. Simultaneously, the multi-channel concurrent sending mechanism fully utilizes physical link bandwidth, significantly improving data loading speed and meeting the demand for high-speed access to massive amounts of data.

[0085] As a further implementation of the above embodiments, this application embodiment specifically illustrates the process by which a computing node intelligently selects a target data channel from multiple initial data channels. (Refer to...) Figure 6 , Figure 6 This is the fifth flowchart illustrating the file access processing method provided by this invention. Figure 6 As shown, the process of determining the target data channel specifically includes the following steps: Step 501: Obtain the load occupancy parameters for each initial data channel.

[0086] Before preparing to send a request data packet, the compute node evaluates the status of all currently available initial data channels.

[0087] Specifically, the compute nodes monitor or query the load status of each initial data channel in real time to obtain the corresponding load occupancy parameters. Load occupancy parameters can reflect the current busy level of the channel. For example, load occupancy parameters can be the number of data packets currently being transmitted by the channel, the percentage of bandwidth used by the channel, or the queue length of pending requests for the channel.

[0088] Step 502: Determine the target data channel based on the load occupancy parameters.

[0089] After obtaining the load occupancy parameters of all channels, the compute nodes make decisions based on the load occupancy parameters.

[0090] As one possible implementation, the compute node performs the following operation: compares the load occupancy parameters of each initial data channel, and selects the initial data channel with the lowest load occupancy parameter as the target data channel.

[0091] For example, if the system has channels A, B, and C, and their current queue lengths are 5, 2, and 8 respectively, the compute node will determine that the load occupancy parameter (value 2) of channel B is the lowest, and then allocate the current request data packet to channel B for transmission.

[0092] This application's embodiments achieve load balancing at the data transmission layer by introducing a dynamic channel selection mechanism based on load occupancy parameters. This method effectively avoids transmission congestion caused by overload of a single data channel, while also preventing resource waste on other idle channels. By automatically routing requests to the most readily available path, it maximizes the utilization of the aggregate bandwidth of all data channels, thereby ensuring low latency and high throughput in data interaction between the computing node and the host. This is particularly suitable for frequent and intensive data reading scenarios during large model training and inference.

[0093] In addition to the aforementioned implementation method of channel selection based on load occupancy parameters, this application also provides another implementation method for determining the target data channel from multiple initial data channels. This method schedules data based on the priority of request types, and specifically includes the following steps: First, parse the file access request to determine the request type.

[0094] The compute nodes first parse the intercepted file access requests to identify the specific operational intent of the request, thereby determining the request type. Request types include at least metadata operation types and data read / write types. Metadata operation types involve querying or modifying file attributes (such as file size, creation time, permissions, etc.), such as getattr and lookup operations; data read / write types involve reading or writing the actual content data of the file, such as read and write operations.

[0095] Next, based on the preset priority mapping relationship, the priority level corresponding to the request type is determined.

[0096] The compute node internally stores a pre-defined priority mapping table, which defines the correspondence between different request types and priority levels. Typically, metadata operations, although involving small amounts of data, are sensitive to latency and are therefore often assigned a high priority level; while large block read / write operations, although having high throughput, have a relatively high tolerance for microsecond-level latency and can be assigned a normal priority level. The compute node determines the priority level to which the current request type belongs based on this mapping relationship.

[0097] Finally, the target data channel is determined from multiple initial data channels based on priority levels.

[0098] The compute nodes select channels using differentiated strategies based on the determined priority levels. Specifically, when the request type is of high priority, the compute node performs the following operation: selects the initial data channel from multiple initial data channels that has the shortest current pending request queue length, or the initial data channel with the lowest round-trip communication latency, as the target data channel.

[0099] For example, for a high-priority metadata query request, the compute node will scan all channels and find that the queue length of channel 1 is 0 (idle), while the queue length of channel 2 is 10, and then immediately select channel 1. Alternatively, the compute node maintains real-time latency statistics for each channel and selects the channel with the smallest current latency value (e.g., RTT).

[0100] Through this implementation method, the embodiments of this application realize service quality-aware channel scheduling, ensuring rapid response to critical control commands and metadata operations, avoiding small but important requests being blocked by large data transmissions, thereby improving the overall response speed of the file system and user experience.

[0101] As a further implementation of the above embodiments, this application describes in detail the specific implementation path for a computing node to obtain a file access request. (Refer to...) Figure 7 , Figure 7 This is the sixth flowchart illustrating the file access processing method provided by this invention. Figure 7 As shown, the process of obtaining a file access request initiated by a computing program specifically involves the interaction between the computing node's operating system kernel and the user-space file system (FUSE) module, including the following steps: Step 601: Send a file access request to the FUSE module through the calculation program.

[0102] During the operation of a compute node, the deep learning computation program, as a user-space application, initiates file access requests through standard file system call interfaces (such as POSIX APIs) when it needs to access storage resources. Since the FUSE module has been pre-registered and mounted with a custom file system path in the compute node's operating system kernel, when the computation program accesses a file under that specific path, the operating system kernel automatically recognizes the file access request and redirects it to the FUSE framework, instead of directly manipulating the local physical disk. It should be noted that the FUSE framework can register the FUSE module with its own operating system kernel for subsequent file access request forwarding.

[0103] Step 602: Receive file access requests via the FUSE module.

[0104] Specifically, the FUSE module is a daemon or service program running in the user space of the compute node. The FUSE module monitors requests from the kernel through the interface provided by the FUSE kernel driver. When the kernel forwards a file access request, the FUSE module receives the request. At this point, the FUSE module obtains all the context information of the request, including the request type, file handle, read position, and length, thus completing the capture of the compute program's file access behavior and preparing for subsequent encapsulation and forwarding to the host.

[0105] This application's embodiments utilize FUSE technology to implement a file access proxy that is transparent to the upper-layer computing program. The computing program can access data on the remote host as if it were a local file without modifying its code. This non-intrusive design effectively reduces the complexity of system integration and improves the versatility and ease of use of the solution.

[0106] Reference Figure 8 , Figure 8 This is one of the structural schematic diagrams of the file access processing system provided by the present invention. The system includes: The data receiving module is used to receive request data packets from the target data channel through the target worker thread; wherein, the request data packets are sent by the computing node through the target data channel, and the target data channel is determined by the computing node from the initial data channel; The request processing module is used to parse the file access operation indicated by the request data packet through the target worker thread and obtain the target data; The response feedback module is used to return a response data packet carrying the target data to the computing node.

[0107] In one possible implementation, the system further includes a connection establishment module; the connection establishment module is used to determine multiple initial data channels connected to the computing node in response to a communication connection request initiated by the computing node, and to create an initial worker thread associated with each initial data channel.

[0108] Reference Figure 9 , Figure 9 This is the second schematic diagram of the file access processing system provided by the present invention. The system includes: The request encapsulation module is used to obtain file access requests initiated by the computing program and encapsulate the file access requests into request data packets. The request sending module is used to determine the target data channel from multiple initial data channels and send the request data packet to the corresponding target worker thread on the host side through the target data channel; wherein, the target worker thread is determined by the host side from the initial worker threads, and the initial data channel corresponds one-to-one with the initial worker thread; The response receiving module is used to receive response data packets carrying target data returned by the host. The data return module is used to return the target data to the calculation program.

[0109] It should be noted that the file access processing system provided by the present invention can execute the file access processing method of any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0110] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 10 As shown, the electronic device may include a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other via the communication bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute the file access processing methods provided in the above embodiments.

[0111] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to execute the file access processing methods provided in the above embodiments.

[0113] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the file access processing methods provided in the above embodiments.

[0114] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0115] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A file access processing method, characterized in that, Applied to the host side, including: The target worker thread receives request data packets from the target data channel; wherein, the request data packets are sent by the computing node through the target data channel, and the target data channel is determined by the computing node from the initial data channel; The target worker thread parses the file access operation indicated by the request data packet to obtain the target data; Return a response data packet carrying the target data to the computing node.

2. The file access processing method according to claim 1, characterized in that, The initial data channel corresponds one-to-one with the initial working thread, and the target working thread is the working thread corresponding to the target data channel.

3. The file access processing method according to claim 2, characterized in that, Also includes: Assign a unique initial list of accessible directories to each of the initial worker threads; Determine the list of target accessible directories corresponding to the target worker thread.

4. The file access processing method according to claim 3, characterized in that, The step of parsing the file access operation indicated by the request data packet through the target worker thread to obtain the target data includes: Parse the request data packet to obtain the target file path corresponding to the file access operation; When it is determined that the target file path is in the target accessible directory list, the target data is read from the storage medium according to the file access operation.

5. The file access processing method according to claim 4, characterized in that, When the target file path is not in the target accessible directory list, the method further includes: Return a null value to the computing node.

6. A file access processing method, characterized in that, Applied to compute nodes, including: Obtain the file access request initiated by the computing program, and encapsulate the file access request into a request data packet; A target data channel is determined from multiple initial data channels, and the request data packet is sent to the corresponding target worker thread on the host through the target data channel; wherein, the target worker thread is determined by the host from the initial worker threads, and the initial data channel corresponds one-to-one with the initial worker thread; Receive the response data packet carrying the target data returned by the host; The target data is returned to the calculation program.

7. The file access processing method according to claim 6, characterized in that, The step of determining the target data channel from the plurality of initial data channels includes: Obtain the load occupancy parameters for each of the initial data channels; The target data channel is determined based on the load occupancy parameters.

8. The file access processing method according to claim 6, characterized in that, The acquisition of file access requests initiated by the computing program includes: The calculation program initiates a file access request to the FUSE module; The file access request is received through the FUSE module.

9. A file access processing system, characterized in that, include: A data receiving module is used to receive request data packets from a target data channel through a target worker thread; wherein the request data packets are sent by the computing node through the target data channel, and the target data channel is determined by the computing node from an initial data channel; The request processing module is used to parse the file access operation indicated by the request data packet through the target worker thread and obtain the target data; The response feedback module is used to return a response data packet carrying the target data to the computing node.

10. A file access processing system, characterized in that, include: The request encapsulation module is used to obtain file access requests initiated by the computing program and encapsulate the file access requests into request data packets. A request sending module is used to determine a target data channel from multiple initial data channels, and send the request data packet to the target worker thread corresponding to the host through the target data channel; wherein, the target worker thread is determined by the host from the initial worker threads, and the initial data channel corresponds one-to-one with the initial worker thread; A response receiving module is used to receive a response data packet carrying the target data returned by the host. The data return module is used to return the target data to the calculation program.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the file access processing method as described in any one of claims 1 to 8.

Images