A method for improving fairness of lustre hsm tasks

Abstract

The application belongs to the technical field of computer storage, and particularly relates to a method for improving fairness of a Lustre HSM task, which aims to solve the problem that a long-term occupation of a Coordinator queue by a first-started HSM archiving task causes a late service of a later-started task. The method exposes the processed message count of each operation type by modifying the HSM Agent, and estimates the queue depth based on the difference between the total number of sent messages and the processed number of messages on the client side. When the depth exceeds a first threshold value, the sending is suspended, and the sending is resumed when the depth falls below a second threshold value, so as to realize adaptive flow control. All concurrent tasks independently execute the mechanism, and cooperatively maintain the controllable global queue length. The application does not need to modify the Lustre kernel or the Coordinator, and only needs to adapt the user-mode Agent and the client script, so that the multi-task concurrent fairness and the overall throughput of the system can be significantly improved, and the application has high compatibility and deployment convenience.

Description

Technical Field

[0001] This invention belongs to the field of computer storage technology, and in particular relates to a method for improving the fairness of Lustre HSM tasks. Background Technology

[0002] With the continuous growth in demand for high-performance computing and large-scale data storage, the Lustre file system (parallel distributed file system) has been widely used in scientific research, cloud computing, and enterprise-level storage scenarios due to its high throughput and low latency. To balance performance and cost, Lustre introduces a Hierarchical Storage Management (HSM) mechanism, which transparently migrates cold data from high-speed local storage to backend object storage, such as cloud storage, achieving capacity expansion and resource optimization. This mechanism relies on the Coordinator component in the HSM architecture, which is responsible for scheduling message flow for operations such as ARCHIVE and RESTORE, and is the core hub connecting the Lustre Metadata Server (MDS) and the external Copytool agent.

[0003] The issue of fairness in HSM task execution is becoming increasingly prominent. The existing Lustre HSM Coordinator employs a single-threaded, single-queue FIFO processing model, where all HSM_ARCHIVE requests from clients are processed in the order they are received. When multiple archiving tasks start concurrently, the earlier started task can quickly submit a large number of archived messages to the MDS, rapidly filling the Coordinator queue. This causes messages from later started tasks to remain at the end of the queue for extended periods, failing to receive timely service. Although some solutions attempt to control the rate of archived request transmission through static rate limiting (such as Robinhood's rate_limit configuration), such methods require users to pre-estimate processing capacity, making it difficult to adapt to real-world scenarios with dynamically changing file sizes. This can easily lead to message backlog or idle Copytool resources, failing to achieve dynamic and fair scheduling among tasks.

[0004] While existing technologies have explored prioritizing ARCHIVE and RESTORE tasks—for example, by isolating critical operations through user-space Coordinators or External Job Queues—they haven't resolved the internal fairness issue when multiple ARCHIVE tasks are executed concurrently. More critically, there's a lack of a lightweight, secure, and low-overhead mechanism that allows each HSM task to be aware of the number of pending messages generated in the Coordinator in real time and adaptively adjust its message sending rate accordingly. Since Lustre doesn't natively provide a client-accessible queue depth interface, directly querying the `hsm.actions` parameter on the MDS side carries permission restrictions and performance risks.

[0005] Therefore, this invention aims to provide a flow control strategy based on end-to-end feedback, which estimates the message backlog status locally on the client and coordinates the sending rhythm, thereby achieving fair and efficient sharing of HSM service resources by multiple tasks without modifying the Lustre kernel. Summary of the Invention

[0006] The purpose of this invention is to provide a method for improving the fairness of Lustre HSM tasks, thereby solving the technical problems existing in the background art.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for improving the fairness of Lustre HSM tasks includes the following steps: S1: Modify the HSM agent to expose its internal processing status. Set up an independent counter for each HSM operation type inside the HSM agent and provide a status query interface to the outside world through local sockets or named pipes, so that external programs can obtain the number of HSM_ARCHIVE messages that the HSM agent has successfully processed in real time. S2: Estimate the message queue depth of the coordinator on the client side. The client program executing the archiving task maintains a local counter to record the total number of archive (HSM_ARCHIVE) messages sent to the Lustre metadata server, and periodically obtains the number of messages processed by the corresponding HSM agent through the status query interface. The difference between the two is taken as the message queue depth to be processed in the HSM coordinator (Lustre HSM Coordinator) for the archiving task. S3: Implement feedback-based adaptive flow control. When the estimated queue depth reaches a preset first threshold, the client pauses sending new archive (HSM_ARCHIVE) messages to the Lustre metadata server and enters a waiting state. In the waiting state, the client continuously queries the processing progress of the HSM agent until the estimated queue depth drops to a preset second threshold, where the second threshold is less than the first threshold. At this point, message sending is resumed. S4: Coordinates the concurrent execution of multiple tasks. All concurrently executed HSM archiving tasks follow the aforementioned flow control rules. By independently controlling the amount of message accumulation generated in the Coordinator, they work together to ensure that the total length of the HSM message queue of the entire file system remains within a controllable range.

[0008] Preferably, the counter set inside the HSM agent also includes a counter for the total amount of data processed, and the status query interface adopts a lightweight request-response protocol. The query request includes the identifier of the target counter, and the response message returns the corresponding count value.

[0009] Preferably, the client program establishes a query connection with the specified HSM agent during the initialization phase, and after each call to the lfshsm_archive command to send an archive request, it synchronously updates the local message sending counter according to the number of files contained in the request.

[0010] Preferably, when estimating the queue depth, the client uses a sliding window mechanism to smooth the processing count returned by the HSM agent, so as to eliminate count fluctuations caused by network jitter or instantaneous processing delays. The sliding window mechanism uses a weighted moving average to calculate the smoothed queue depth, giving higher weight to recent samples.

[0011] Preferably, in step S3, while in the waiting state, the client dynamically calculates the sleep time based on the historical processing speed. Specifically, this includes recording the total time elapsed since the task started, calculating the average processing rate of the HSM agent in combination with the number of messages currently processed, estimating the time required to clear the messages based on the number of messages currently pending, and then putting the client into sleep mode by a predetermined proportional factor of the estimated time required to clear the messages.

[0012] Preferably, in step S4, each HSM archiving task is bound to a designated HSM agent through a unique archiveid, so that the flow control of each task is independent and does not interfere with each other.

[0013] Preferably, the status query interface is designed in read-only mode, allowing only query operations and not accepting any modification instructions.

[0014] Preferably, the HSM agent maintains the last update timestamp of the counter in the status query interface. The client judges the freshness of the counter data by comparing the timestamp, and determines that the HSM agent is abnormal when the timestamp has not been updated within a preset time limit.

[0015] Preferably, the flow control check adopts a periodic triggering strategy, and the flow control evaluation process is only initiated when the total number of messages sent by the client reaches an integer multiple of the first threshold and is not less than the first threshold.

[0016] The beneficial effects of this invention include: 1. Significantly improves the fairness of multi-task concurrency: By introducing an adaptive flow control mechanism based on end-to-end feedback, the problem of tasks starting first monopolizing the Coordinator queue is effectively solved. Archive tasks that start later do not need to wait for the massive amount of messages from tasks that started first to be processed before they can obtain service resources. This reduces the task startup delay from extremely long message processing time to a shorter message level, achieving a fundamental shift from time fairness to resource fairness.

[0017] 2. Efficient Utilization of Overall System Bandwidth: This invention enables multiple HSM archiving tasks to execute concurrently, fully aggregating the Lustre front-end network bandwidth and the back-end object storage egress bandwidth. Each Copytool agent is no longer idle for extended periods but continuously performs data migration, significantly improving the overall throughput and resource utilization of the HSM system, with particularly noticeable benefits in large-scale offline computing center scenarios.

[0018] 3. Adaptive and Configuration-Free Flow Control: Compared to existing solutions that rely on users manually configuring static rate limits, this invention dynamically adjusts the message sending rate by sensing the actual processing capacity of the HSM proxy in real time. This mechanism can automatically adapt to changes in file size distribution, network conditions, and storage backend performance, avoiding message backlog and server performance degradation caused by excessively high rates, and preventing idle Copytool resources caused by excessively low rates, thus achieving "worry-free" intelligent flow control.

[0019] 4. High compatibility and low invasiveness: The core improvement of this solution lies in the user-space HSM proxy and client archiving scripts, avoiding modifications to the Lustre kernel and core components such as the Coordinator. This allows the invention to be easily integrated into existing Lustre HSM deployments, is compatible with all standard Copytool interfaces, protects existing user investments, and lowers the threshold and risk of technology upgrades. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the method for improving the fairness of Lustre HSM tasks according to the present invention. Detailed Implementation

[0021] The following is in conjunction with the appendix Figure 1 The present invention will be further described in detail below: See appendix Figure 1As shown, the methods for improving the fairness of Lustre HSM tasks specifically include: modifying the HSM agent to expose its internal processing status: enhancing the functionality of the existing HSM agent, Copytool, by setting an independent processed message counter for each HSM operation type internally, and providing a status query interface to the outside world through local sockets or named pipes, so that external programs can obtain the number of HSM_ARCHIVE messages that the HSM agent has successfully processed in real time.

[0022] Specifically, the counters set up inside the HSM agent include not only counters for the number of processed messages differentiated by operation type, but also counters for the total amount of processed data. The status query interface adopts a lightweight request-response protocol. The query request contains the identifier of the target counter, and the response message returns the corresponding count value.

[0023] For the engineering implementation of this step, the core event loop module of the HSM proxy is first refactored, embedding a set of atomic counter structures into its main processing thread. The structure is defined as follows: struct hsm_agent_counters{ atomic_uint64_tarchive_msg_count; / / Number of HSM_ARCHIVE (archived) messages successfully processed atomic_uint64_trestore_msg_count; / / Number of HSM_RESTORE messages successfully processed atomic_uint64_tremove_msg_count; / / Number of HSM_REMOVE messages that have been successfully processed. atomic_uint64_ttotal_archive_bytes; / / Total data size (in bytes) processed by the archiving operation. atomic_uint64_ttotal_restore_bytes; / / Total data size (in bytes) processed during the restore operation. uint64_tlast_update_time; / / Counter last updated timestamp (nanoseconds) }

[0024] Whenever the HSM agent completes an HSM_ARCHIVE operation and successfully sends an hsm_action_end confirmation message to the Lustre HSM Coordinator, it immediately executes the atomic_fetch_add(&counters.archive_msg_count,1) operation, simultaneously incrementing the total number of bytes involved in this operation into the total_archive_bytes field. This incrementing operation occurs within the same transaction context as the message confirmation sending, ensuring the atomicity and consistency of the count.

[0025] The status query interface is implemented through a Unix domain socket (AF_UNIX). During the startup initialization phase, the HSM agent creates a listening socket with a fixed path name, such as / var / run / hsm_agent_<archive_id> .sock, where<archive_id> This is the archive identifier bound to the HSM agent. This socket only accepts local connections and rejects any remote network access to ensure system security. The query protocol uses a binary format; the request message consists of a 2-byte opcode and a 4-byte counter identifier, while the response message contains an 8-byte count value. Supported counter identifiers include: 0x01 (archive_msg_count), 0x02 (restore_msg_count), 0x03 (remove_msg_count), 0x04 (total_archive_bytes), and 0x05 (total_restore_bytes).

[0026] To ensure read-only security of the interface, the HSM proxy only performs memory read operations upon receiving a query request, without triggering any state changes or control commands. All counter values ​​are obtained via the `atomic_load` instruction to avoid lock contention in high-concurrency query scenarios. Furthermore, the HSM proxy maintains an internal heartbeat mechanism, updating the `last_update_time` field every 500 milliseconds. Clients can compare this timestamp to determine the freshness of the counter data. If the timestamp has not been updated for more than 2 seconds, it is determined that the HSM proxy may have exited abnormally, and the client should terminate the flow control logic and report an error.

[0027] In the above method for improving the fairness of Lustre HSM tasks, step S2 estimates the Coordinator message queue depth on the client side: the client program executing the archiving task maintains a local counter to record the total number of HSM_ARCHIVE messages sent to the Lustre metadata server, and periodically obtains the number of messages processed by the corresponding HSM agent through the status query interface. The difference between the two is taken as the depth of the pending message queue formed by the archiving task in the Lustre HSMCoordinator. Specifically, in step S2, the client program establishes a query connection with the specified HSM agent during the initialization phase, and after each call to the lfshsm_archive command to send an archiving request, it synchronously updates the local message sending counter according to the number of files contained in the request to ensure the accuracy of the estimation.

[0028] For the engineering implementation of this step, the client program parses the `--archive-id` option in the command-line arguments at startup to determine the unique identifier of the target HSM agent. Subsequently, the client attempts to connect to the path ` / var / run / hsm_agent_`.<archive_id> The .sock is a Unix domain socket. If the connection fails, it will retry 3 times, each time with a 100-millisecond interval; if it still fails, a runtime exception will be thrown and execution will terminate. After the connection is established, the client initializes its local state structure: struct client_state{ uint64_tsent_msg_count; / / Total number of HSM_ARCHIVE messages sent to MDS uint64_tlast_query_time; / / Timestamp of the last query to the HSM proxy uint64_tsmoothed_pending; / / Number of messages to be processed after smoothing by the sliding window uint64_twindow_buffer

[10] ; / / Sliding window buffer, storing the number of the last 10 pending messages. int window_index; / / Current position in the write buffer }; During the process of traversing the list of files to be archived, each time the client calls the `lfshsm_archive` command, regardless of whether the command processes a single file or a batch of files, its standard output is parsed to obtain the actual number of files committed. For example, when executing `lfshsm_archive -a`...<archive_id> When processing file1, file2, and file3, the client records that this operation involves three files and increments send_msg_count by 3. This counting operation is performed immediately after the lfs command returns a success status code, ensuring strict synchronization with the actual reception status of MDS.

[0029] To improve the robustness of the estimation, the client employs a sliding window mechanism to smooth the original number of messages to be processed. The formula for calculating the original number of messages to be processed is: raw_pending=sent_msg_count-archived_num; Here, `sent_msg_count` represents the total number of HSM_ARCHIVE messages sent to MDS, and `archived_num` represents the number of processed HSM_ARCHIVE messages retrieved from the HSM agent. The client stores `raw_pending` in a `window_buffer` loop array and calculates the arithmetic mean of all values ​​within the window as `smoothed_pending`. The sliding window size is fixed at 10, effectively filtering out count fluctuations caused by network jitter or instantaneous processing delays by the HSM agent. For example, when the HSM agent temporarily pauses processing large files, a single query might return the same `archived_num`, causing a brief spike in `raw_pending`; however, the sliding window averaging smooths out such spikes, preventing premature triggering of flow control mechanisms.

[0030] In the above method for improving the fairness of Lustre HSM tasks, feedback-based adaptive flow control is implemented: when the estimated queue depth reaches a preset first threshold, the client suspends sending new HSM_ARCHIVE messages to the Lustre metadata server and enters a waiting state; in the waiting state, the client continuously queries the processing progress of the HSM agent until the estimated queue depth drops to a preset second threshold, where the second threshold is less than the first threshold, at which point message sending is resumed.

[0031] Specifically, the value ranges of the first threshold and the second threshold are configured according to the network bandwidth, the performance of the storage backend, and the load capacity of the Lustre server in the actual deployment environment to achieve a balance between avoiding message accumulation and maintaining high throughput. In addition, in step S3, in the waiting state, the client does not perform busy waiting but dynamically calculates a reasonable sleep time according to the historical processing speed. Specifically, the client records the total elapsed time since the task started, calculates the average processing rate of the HSM proxy based on the number of messages processed so far, estimates the time required to empty the queue based on the number of messages to be processed currently, and finally sleeps for a predetermined scaling factor of the estimated time required to empty the queue to reduce unnecessary query overhead and quickly respond to changes in the processing progress.

[0032] For the engineering implementation of this step, the client reads two key thresholds from the configuration file or environment variables during the initialization phase: flow_control_high_watermark (the first threshold N) and flow_control_low_watermark (the second threshold M), where N > M ≥ 0. In a typical deployment, N can be set to 1000 and M can be set to 200, but users are allowed to adjust according to the actual environment. The flow control check is not performed after each message is sent, but a periodic trigger strategy is adopted: only when sent_msg_count reaches an integer multiple of N, that is, sent_msg_count % N == 0 and sent_msg_count >= N, does it enter the flow control evaluation process.

[0033] The specific process of the flow control evaluation process is as follows: A: The client sends a counter query request with the identifier 0x01 to the HSM proxy to obtain the current archived_num value; B: Calculate the smoothed number of messages to be processed smoothed_pending; C: If smoothed_pending < M, it is determined that the queue depth has dropped to a safe level, exit the flow control process, and continue to send new messages; D: If smoothed_pending ≥ N, enter the waiting loop; E: Inside the waiting loop, the client first calculates the total elapsed time since the task started, and the formula is as follows: time_diff = current_time - task_start_time; F: If time_diff > 0 and archived_num > 0, calculate the average processing rate of the HSM proxy ct_perf = archived_num / time_diff, unit: messages per second; G: Based on the current number of pending messages smoothed_pending and the processing rate ct_perf, estimate the time required to clear the queue: time_wait = smoothed_pending / ct_perf; H: To avoid excessive waiting due to uneven file size distribution, the actual client sleep time is time_wait*0.7, and the scaling factor of 0.7 can be adjusted based on actual testing; I: After the hibernation ends, repeat steps A to C until smoothed_pending. <M。

[0034] This dynamic sleep mechanism significantly reduces query frequency. For example, when the HSM agent processes 100 messages / second and there are 500 messages to process, the estimated clearing time is 5 seconds, and the actual sleep time is 3.5 seconds. During this period, the client does not consume CPU resources for busy waiting and can respond promptly to changes in processing progress. The scaling factor of 0.7 takes into account the non-linear characteristics of processing rate. Large file processing may cause a momentary drop in rate, and excessively long sleep may cause the queue to be cleared prematurely, resulting in idle resources. The coefficient of 0.7 ensures that the client wakes up before the queue is nearly empty, maintaining the continuous workload of the HSM agent.

[0035] In the above methods for improving the fairness of Lustre HSM tasks, the concurrent execution of multiple tasks is coordinated: all concurrently executed HSM archiving tasks follow the aforementioned flow control rules, and by independently controlling the amount of message backlog they generate in the Coordinator, they work together to ensure that the total length of the HSM message queue of the entire file system is kept within a controllable range, thereby providing timely service opportunities for tasks that start later.

[0036] Specifically, since the number of archiveids supported by a single Lustre file system is limited, each archiving task is bound to a specific HSM agent through a unique archiveid. This ensures that the flow control of each task is independent and does not interfere with each other, working together to achieve a stable global queue depth. Furthermore, this method requires all clients participating in concurrent archiving to adopt the flow control mechanism described in this invention. If unmodified traditional archiving tools are running simultaneously, their unrestricted message sending behavior will still fill the Coordinator queue, causing the fairness mechanism of this invention to fail.

[0037] To implement this step, system administrators need to pre-plan the archiveid resource pool before deploying multi-task concurrent archiving. Taking Alibaba Cloud CPFS as an example, a single file system supports a maximum of 10 archiveids (numbered 1 to 10), therefore a maximum of 10 independent archiving tasks can run concurrently. Each task must explicitly specify a unique `--archive-id` parameter when starting, which is also used for: Determine the target archive backend for the lfshsm_archive command; Locate the corresponding HSM proxy status query socket path; Isolate the flow control counter space for each task.

[0038] Inter-task coordination is achieved entirely through autonomous flow control, without the need for a central coordinator. Assume there are K concurrent tasks, where K ≤ the maximum number of archiveids. Each task i independently maintains its `sent_msg_count_i` and `smoothed_pending_i`. Since the global queue depth of the Lustre HSM Coordinator is equal to the sum of the queue depths corresponding to all archiveids, i.e.: global_queue_depth=∑ i=1 Ｋ (sent_msg_count i -archived_num i ); Where global_queue_depth is the global queue depth, and sent_msg_count is the global queue depth. i For the first i Total number of messages sent by each task; archived_num i For the first i Number of messages processed by each task; When each task i When its smoothed_pending_i is controlled within the range of [M, N], the global queue depth is constrained to the range of [K×M, K×N]. For example, if K=5, N=1000, and M=200, the global depth will always be between 1000 and 5000, which is much lower than the depth caused by sending 100,000 messages in a single task in the traditional mode.

[0039] To prevent traditional tools from compromising fairness, the system can implement an admission control policy: before launching the client of this invention, the current queue depth is checked via `lctlget_param-nmdt.$FSNAME-MDT0000.hsm.actions` (executed on the MDS node). If the depth exceeds a safety threshold (e.g., 5000), the launch of a new task is refused, prompting the user to wait for existing tasks to complete or terminate the traditional tool. Furthermore, the HSM agent can be extended to support a "flow control mode" flag, which proactively reduces the message processing priority of non-flow control client connections; however, this function is an enhancement and not a core feature of this invention.

[0040] Specific application examples: Typical application scenario: A high-performance computing center on-premises deploys the Lustre file system fs1, supporting 8 archiveids. The user needs to archive 5 large scientific datasets (dataset_A to dataset_E) simultaneously, each containing approximately 2 million files, for a total data volume of 50TB. In the traditional model, if the 5 archiving tasks are started sequentially, the later-started task will have to wait several hours before it can begin processing.

[0041] The operation flow of the method of the present invention is as follows: Environmental preparation: Start 5 modified HSM proxy instances and bind them to archiveid1 to 5 respectively; Each HSM agent creates a corresponding status query socket from / var / run / hsm_agent_1.sock to / var / run / hsm_agent_5.sock; Configure flow control parameters: N=1000, M=200, scaling factor=0.7.

[0042] Task initiated: Simultaneously launch 5 client processes, each executing: . / smart_archiver--archive-id=1--input=dataset_A . / smart_archiver--archive-id=2--input=dataset_B ... (and so on) Runtime behavior: Each client independently iterates through its list of dataset files; A flow control check is triggered after every 1000 file archive requests are sent. Assuming the HSM agent's average processing rate is 80 messages / second, when the number of messages to be processed reaches 1000, the client will sleep for approximately (1000 / 80)*0.7≈8.75 seconds. During the hibernation period, other clients continue to send messages to ensure that all HSM agents remain active; The global queue depth is stable between 5×200 and 5×1000, that is, between 1000 and 5000.

[0043] Performance comparison: Traditional mode: Task E needs to wait for Task AD to finish processing 8 million messages. Assuming the processing rate is 80 messages / second, the waiting time is approximately 8,000,000 / 80 / 3,600≈27.8 hours. In this invention's mode: the maximum number of messages that task E can wait for is 4 × 1000 = 4000 (with each of the other tasks accumulating 1000), and the waiting time is approximately 4000 / 80 / 3600 ≈ 0.014 hours (about 50 seconds). Backend object storage bandwidth utilization increased from 20% in the traditional model to over 95% due to the continuous operation of all HSM proxies.

[0044] Exception handling: If HSM agent_3 crashes unexpectedly, its corresponding client will mark the task as failed after the query times out and release archiveid3; The system log records the error details, but does not affect the normal flow control of the other four tasks; After the administrator restarts HSM agent_3, the remaining files of dataset_C can be resubmitted.

[0045] As can be seen from the above examples, the present invention achieves a balance between task fairness, efficient resource utilization, and system robustness in real-world large-scale scenarios.

[0046] Example 2 Based on Example 1, an alternative technical solution is provided, in which the status query interface is implemented using named pipes (FIFO) instead of Unix domain sockets. This solution is suitable for older Linux kernel environments that do not support AF_UNIX sockets.

[0047] Specifically, the HSM agent calls mkfifo(" / var / run / hsm_agent_) at startup.<archive_id> The `.fifo",0600` command creates a named pipe and opens it in write-only mode for responses. Clients open the same pipe in read-only mode for queries. To avoid blocking, the HSM broker uses non-blocking I / O: when there are no readers on the pipe, the write operation immediately returns an EAGAIN error, and the HSM broker discards the response; when there are readers, the HSM broker serializes the counter value into a fixed-length binary block (16 bytes: 8 bytes of counter value + 8 bytes of timestamp) and writes it to the pipe.

[0048] The client query process has been adjusted as follows: Open a named pipe (O_RDONLY|O_NONBLOCK); Query requests are sent through a separate control file (such as / var / run / hsm_agent_).<archive_id> (.ctl), write the counter identifier; Read the response from the named pipe; if it returns EAGAIN, retry. After a successful read, close the pipe to complete the single query.

[0049] Although this solution sacrifices some performance (pipeline I / O overhead is higher than socket I / O), it keeps the core flow control logic unchanged and is suitable for resource-constrained embedded HSM deployment scenarios.

[0050] Example 3 Further, an enhanced sliding window mechanism is considered to address scenarios with extreme file size distributions. Based on the 10-element window of Example 1, a weighted moving average (WMA) is introduced to replace the simple arithmetic average.

[0051] Specifically, the client maintains a weight array weights

[10] ={1,2,3,...,10}, assigning higher weights to the most recently processed messages. The formula for calculating the queue depth after smoothing is: ; This mechanism makes flow control decisions more sensitive to recent trends. For example, when the HSM agent's processing rate suddenly increases, new samples with high weights can lower the smoothed_pending value more quickly, prompting clients to resume sending earlier and reducing resource idleness. Real-world testing shows that in archiving scenarios with a mix of small (1KB) and large (10GB) files, the WMA scheme improves throughput by an average of 12% compared to the simple approach.

[0052] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for improving the fairness of Lustre HSM tasks, characterized in that, Includes the following steps: S1: Modify the HSM agent to expose the internal processing status. Set up an independent processed message counter for each HSM operation type inside the HSM agent and provide a status query interface to the outside world through local sockets or named pipes. S2: Estimate the coordinator message queue depth on the client side. The client program executing the archiving task maintains a local counter to record the total number of archived messages sent to the Lustre metadata server. S3: Periodically obtain the number of messages processed by the corresponding HSM agent through the status query interface, and use the difference between the total number of archived messages and the number of messages processed by the corresponding HSM agent as the depth of the pending message queue formed by the archived task in the HSM coordinator. S4: Implement feedback-based adaptive flow control. When the estimated queue depth reaches the preset first threshold, the client pauses sending new archived messages to the Lustre metadata server and enters a waiting state. In the waiting state, it continuously queries the processing progress of the HSM agent until the estimated queue depth drops to the preset second threshold, where the second threshold is less than the first threshold. At this time, message sending is resumed. S5: Coordinates the concurrent execution of multiple tasks. All concurrently executed HSM archiving tasks follow the aforementioned flow control rules. By independently controlling the amount of message accumulation generated in the coordinator, they work together to ensure that the total length of the HSM message queue of the entire file system remains within a controllable range.

2. The method for improving the fairness of Lustre HSM tasks according to claim 1, characterized in that, The counters set up inside the HSM agent also include a counter for the total amount of data processed. The status query interface adopts a lightweight request-response protocol. The query request contains the identifier of the target counter, and the response message returns the corresponding count value.

3. The method for improving the fairness of Lustre HSM tasks according to claim 1, characterized in that, During the initialization phase, the client program establishes a query connection with the specified HSM agent and updates the local message sending counter synchronously based on the number of files contained in the request after each call to the lfshsm_archive command to send an archive request.

4. A method for improving the fairness of Lustre HSM tasks according to claim 3, characterized in that, When estimating queue depth, the client uses a sliding window mechanism to smooth the processing count returned by the HSM agent, in order to eliminate count fluctuations caused by network jitter or instantaneous processing delays. The sliding window mechanism uses a weighted moving average to calculate the smoothed queue depth, giving higher weight to recent samples.

5. A method for improving the fairness of Lustre HSM tasks according to claim 1, characterized in that, In step S4, while in the waiting state, the client dynamically calculates the sleep time based on the historical processing speed. Specifically, this includes recording the total time since the task started, calculating the average processing rate of the HSM agent in combination with the number of messages currently processed, estimating the time required to clear the messages based on the number of messages currently pending, and then putting the client into sleep mode by a predetermined proportional factor of the estimated time required to clear the messages.

6. A method for improving the fairness of Lustre HSM tasks according to claim 1, characterized in that, In step S5, each HSM archiving task is bound to a designated HSM agent through a unique archiveid, so that the flow control of each task is independent and does not interfere with each other.

7. A method for improving the fairness of Lustre HSM tasks according to claim 1, characterized in that, The status query interface is designed to be read-only, allowing only query operations and not accepting any modification instructions.

8. A method for improving the fairness of Lustre HSM tasks according to claim 1, characterized in that, The HSM agent maintains the last update timestamp of the counter in the status query interface. The client judges the freshness of the counter data by comparing the timestamp, and determines that the HSM agent is abnormal when the timestamp has not been updated within a preset time limit.

9. A method for improving the fairness of Lustre HSM tasks according to claim 1, characterized in that, The flow control check adopts a periodic triggering strategy. The flow control evaluation process is initiated only when the total number of messages sent by the client reaches an integer multiple of the first threshold and is not less than the first threshold.

Images