A lustre file system data balancing method based on multi-dimension customized strategy

Abstract

The present application belongs to the technical field of file system data balancing, and particularly relates to a Lustre file system data balancing method based on multi-dimensional customized strategy, aiming to solve the problems of single balancing granularity, low efficiency and rigid strategy in the prior art. The method comprises the following steps: defining a multi-dimensional balancing strategy containing conditional dimensions, data value weight dimensions and priority assignment, and storing the multi-dimensional balancing strategy in a rule library; after triggering balancing by monitoring the system, obtaining file metadata and business associated data, matching the strategy, and calculating a business value evaluation value; combining the business value evaluation value and a preset weight proportion to calculate a file priority score; generating and dynamically optimizing a first-in first-out task queue according to the priority; and migrating files according to the queue order, and preferentially allocating system resources to high-priority tasks. The present application realizes precise balancing driven by business value, improves resource utilization efficiency and system adaptability, and ensures stable operation.

Description

A Data Balancing Method for Lustre File System Based on Multi-Dimensional Customization Strategy Technical Field

[0001] This invention belongs to the field of file system data balancing technology, and in particular relates to a Lustre file system data balancing method based on a multi-dimensional customization strategy. Background Technology

[0002] With the rapid development of high-performance computing (HPC), big data analytics, and other fields, the scale of data is exploding, placing higher demands on the storage performance, resource utilization, and business adaptability of distributed file systems. Lustre file system, a massively parallel distributed file system widely used in high-performance computing scenarios, employs striped storage to distribute data across multiple object storage servers (OSS) and storage targets (OSTs). By improving data processing efficiency through parallel I / O operations, it has become one of the core infrastructures supporting massive data storage and high-performance computing.

[0003] In the actual operation of the Lustre file system, data distribution balance directly affects the overall system performance. When new OST nodes are added, the frequency of business data read and write varies greatly, or data lifecycle management is inadequate, significant differences in data volume will occur between OSTs, creating a "barrel effect"—some OSTs experience a surge in I / O pressure and increased response latency due to excessive data density, while other OSTs remain idle, ultimately limiting the throughput and processing efficiency of the entire file system. Therefore, data balancing is a crucial element for ensuring the stable performance of the Lustre file system, with its core objective being to migrate data from high-load OSTs to low-load OSTs, achieving a reasonable allocation of resources.

[0004] In existing technologies, the data balancer built into the Lustre system is the most widely used balancing solution. Its core logic revolves around equalizing space utilization: by monitoring the space utilization of all OSTs, when the difference between the maximum and minimum utilization exceeds a preset threshold, a balancing operation is triggered. The amount of data to be migrated is calculated based solely on the space load of the OST, and files to be migrated are selected according to the physical storage order of the files or a random algorithm. Finally, the data is migrated from high-utilization OSTs to low-utilization OSTs to achieve a convergence of space utilization across all OSTs.

[0005] However, the aforementioned prior art has the following significant drawbacks:

[0006] The current technology relies solely on OST space utilization as the only balancing metric, completely ignoring the intrinsic business value and dynamic attributes of the data, such as the importance of the project to which the data belongs, access frequency, file size, and compliance requirements. This results in critical data, especially hot data or key project data, not being prioritized for balancing because the OST space utilization rate of its host location does not reach the trigger threshold. This fails to effectively address I / O performance bottlenecks and may even affect the normal operation of core business processes.

[0007] Inefficient balancing and severe resource waste: Due to a lack of data value differentiation, existing technologies indiscriminately migrate all data that meets spatial balancing conditions, including a large amount of inactive cold data. Migrating such data not only fails to significantly improve overall system I / O performance but also consumes valuable network bandwidth, CPU resources, and storage I / O bandwidth, resulting in a high-cost, low-efficiency pseudo-balancing that reduces the effective utilization of system resources.

[0008] Rigid balancing strategies and poor business adaptability: Existing balancing solutions have fixed decision-making logic, making it impossible for administrators to customize configurations based on actual business needs. For example, the balancing process cannot be guided by business priorities, data characteristics, or time factors, resulting in the system's inability to adapt to diverse business scenarios and insufficient intelligence and management flexibility.

[0009] Therefore, how to break through the limitations of existing technologies that emphasize space balancing while neglecting business value, and provide a Lustre file system data balancing method that combines intelligence, customizability, and high efficiency to achieve differentiated priority balancing based on business logic and data characteristics, accurately optimize system performance, and improve resource utilization efficiency has become an urgent technical problem to be solved. Summary of the Invention

[0010] The purpose of this invention is to provide a data balancing method for the Lustre file system based on a multi-dimensional customization strategy, in order to overcome the limitations of existing technologies that emphasize space balancing while neglecting business value, and to achieve differentiated priority balancing based on business logic and data characteristics, thereby accurately optimizing system performance and improving resource utilization efficiency.

[0011] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0012] A data balancing method for the Lustre file system based on a multi-dimensional customization strategy includes the following steps:

[0013] S1: Define multi-dimensional balancing strategies and store them in the rule base. Each strategy includes a condition dimension, a data value weight dimension, and a priority assignment.

[0014] S2: Monitor the Lustre file system in real time to determine whether the load balancing triggering conditions are met. If so, determine the source OST to be migrated, load the rule base, traverse the file list on the source OST to be migrated, obtain the metadata information and business-related data of each file, match the metadata information with the condition dimensions of the policy rules, and calculate the business value assessment value of the file.

[0015] S3: For files that successfully match the policy rule conditions, calculate the file's priority score by combining its business value assessment value and the weight percentage corresponding to the preset business value assessment value.

[0016] S4: Receive file-related information, including file path, priority score, file size, source OST information, and business value assessment value; sort all files to be migrated in descending order according to priority score; generate an initial first-in-first-out task queue; and dynamically optimize the initial first-in-first-out task queue.

[0017] S5: Migrate files sequentially starting from the head of the priority task queue. System resources are allocated to high-priority migration tasks first, while low-priority tasks are paused or executed during system idle periods.

[0018] Preferably, the specific process of step S2 is as follows:

[0019] S21: Monitor the running metrics of all OSTs in the Lustre file system in real time, including: space utilization, I / O pressure, CPU load correlation value of each OST, and determine whether the balanced triggering conditions are met based on the preset balanced triggering conditions for different scenarios. If so, proceed to step S22.

[0020] S22: Determine the source OST to be migrated for different trigger scenarios, and generate a list of source OSTs to be migrated, including the basic information of each source OST such as its identifier ID, current space utilization, I / O pressure data, and the size of the used storage space;

[0021] S23: For each OST in the list of source OSTs to be migrated, establish an independent file traversal process and obtain the file storage index of each OST; traverse all files on the corresponding OST in a directory-level traversal manner to generate a list of files to be traversed;

[0022] S24: For each file traversed, obtain metadata and business-related data through the Lustre file system's metadata query interface;

[0023] S25: Perform file data matching with policy rules;

[0024] S26: Organize the matching results, metadata information, business-related data, and preliminary calculation results of business value assessment for each file into structured data entries to form a file dataset to be prioritized for calculation.

[0025] Preferably, in step S25, the matching of file data with policy rules involves matching the collected file metadata information and business-related data with the loaded policy rule set one by one. The matching process is performed by classifying and verifying the conditions according to their dimensions. The specific process is as follows:

[0026] Directory / path dimension matching: If the policy rule contains a directory / path condition, check whether the full path of the current file contains the directory / path segment specified by the rule. If so, the match is successful.

[0027] File size dimension matching: If the policy rule contains a file size condition, check whether the actual data size of the current file is less than or equal to the file size threshold set by the rule. If so, the match is successful.

[0028] Time attribute dimension matching: If the policy rule contains a time attribute condition, check whether the time attribute of the file is greater than or equal to the time threshold set by the rule. If so, the match is successful.

[0029] Data value weight dimension matching: If the strategy rules include data value weight conditions, match the corresponding value weight coefficients based on the business-related data in the file to complete the preliminary calculation of the business value assessment value;

[0030] Matching result recording: If a file meets the matching conditions of any policy rule, it is marked as a successful match, and the matched rule ID and the corresponding condition dimension type are recorded; if it does not meet the matching conditions of any rule, it is marked as a failed match and temporarily stored in the low-priority candidate queue.

[0031] Preferably, the specific process for priority calculation in step S3 is as follows:

[0032] S31: Calculate the weighted sum P based on the priority assignment of the calculation rules. r The formula is as follows:

[0033] P r =W r ×ΣP i ;

[0034] Where, ΣP i W is the sum of the priorities of all rules that match the file. r This represents the percentage of total weight for rule priority.

[0035] S32: Adjust the business value assessment value V according to the preset business value weighting W. v Weighted calculation of the business value weighted value Pv The formula is as follows:

[0036] P v =W v ×V;

[0037] S33: Calculate the final priority score P using the following formula:

[0038] P=P r +P v .

[0039] Preferably, the specific process of generating the initial first-in-first-out task queue in step S4 is as follows:

[0040] S41: Set sorting rule priorities, including first priority to fifth priority, based on descending priority score as the core sorting criterion and combined with auxiliary rules to solve the sorting problem of files with the same score;

[0041] S42: Use a stable sorting algorithm to perform multi-dimensional sorting to avoid disorder of files with the same priority due to the characteristics of the sorting algorithm;

[0042] S43: Transform the sorted list of files into a first-in-first-out migration task queue, where each task node contains complete file-related information.

[0043] Preferably, the specific process of step S42 is as follows:

[0044] S421: Perform a preliminary sorting of all files to be migrated according to the first priority to form a core sorting result set;

[0045] S422: For file groups with the same priority score in the core sorting result set, sort them again according to the second priority.

[0046] S423: For file groups that still have the same order after the second sort, the sorting is gradually refined according to the third, fourth and fifth priority rules, and finally a unique ordered list of files is obtained.

[0047] Preferably, the specific process of dynamically optimizing the initial first-in-first-out task queue in step S4 is as follows:

[0048] S44: Queue adjustment based on file access status: Monitor the access status of files waiting to be migrated in the queue in real time, and dynamically adjust the queue order according to the following rules;

[0049] S45: Queue partitioning scheduling based on system load: Real-time acquisition of system resource load data, and dynamic partitioning and management of queues;

[0050] S46: Queue adjustment based on the migration target OST status: Real-time monitoring of target OST space utilization and I / O pressure to avoid target OST overload.

[0051] Preferably, the specific process of step S5 is as follows:

[0052] S51: Task scheduling initialization: Obtain system resource information; calculate the maximum resource quota available for data migration; establish a resource allocation strategy to allocate differentiated resource quotas to tasks of different priorities, with higher priority tasks receiving more system resources;

[0053] S52: Priority task scheduling and execution: Retrieves high-priority tasks sequentially from the head of the task queue and allocates better system resources to high-priority tasks; monitors system load to dynamically adjust resource allocation, increases the proportion of balanced resources during off-peak periods, and dynamically adjusts the maximum number of parallel tasks.

[0054] S53: Monitor the impact of the migration process on system performance in real time and dynamically adjust the migration rate; execute corresponding recovery strategies for different migration anomalies, including automatic retry, task transfer, and error isolation.

[0055] S54: After the migration is complete, perform integrity checks on the files to ensure data consistency, update the system status, and release system resources.

[0056] Preferably, step S4 also includes queue status monitoring and exception handling:

[0057] Queue status monitoring: Real-time monitoring of key metrics of the task queue, including: queue length, percentage of tasks in each priority partition, average task waiting time, and task status distribution.

[0058] Exception handling mechanism:

[0059] Queue data corruption: If persistent queue data corruption is detected, the most recent full backup data will be automatically loaded to restore the queue state. At the same time, the corrupted data will be marked in the log and the administrator will be notified to investigate the cause.

[0060] Task deadlock: If a task remains in execution for an extended period without any progress updates, it is considered a deadlocked task. It is removed from the queue and marked as a migration failure. After logging, subsequent tasks continue to be executed.

[0061] Queue Overflow: If the number of files to be migrated exceeds the queue storage limit, the queue storage space will be automatically expanded, and files will be filtered according to priority scores. High-priority files will be retained first, and low-priority files will be temporarily stored in a temporary cache and added gradually when there is free space in the queue.

[0062] The beneficial effects of this invention include:

[0063] 1. Achieve precise balance driven by business value, breaking through the limitations of traditional physical balancing: Innovatively introducing a data value weight dimension during the strategy definition phase, combining file metadata characteristics with business-related data for multi-dimensional matching and value assessment, so that balancing decisions are no longer limited to the single physical indicator of OST space utilization. By incorporating business value assessment values ​​into priority calculations, core business data, frequently accessed hot data, and high-compliance-level data receive higher priority and are migrated first, directly and quickly eliminating I / O hotspot bottlenecks. This upgrades from physical balancing with equal space to logical balancing with performance optimization and business adaptation, significantly improving the system's support capability for core businesses.

[0064] 2. Improve resource utilization efficiency and avoid resource waste caused by pseudo-equilibrium: Through a differentiated priority mechanism, system resources are tilted towards high-value data migration tasks, while low-value cold data is migrated only during system idle periods or when resources are sufficient. This achieves "on-demand allocation and key protection" of resource scheduling, avoiding the resource consumption caused by indiscriminate migration in existing technologies. Valuable system resources are concentrated on the migration of core data that can generate significant performance benefits, greatly improving the input-output ratio of equilibrium operations, reducing ineffective resource occupation, and improving the resource utilization and operating efficiency of the entire file system.

[0065] 3. Flexible strategy customization to adapt to diverse business scenarios: It provides multi-dimensional, dynamically configurable balancing strategy rules. Administrators can flexibly define condition dimensions, data value weight dimensions, and priority assignments through the management interface, supporting dynamic updates of rules based on business changes. Whether prioritizing the performance of specific project data, optimizing the efficiency of small file metadata processing, or adapting to the storage requirements of data with different compliance levels, precise adaptation can be achieved through strategy customization. This solves the problems of rigid technical strategies and poor business adaptability in existing systems, significantly enhancing the system's flexibility and manageability, and meeting the diverse needs of complex real-world production scenarios.

[0066] 4. Intelligent and controllable balancing process to ensure system stability and data security: Multiple intelligent optimization mechanisms ensure the stability and reliability of the balancing process: During the task queue stage, dynamic adjustments are made based on file access status, system load, and target OST status to avoid conflicts between balancing operations and business read / write operations, and to prevent target OST overload; During the migration execution stage, system load and migration rate are monitored in real time, and resource allocation and the number of parallel tasks are dynamically adjusted to avoid system overload; At the same time, a comprehensive exception handling mechanism and data integrity verification process are set up to ensure that data is not lost or damaged during the migration process, and to ensure the stable operation of the system and data security during the balancing process.

[0067] 5. Dynamic queue optimization and exception handling enhance the robustness of the load balancing process: After the task queue is generated, dynamic sorting of the queue is achieved by monitoring the file access status in real time, ensuring that temporary hot files are migrated first; partitioned scheduling is performed based on system load to balance load balancing efficiency and business operation stability; a comprehensive response mechanism is designed for abnormal situations such as queue data corruption, task deadlock, and queue overflow, which can quickly restore the queue state, handle abnormal tasks, avoid interruption of the load balancing process, significantly improve the robustness and fault tolerance of the entire load balancing method, and adapt to complex and ever-changing actual operating environments. Attached Figure Description

[0068] Figure 1 is a flowchart illustrating the data balancing method of the Lustre file system based on a multi-dimensional customization strategy according to the present invention. Detailed Implementation

[0069] The present invention will now be described in further detail with reference to Figure 1:

[0070] Example 1

[0071] Referring to Figure 1, a data balancing method for the Lustre file system based on a multi-dimensional customization strategy includes the following steps:

[0072] S1: Define a multi-dimensional balancing strategy and store it in the rule base. Each strategy includes a condition dimension, a data value weight dimension, and a priority assignment. The condition dimension includes at least directory / path, file size, and time attribute. A new data value weight dimension is added, and the data value weight dimension is preset with corresponding weight coefficients based on the business importance level of the project to which the data belongs, the data access frequency threshold, and the data compliance requirement level.

[0073] S2: Monitor the Lustre file system in real time and determine if the load balancing triggering conditions are met. If so, identify the OST to be migrated, load the rule base, traverse the file list on the OST to be migrated, and obtain the metadata information and business-related data for each file. The metadata information includes file path, file size, and time attributes. The business-related data includes the project identifier, historical access frequency, and compliance level identifier. Match the metadata information with the basic condition dimensions of the policy rules, and calculate the business value assessment value of the file based on the preset coefficients of the business-related data and data value weight dimensions.

[0074] S3: Priority Calculation: For files that successfully match the basic condition dimensions of the strategy rules, calculate the priority score of the file by combining its business value assessment value and the weight ratio corresponding to the preset business value assessment value.

[0075] S4: Task Queue Generation and Dynamic Optimization: Receive file-related information, including file path, priority score, file size, source OST information, and business value assessment value. Sort all files to be migrated in descending order according to the priority score, generate an initial first-in-first-out (FIFO) task queue, and dynamically optimize the initial FIFO task queue.

[0076] Simultaneously, the access status of files to be migrated is monitored in real time. If a file is frequently accessed while waiting in the queue, its priority score is automatically increased and the queue order is adjusted. If a file is marked as archived and frozen, its priority is reduced to the end of the queue.

[0077] S5: Intelligent Balanced Execution: Files are migrated sequentially starting from the head of the priority task queue. System resources (network bandwidth, CPU time slices) are preferentially allocated to high-priority migration tasks. Low-priority tasks may be paused or executed during system idle periods.

[0078] In this embodiment, the specific process of step S2 is as follows:

[0079] S21: Real-time monitoring of key performance indicators of all object storage targets (OSTs) in the Lustre file system, including: space utilization, I / O pressure (including I / O requests per second (IOPS), I / O throughput, I / O response latency), and CPU load correlation values ​​for each OST.

[0080] Triggering conditions include:

[0081] Space utilization trigger: Calculate the difference between the maximum and minimum space utilization of all OSTs. When the difference exceeds the preset threshold (which can be set to 5% and can be adjusted through policy configuration), the trigger condition is determined to be met.

[0082] I / O heat trigger: Monitor the I / O pressure metrics (IOPS, throughput, response latency) of each OST. When the I / O pressure of a certain OST exceeds the preset percentage (which can be set to 30%) of the average level of other OSTs for a preset duration (which can be set to 3 minutes), the trigger condition is determined to be met, regardless of whether its space utilization rate meets the standard.

[0083] Upon receiving a manual trigger command from the administrator via the management interface (Web interface / configuration file), the trigger condition is directly determined to be met;

[0084] If any of the above triggering conditions are met, the process will proceed to the next step; otherwise, the monitoring status will continue.

[0085] S22: Identify the source OST to be migrated:

[0086] For space utilization trigger scenarios: Sort all OSTs in descending order of space utilization, and select OSTs whose utilization exceeds the average utilization of all OSTs plus a preset offset (which can be set to average utilization + 2%) as source OSTs to be migrated; if there is only a single OST with a significantly high utilization (exceeding the minimum utilization threshold difference), then directly list it as the only source OST.

[0087] I / O heat trigger scenario: OSTs with excessive I / O pressure are directly listed as source OSTs to be migrated, and the migration of hot files on the OST is focused.

[0088] Manually triggered scenario: When triggered, the administrator can specify a specific OST as the source OST, or determine the source OST by default based on the space usage filtering rules.

[0089] Generate a list of source OSTs to be migrated, which includes basic information such as the identifier ID of each source OST, current space usage, I / O pressure data, and used storage space size, providing a basis for subsequent file traversal.

[0090] S23: Traversing the list of source OST files to be migrated:

[0091] Traversal initialization: For each OST in the list of source OSTs to be migrated, an independent file traversal process is established to avoid mutual interference between cross-OST traversals; at the same time, the file storage index of each OST is obtained, including the striped storage index of the Lustre file system and the file physical storage address mapping table, to ensure that the traversal is complete.

[0092] Traversal execution: Traverse all files on the corresponding OST in a directory-level traversal manner (recursively from the root directory to the subdirectories), generating a list of files to be traversed. The list contains basic identifiers of the files, such as file ID and file name.

[0093] Traversal optimization: During the traversal process, skip files that have been marked as non-migrateable, such as system core configuration files, files that are being exclusively locked, and archive frozen files, and directly exclude them from the scope of files to be migrated, reducing the processing of invalid data.

[0094] S24: Obtain metadata information and business-related data:

[0095] Metadata information collection:

[0096] For each file traversed, the following core metadata is retrieved through the Lustre file system's metadata query interface (which can also be the Lustre native metadata access interface) to ensure data accuracy:

[0097] Path information: The complete storage path of the file, including the root directory, intermediate directories, and filename;

[0098] Size information: The actual data size of the file, in bytes;

[0099] Time attributes: Last access time (atime), Last modification time (mtime), accurate to the second;

[0100] Extended attributes: File type (regular file, directory file, log file), striping count (number of OSTs in the data stripe distribution);

[0101] Storage association information: the source OST identifier ID where the file is currently located, and the striped storage block address corresponding to the file.

[0102] Business-related data collection:

[0103] By using the system-integrated business data association interface, file extended attribute fields, or related database queries, business dimension data for each file can be obtained, achieving dual-dimensional coverage of technical attributes and business attributes.

[0104] Project-related data: The identifier of the project to which the file belongs (project ID, project name), and the project's business importance level (core level, normal level, test level);

[0105] Access characteristic data: historical access frequency of the file (number of accesses in the last 30 days / 90 days), and the user / user group that initiated the most recent access;

[0106] Compliance attribute data: compliance requirement level of the document (confidential, sensitive, public), compliance storage period requirements;

[0107] Data value associated with data: preset value weight coefficients based on the project's business importance level, historical access frequency, and compliance requirement level;

[0108] S25: Perform file data and policy rule matching: Match the collected file metadata information and business-related data with the loaded policy rule set one by one. The matching process is performed in the manner of "checking in sequence according to condition dimensions". The specific process is as follows:

[0109] Directory / path dimension matching: If the policy rule contains directory / path conditions, check whether the full path of the current file contains the directory / path segment specified by the rule. If so, the match is successful.

[0110] File size dimension matching: If the policy rule includes a file size condition, check whether the actual data size of the current file is less than or equal to the file size threshold set by the rule. If so, the match is successful.

[0111] Time attribute dimension matching: If the policy rule contains a time attribute condition (atime / mtime), check whether the time attribute of the file is greater than or equal to the time threshold set by the rule (which can be set to atime≥30 days ago). If so, the match is successful.

[0112] Data value weight dimension matching: If the strategy rules include data value weight conditions, the corresponding value weight coefficients are matched based on the business-related data in the file (project importance level, access frequency, compliance level) to complete the preliminary calculation of the business value assessment value.

[0113] Matching result recording: If a file meets the matching conditions of any policy rule, it is marked as "match successful" and the matched rule ID and corresponding condition dimension type are recorded; if it does not meet the matching conditions of any rule, it is marked as "match failed" and temporarily stored in the low-priority candidate queue, and will be processed according to the default policy later.

[0114] S26: Organize the "matching results + metadata information + business-related data + preliminary calculation results of business value assessment" of each file into structured data entries to form a file dataset to be prioritized for calculation. The dataset is stored according to the source OST to be migrated.

[0115] Example 2

[0116] Based on Example 1, the specific process of priority calculation in step S3 is as follows:

[0117] S31: Calculate the weighted sum P based on the priority assignment of the calculation rules. r :

[0118] For all rules matched in the file, first sum them and assign their priorities, then assign them according to their total weight W based on their priority. r Weighted summation, the formula is as follows:

[0119] P r =W r ×ΣP i ;

[0120] Where, ΣP i The sum of the priority values ​​of all rules matching a file is assigned. For example, if there are two rules matching a file, P1=30 and P2=20, then ΣP i =50.

[0121] S32: Calculate the weighted value P of the business value v :

[0122] The business value assessment value V is weighted according to the preset business value weight W. v Weighted summation, the formula is as follows:

[0123] Pv =W v ×V;

[0124] For example: V=80 (high business value), W v =0.4, then P v =0.4×80=32.

[0125] S33: Calculate the final priority score P:

[0126] Weighted sum of rule priorities and P r Weighted by business value P v The scores are added together to obtain the final priority score, as shown in the following formula:

[0127] P=P r +P v ;

[0128] For example: a file matches two rules (P1=30, P2=20, ΣPi=50), W r =0.6, V=80, W v =0.4, then:

[0129] P r =0.6×50=30; P v =0.4×80=32; final P=30+32=62.

[0130] The specific process of generating the initial first-in-first-out task queue in step S4 is as follows:

[0131] S41: Definition of Multi-Dimensional Sorting Rules:

[0132] The sorting is based on descending priority scores, supplemented by auxiliary rules to solve the sorting problem of files with the same score, ensuring a unique and reasonable queue order. The priority of the sorting rules is as follows:

[0133] First priority: Sort by file priority score (P) in descending order, with higher scores appearing earlier in the list;

[0134] Second priority: If priority scores are the same, sort by business value assessment value (V) in descending order, with higher business value documents taking priority;

[0135] Third priority: If the business value assessment values ​​are the same, sort them in ascending order by file size, and migrate smaller files first (which can quickly release source OST resources and improve migration throughput).

[0136] Fourth priority: If files are the same size, sort them in descending order by last access time (atime), with the most recently accessed files taking precedence;

[0137] Fifth priority: If all the above conditions are the same, sort by file unique ID in ascending order to ensure the deterministic nature of the sorting result.

[0138] S42: A stable sorting algorithm (merge sort) is used to perform multi-dimensional sorting to avoid the disorder of files with the same priority due to the characteristics of the sorting algorithm;

[0139] S43: FIFO Task Queue Construction:

[0140] The sorted file list is transformed into a first-in-first-out (FIFO) migration task queue. Each task node in the queue contains complete file-related information, and the construction rules are as follows:

[0141] Queue head: The file ranked first after sorting (highest priority, best overall value) is the first node of the queue;

[0142] In the middle of the queue: files are inserted into the queue in order of sorting results. Each node records the index of the previous and next nodes, forming a doubly linked list structure, which is convenient for subsequent dynamic adjustment.

[0143] Tail of the queue: The file ranked last after sorting is the tail node of the queue, and low-priority files (such as priority score <50) are concentrated in the tail area.

[0144] The specific process of step S42 is as follows:

[0145] S421: Perform a preliminary sorting of all files to be migrated according to the first priority (in descending order of priority score) to form a core sorting result set;

[0146] S422: For file groups with the same priority score in the core sorting result set, sort them a second time according to the second priority (descending order of business value assessment value);

[0147] S423: For file groups that still have the same order after the second sort, the sorting is gradually refined according to the third, fourth and fifth priority rules, and finally a unique ordered list of files is obtained.

[0148] The specific process of dynamically optimizing the initial first-in-first-out task queue in step S4 is as follows:

[0149] S44: Queue adjustments triggered by file access status:

[0150] The access status of files waiting to be migrated in the queue is monitored in real time through the Lustre file system access monitoring interface, and the queue order is dynamically adjusted according to the following rules:

[0151] High-frequency access file upgrade: If a file in the queue is accessed by the business process more than a preset threshold (up to 5 times) during the waiting period (up to 1 minute), it is determined to be a temporary hot file. Its priority score is automatically increased by a preset amount (up to +10, with a maximum of 100), and it is re-inserted into the front of the queue according to the sorting rules.

[0152] Archive frozen file downgrade: If a file in the queue is marked as archive frozen (archive status set by the business system), its priority score is automatically reduced to the lowest level (can be set to 0), and it is moved to the end of the queue. The migration is only performed when the system is idle.

[0153] Locked file temporary storage: If a file to be migrated in the queue is exclusively locked by a business process, it is removed from the current queue position and temporarily stored in the "Locked File Cache Queue", and the locking status is continuously monitored; when the lock is released, it is reinserted into the corresponding position in the queue according to the original priority score.

[0154] S45: System load-triggered queue partitioning scheduling:

[0155] Real-time acquisition of system resource load data, including CPU load, network bandwidth utilization, and source / target OST I / O response latency, and dynamic partitioning and management of queues:

[0156] Queue partitioning: The task queue is divided into three partitions: a high-priority partition (P≥80), a medium-priority partition (50≤P<80), and a low-priority partition (P<50).

[0157] Load adaptation adjustment:

[0158] When the system load is at its lowest, i.e., CPU load <30% and network bandwidth utilization <40%, tasks in the three partitions are allowed to be executed in parallel according to the queue order, improving the overall migration efficiency.

[0159] When the system load is at its peak, i.e., CPU load ≥70% or network bandwidth utilization ≥80%: suspend the execution of low-priority tasks and retain only high-priority tasks; reduce the migration resource consumption of medium-priority tasks according to a preset ratio (which can be set to 50%) to avoid affecting core business.

[0160] When the system load returns to the normal range, i.e., 30%≤CPU load<70% and 40%≤network bandwidth utilization<80%, gradually resume the execution of medium and low priority tasks and start them in the order of the queue.

[0161] S46: Queue adjustment triggered by migration target OST state:

[0162] Real-time monitoring of the target OST's space utilization and I / O pressure helps prevent overloading of the target OST.

[0163] If the space utilization rate of a target OST is close to the threshold (which can be 90%) or the I / O pressure continues to exceed the standard (more than 30% higher than the average of other target OSTs), the allocation of new migration tasks to that target OST will be suspended. The file tasks that originally pointed to that target OST in the queue will be reassigned to other low-load target OSTs, and the order of tasks in the queue will be adjusted (keeping the original priority unchanged, only updating the target OST information).

[0164] When the target OST status returns to normal, i.e., space utilization is less than 80% and I / O pressure is less than 20% of the average, the task allocation of the target OST is restored.

[0165] Example 3

[0166] Based on Example 1 or Example 2, the specific process of step S5 is as follows:

[0167] S51: Task scheduling initialization:

[0168] Obtain system resource information, including network bandwidth utilization, CPU load, etc.

[0169] Calculate the maximum resource quota available for data migration to ensure that the data balancing operation does not affect normal business operations;

[0170] Establish a resource allocation strategy to allocate differentiated resource quotas to tasks with different priorities. High-priority tasks will receive more system resources (priority greater than or equal to 80 is high priority, priority less than 50 is low priority, and others are medium priority).

[0171] S52: Priority Task Scheduling and Execution: High-priority tasks are retrieved sequentially from the head of the task queue to ensure that critical data is migrated first. Better system resources are allocated to high-priority tasks, and progress is monitored in real time to ensure that high-priority tasks are migrated as soon as possible. The system load is monitored to dynamically adjust resource allocation. During off-peak business periods, the proportion of balanced resources is appropriately increased, and the maximum number of parallel tasks is dynamically adjusted to avoid excessive parallelism that could lead to system overload.

[0172] S53: Intelligent control of the migration process: Real-time monitoring of the impact of the migration process on system performance and dynamic adjustment of the migration rate; execution of corresponding recovery strategies for different migration anomalies, including automatic retry, task transfer, and error isolation.

[0173] S54: Post-migration processing and verification: After the migration is completed, perform integrity verification on the files to ensure data consistency; update the system status; and release system resources.

[0174] Step S4 also includes queue monitoring and exception handling:

[0175] Queue status monitoring:

[0176] Real-time monitoring of key metrics in the task queue includes: queue length (number of tasks to be executed), task percentage for each priority partition, average task waiting time, and task status distribution (waiting / executing / paused / failed). The monitoring data is then fed back to the management interface, allowing administrators to monitor progress in real time.

[0177] Exception handling mechanism:

[0178] Queue data corruption: If persistent queue data corruption is detected, the most recent full backup data will be automatically loaded to restore the queue state; at the same time, the corrupted data will be marked in the log and reported to the administrator to investigate the cause.

[0179] Task deadlock: If a task remains in the "in execution" state for an extended period without any progress updates, such as exceeding the preset timeout period of 1 hour, it is determined to be a deadlocked task. It is removed from the queue and marked as "migration failed". After logging, subsequent tasks continue to be executed.

[0180] Queue Overflow: If the number of files to be migrated exceeds the queue storage limit, the queue storage space will be automatically expanded, and files will be filtered according to priority scores. High-priority files (P≥60) will be retained first, while low-priority files (P<60) will be temporarily stored in a temporary cache and added gradually when there is free space in the queue.

Claims

1. A data balancing method for the Lustre file system based on a multi-dimensional customization strategy, characterized in that, Includes the following steps: S1: Define multi-dimensional load balancing strategies and store them in the rule base. Each strategy includes a condition dimension, a data value weight dimension, and a priority assignment. S2: Monitor the Lustre file system in real time to determine if the load balancing trigger conditions are met. If so, identify the source OST to be migrated, load the rule base, traverse the file list on the source OST to be migrated, and obtain the metadata information and business-related data for each file. Match the metadata information with the condition dimensions of the strategy rules to calculate the business value assessment value of the file. S3: For files that successfully match the policy rule condition dimensions, calculate the file's priority score based on its business value assessment value and the weight ratio corresponding to the preset business value assessment value; S4: Receive file-related information, including file path, priority score, file size, source OST information, and business value assessment value. Sort all files to be migrated in descending order according to the priority score, generate an initial first-in-first-out (FIFO) task queue, and dynamically optimize the initial FIFO task queue; S5: Migrate files sequentially starting from the head of the priority task queue. System resources are preferentially allocated to high-priority migration tasks, while low-priority tasks are paused or executed during system idle periods; In step S2, matching metadata information with the policy rule condition dimensions involves matching the collected file metadata information and business-related data with the loaded policy rule set one by one. The matching process is performed by classifying and verifying the condition dimensions sequentially. The specific process is as follows: Directory / path dimension matching: If the policy rule... The matching process includes several key aspects: First, if the policy rule includes a directory / path condition, it checks whether the complete path of the current file contains the directory / path segment specified by the rule; if so, the match is successful. Second, if the policy rule includes a file size condition, it checks whether the actual data size of the current file is less than or equal to the file size threshold set by the rule; if so, the match is successful. Third, if the policy rule includes a time attribute condition, it checks whether the corresponding time attribute of the file is greater than or equal to the time threshold set by the rule; if so, the match is successful. Fourth, if the policy rule includes a data value weight condition, it matches the corresponding value weight coefficient based on the business-related data of the file to complete the preliminary calculation of the business value assessment value. Finally, if the file meets the matching condition of any policy rule, it is marked as a successful match, and the matched rule ID and corresponding condition dimension type are recorded. If no matching condition is met, it is marked as a failed match and temporarily stored in a low-priority candidate queue.

2. The data balancing method for Lustre file system based on a multi-dimensional customization strategy according to claim 1, characterized in that, The specific process of step S2 is as follows: S21: Monitor the running indicators of all OSTs in the Lustre file system in real time, including: space utilization, I / O pressure, and CPU load correlation value of each OST. Determine whether the balanced triggering conditions are met based on the preset different scenario balanced triggering conditions. If so, proceed to step S22; S22: Determine the source OSTs to be migrated for different triggering scenarios and generate a list of source OSTs to be migrated, including the identifier ID, current space utilization, I / O pressure data, and basic information of the used storage space size of each source OST; S23: For each OST in the list of source OSTs to be migrated, establish an independent file traversal process and obtain the file storage index of each OST; traverse all files on the corresponding OST in the directory hierarchy traversal method to generate a list of files to be traversed; S24: For each traversed file, obtain metadata and business-related data through the metadata query interface of the Lustre file system; S25: Perform file data matching with policy rules; S26: Organize the matching results of each file + metadata information + business-related data + preliminary calculation results of business value assessment into structured data entries to form a file dataset to be prioritized.

3. The data balancing method for Lustre file system based on a multi-dimensional customization strategy according to claim 2, characterized in that, The specific process of priority calculation in step S3 is as follows: S31: Calculate the priority assignment and weighted sum P of the calculation rules. r The formula is as follows: P r =W r ×ΣP i ; where ΣP i W is the sum of the priorities of all rules that match the file. r S32: The business value assessment value V is assigned a weighted percentage based on the preset business value weight W. v Weighted calculation of the business value weighted value P v The formula is as follows: P v =W v ×V; S33: Calculate the final priority score P using the following formula: P = P r +P v .

4. The data balancing method for Lustre file system based on a multi-dimensional customization strategy according to claim 1, characterized in that, The specific process of generating the initial first-in-first-out (FIFO) task queue in step S4 is as follows: S41: Using the priority score in descending order as the core sorting criterion, and combining auxiliary rules to solve the sorting problem of files with the same score, set the sorting rule priority, including the first priority to the fifth priority; S42: Use a stable sorting algorithm to perform multi-dimensional sorting to avoid the disorder of files with the same priority due to the characteristics of the sorting algorithm; S43: Transform the sorted file list into a migration task queue with a first-in-first-out structure, where each task node contains complete file-related information.

5. A data balancing method for the Lustre file system based on a multi-dimensional customization strategy according to claim 4, characterized in that, The specific process of step S42 is as follows: S421: Perform a preliminary sorting of all files to be migrated according to the first priority to form a core sorting result set; S422: For file groups with the same priority score in the core sorting result set, sort them again according to the second priority; S423: For file groups that still have the same order after the second sorting, refine the sorting step by step according to the third, fourth and fifth priority rules to finally obtain a unique ordered list of files.

6. The data balancing method for Lustre file system based on a multi-dimensional customization strategy according to claim 4, characterized in that, The specific process of dynamically optimizing the initial first-in-first-out task queue in step S4 is as follows: S44: Queue adjustment triggered by file access status: Real-time monitoring of the access status of files waiting to be migrated in the queue, and dynamic adjustment of the queue order according to the following rules; S45: Queue partition scheduling triggered by system load: Real-time acquisition of system resource load data, and dynamic partition management of the queue; S46: Queue adjustment triggered by the status of the migration target OST: Real-time monitoring of the space utilization and I / O pressure of the target OST to avoid overloading the target OST.

7. The data balancing method for Lustre file system based on a multi-dimensional customization strategy according to claim 1, characterized in that, The specific process of step S5 is as follows: S51: Task scheduling initialization: Obtain system resource information; calculate the maximum resource quota available for data migration; establish a resource allocation strategy to allocate differentiated resource quotas to tasks of different priorities, with high-priority tasks receiving more system resources; S52: Priority task scheduling execution: Sequentially obtain high-priority tasks from the head of the task queue and allocate better system resources to them; monitor system load and dynamically adjust resource allocation, increase the proportion of balanced resources during off-peak periods, and dynamically adjust the maximum number of parallel tasks; S53: Monitor the impact of the migration process on system performance in real time and dynamically adjust the migration rate, and execute corresponding recovery strategies for different migration anomalies, including automatic retry, task transfer, and error isolation; S54: After the migration is complete, perform integrity checks on the files to ensure data consistency, update the system status, and release system resources.

8. A data balancing method for Lustre file system based on a multi-dimensional customization strategy according to claim 1, characterized in that, Step S4 also includes queue status monitoring and anomaly handling: Queue status monitoring: Real-time monitoring of key indicators of the task queue, including: queue length, task proportion of each priority partition, average task waiting time, and task status distribution; Anomaly handling mechanism: Queue data corruption: If persistent queue data corruption is detected, the most recent full backup data is automatically loaded to restore the queue status, and the corrupted data is logged and reported to the administrator for investigation; Task deadlock: If a task is in the execution state for a long time without progress updates, it is determined to be a deadlocked task, removed from the queue and marked as a migration failure, and the log is recorded before continuing to execute subsequent tasks; Queue overflow: If the number of files to be migrated exceeds the queue storage limit, the queue storage space is automatically expanded, and files are filtered according to priority scores, with high-priority files being retained first, and low-priority files being temporarily stored in a temporary cache and added gradually when there is free space in the queue.

Images