lustre storage system deployment method and system

Abstract

This application discloses a Lustre storage system deployment method and system, relating to the field of data storage technology. The disclosed Lustre storage system deployment method and system, through technologies such as automated hardware detection agents, parameter generation rule engines, conflict rule base verification, and transactional deployment engines, achieve standardization and intelligence throughout the entire Lustre storage system deployment process. This effectively avoids data omissions and misjudgments caused by manual operation, ensures accurate matching between configuration parameters and hardware capabilities, and can prevent configuration conflicts in real time. It has the advantages of efficient deployment process, accurate configuration, preventable conflicts, and recoverable state, thus improving the deployment efficiency and operational stability of the Lustre storage system.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a method and system for deploying a Lustre storage system. Background Technology

[0002] In the current technological environment, Lustre, as a high-performance distributed file system, has become a core infrastructure for large-scale data storage and processing scenarios. However, its deployment process has long relied on manual operation, requiring technicians to manually complete hardware information collection, parameter configuration, and system integration. Specifically, during the server hardware topology detection phase, details such as the number of CPU cores, NUMA node distribution, network card type, and disk configuration must be recorded for each server. This is prone to data omissions or misjudgments when dealing with heterogeneous hardware environments. The parameter generation phase lacks standardized rules, requiring technicians to estimate the number of service threads, the number of target object storage devices, and network binding strategies based on experience, leading to mismatches between configurations and actual hardware capabilities. Furthermore, the parsing and verification of deployment templates rely entirely on manual checks, making it difficult to identify conflicts between network bandwidth and the number of target storage devices in a timely manner, often causing cluster performance bottlenecks or service interruptions. In addition, due to the lack of unified deployment standards, Lustre systems implemented by different teams exhibit significant differences in configuration logic, resulting in prominent cross-platform compatibility issues and severely limited system interoperability. These issues not only significantly extend the deployment cycle and increase the manpower costs of repeated debugging, but also lead to decreased system stability due to configuration errors, and even trigger data consistency risks, hindering the large-scale application and operational efficiency improvement of Lustre storage systems in complex production environments.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a Lustre storage system deployment method and system, which aims to improve the deployment efficiency and operational stability of the Lustre storage system.

[0005] To achieve the above objectives, this application proposes a Lustre storage system deployment method, the method comprising:

[0006] Hardware topology feature data of the target server is collected through a hardware detection agent;

[0007] The hardware topology feature data is input into the parameter generation rule engine to generate the Lustre deployment parameter set.

[0008] Obtain the deployment template file filled in by the user, and parse the deployment template file to obtain structured configuration data;

[0009] The structured configuration data and the Lustre deployment parameter set are input into the conflict rule base for verification to generate optimized configuration data;

[0010] A cluster deployment instruction sequence is constructed based on the optimized configuration data and the Lustre deployment parameter set;

[0011] The cluster deployment instruction sequence is executed through a transactional deployment engine, and a deployment report is output; wherein, before executing each instruction, a system state snapshot is created and the association between the snapshot and the instruction is recorded.

[0012] In one embodiment, the step of collecting hardware topology feature data of the target server through a hardware detection agent includes:

[0013] Execute processor detection instructions to obtain information on the number of CPU cores and the distribution of NUMA nodes;

[0014] Execute network device detection commands to obtain network interface card type and queue depth data;

[0015] Execute storage device detection commands to obtain disk type and quantity data;

[0016] The hardware topology feature data is generated by integrating the CPU core count and NUMA node distribution information, the network card type and queue depth data, and the disk type and quantity data.

[0017] In one embodiment, the step of executing a network device detection command to obtain network interface card (NIC) type and queue depth data includes:

[0018] Query the network card model identifier through the network card driver interface;

[0019] Retrieve the network card capability lookup table based on the network card model identifier;

[0020] When RDMA protocol support is detected, activate the network card's RoCE mode and obtain queue depth data.

[0021] In one embodiment, the step of inputting the hardware topology feature data into a parameter generation rule engine to generate a Lustre deployment parameter set includes:

[0022] The number of service threads is calculated based on the number of CPU cores in the hardware topology feature data.

[0023] Calculate the target number of object storage based on the number of disks in the hardware topology feature data;

[0024] A network bonding strategy is generated based on the network interface card (NIC) type in the hardware topology feature data.

[0025] The Lustre deployment parameter set is generated by combining the number of service threads, the number of object storage targets, and the network binding strategy.

[0026] In one embodiment, the step of inputting the structured configuration data and the Lustre deployment parameter set into a conflict rule base for verification to generate optimized configuration data includes:

[0027] Load a conflict rule base containing thresholds for matching network bandwidth and the target number of object storage items;

[0028] Extract the target number of objects stored in the Lustre deployment parameter set;

[0029] Compare the network bandwidth data in the structured configuration data with the target number of objects to be stored;

[0030] When the target number of objects to be stored exceeds the matching threshold, the target number of objects to be stored is adjusted to output the optimized configuration data.

[0031] In one embodiment, the step of constructing a cluster deployment instruction sequence based on the optimized configuration data and the Lustre deployment parameter set includes:

[0032] Merge the network address parameters in the optimized configuration data with the network binding strategy in the Lustre deployment parameter set;

[0033] Generate a software package distribution instruction sequence based on the merge results;

[0034] Generate a cluster configuration instruction sequence based on the target number of object storage in the Lustre deployment parameter set;

[0035] The cluster deployment instruction sequence is generated by combining the package distribution instruction sequence and the cluster configuration instruction sequence in the execution order.

[0036] In one embodiment, the steps of creating a system state snapshot and recording the association between the snapshot and instructions include:

[0037] Identify the current system storage volume status;

[0038] Create logical volume snapshots using storage volume management tools;

[0039] Record the association between the logical volume snapshot and the current instruction.

[0040] In one embodiment, the step of creating a logical volume snapshot using a storage volume management tool includes:

[0041] Send a snapshot creation command containing the storage volume identifier to the storage volume management tool;

[0042] Execute the snapshot creation instruction to generate a logical volume snapshot.

[0043] In one embodiment, the method further includes:

[0044] When an instruction execution failure is detected, the nearest logical volume snapshot is located based on the aforementioned association.

[0045] Restore the system to the state recorded in the logical volume snapshot;

[0046] Clear intermediate data files generated when command execution fails.

[0047] Furthermore, to achieve the above objectives, this application also proposes a Lustre storage system deployment system, which includes: a memory, a processor, and a Lustre storage system deployment program stored on the memory and executable on the processor, wherein the Lustre storage system deployment program is configured to implement the steps of the Lustre storage system deployment method.

[0048] The Lustre storage system deployment method and system proposed in this application achieve standardization and intelligence throughout the entire Lustre storage system deployment process through technologies such as automated hardware detection agents, parameter generation rule engines, conflict rule base verification, and transactional deployment engines. This effectively avoids data omissions and misjudgments caused by manual operation, ensures accurate matching between configuration parameters and hardware capabilities, and can prevent configuration conflicts in real time. It has the advantages of efficient deployment process, accurate configuration, preventable conflicts, and recoverable state, thus improving the deployment efficiency and operational stability of the Lustre storage system. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating an embodiment of the Lustre storage system deployment method of this application.

[0052] Figure 2 This is a schematic diagram of a configuration of the Lustre storage system deployed according to an embodiment of this application.

[0053] Explanation of icon numbers:

[0054] 10. Memory; 20. Processor.

[0055] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0056] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of this application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0057] It should be understood that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and are not intended to indicate or imply relative importance.

[0058] In existing technologies, Lustre deployment has long relied on manual operation, requiring technicians to manually complete hardware information collection, parameter configuration, and system integration. Specifically, during the server hardware topology detection phase, details such as the number of CPU cores, NUMA node distribution, network card type, and disk configuration must be recorded for each server. This process is highly susceptible to data omissions or misjudgments in heterogeneous hardware environments. The parameter generation phase lacks standardized rules, forcing technicians to estimate the number of service threads, target object storage quantities, and network binding strategies based on experience, leading to mismatches between configurations and actual hardware capabilities. Furthermore, the parsing and verification of deployment templates rely entirely on manual checks, making it difficult to promptly identify conflicts between network bandwidth and target storage quantities, often resulting in cluster performance bottlenecks or service interruptions. In addition, the lack of unified deployment standards leads to significant differences in configuration logic across Lustre systems implemented by different teams, resulting in prominent cross-platform compatibility issues and severely limited system interoperability.

[0059] Based on this, this application provides a method for deploying a Lustre storage system, referring to... Figure 1 The Lustre storage system deployment method includes steps S100 to S600, wherein:

[0060] Step S100: Collect hardware topology feature data of the target server through a hardware detection agent;

[0061] Step S200: Input the hardware topology feature data into the parameter generation rule engine to generate the Lustre deployment parameter set;

[0062] Step S300: Obtain the deployment template file filled in by the user, and parse the deployment template file to obtain structured configuration data;

[0063] Step S400: Input the structured configuration data and the Lustre deployment parameter set into the conflict rule base for verification, and generate optimized configuration data;

[0064] Step S500: Construct a cluster deployment instruction sequence based on the optimized configuration data and the Lustre deployment parameter set;

[0065] Step S600: Execute the cluster deployment instruction sequence through the transactional deployment engine and output a deployment report; wherein, before executing each instruction, a system status snapshot is created and the association between the snapshot and the instruction is recorded.

[0066] In this embodiment, the hardware detection agent can be a tool or program module used to automatically acquire target server hardware information. This can be achieved by running specific hardware detection scripts, such as by calling hardware interface functions provided by the operating system to obtain hardware information, or by extracting hardware features by reading register data from hardware devices. Hardware topology feature data refers to a data set describing the hardware composition and connection relationships of the target server, which can be organized and stored in formats such as JSON and XML. The parameter generation rule engine is a logical processing unit that dynamically generates configuration parameters based on input data. It can be implemented through preset rule tables, such as calculating the number of service threads and the target number of object storage based on the number of CPU cores and disks in the hardware feature data, according to a fixed ratio. The deployment template file refers to a user-defined configuration file containing deployment requirements. It can be generated using a text editor or a graphical interface tool, and the process of parsing this file can be implemented using regular expression matching or syntax analysis techniques.

[0067] In this embodiment, the conflict rule base is a logical component used to detect and adjust potential conflicts between configuration parameters. It can be implemented by loading predefined rule files, such as rule files containing thresholds for matching network bandwidth with the target number of object storage items. Optimized configuration data refers to the final set of configuration parameters generated after conflict detection and adjustment, which can be stored in key-value pairs. The transactional deployment engine is a deployment execution tool that supports atomic operations. It can achieve rollback functionality by recording state snapshots before and after executing each instruction, for example, restoring to the most recent state snapshot in case of execution failure.

[0068] In this embodiment, this application fundamentally solves the problems of low efficiency, frequent errors, and poor compatibility in the deployment of traditional Lustre storage systems by replacing manual configuration with an automated process. Specifically, a hardware detection agent automatically collects hardware topology feature data, avoiding the tediousness and errors of manually entering hardware information; a parameter generation rule engine generates a deployment parameter set that accurately matches hardware capabilities, eliminating configuration deviations caused by manual estimation; a conflict rule base verifies and adjusts configuration parameters, ensuring configuration compatibility and stability; and a transactional deployment engine combined with a snapshot mechanism enables atomic operations and traceable rollback of deployment steps, significantly reducing the risk of deployment failure and simplifying the fault recovery process.

[0069] In this embodiment, the Lustre storage system deployment method achieves automated deployment through the coordinated operation of multiple steps, effectively solving the problems of low efficiency, error-proneness, and poor compatibility associated with manual configuration. Specifically, firstly, hardware topology feature data of the target server is collected through a hardware detection agent. This process avoids errors that may arise from manual input of hardware information, ensuring that subsequent configuration is based on the actual hardware environment. Furthermore, the collected hardware topology feature data is input into a parameter generation rule engine to generate a Lustre deployment parameter set. This parameter set is dynamically derived based on hardware features, accurately matching the actual capabilities of the server and eliminating configuration deviations caused by manual estimation.

[0070] In this embodiment, after obtaining the deployment template file filled in by the user, the file is parsed to obtain structured configuration data. This transforms user requirements into standardized input, preserving the flexibility of user customization while preventing conflicts caused by non-standard configurations through structured parsing. Subsequently, the structured configuration data is validated against the Lustre deployment parameter set input conflict rule base to generate optimized configuration data. This process automatically detects and adjusts parameter mismatch issues based on preset rules, such as the threshold relationship between network bandwidth and the target number of object storage, thereby ensuring the compatibility and stability of the configuration.

[0071] In this embodiment, a cluster deployment instruction sequence is constructed based on optimized configuration data and the Lustre deployment parameter set. The optimized parameters and strategies are then integrated to generate an ordered instruction stream, ensuring the continuity and consistency of the deployment process. Finally, the cluster deployment instruction sequence is executed through a transactional deployment engine, and a deployment report is output. Before executing each instruction, a system state snapshot is created, and the association between the snapshot and the instruction is recorded, achieving atomic operations and traceable rollback of the deployment steps. This significantly reduces the risk of deployment failure and simplifies the fault recovery process. Therefore, the overall technical solution of this application automates the entire process from hardware awareness to instruction execution, effectively avoiding the shortcomings of manual intervention. Simultaneously, rule verification and snapshot mechanisms enhance the robustness of the system, fundamentally improving the reliability and maintainability of the Lustre storage system deployment.

[0072] In one feasible implementation, refer to Figure 2 The steps for collecting hardware topology feature data of a target server through a hardware detection proxy include: executing processor detection instructions to obtain the number of CPU cores and NUMA node distribution information; executing network device detection instructions to obtain network card type and queue depth data; executing storage device detection instructions to obtain disk type and quantity data; and integrating the CPU core count and NUMA node distribution information, the network card type and queue depth data, and the disk type and quantity data to generate the hardware topology feature data.

[0073] In this embodiment, the processor detection instruction refers to a low-level command used to extract computing resource-related information from the processor hardware of the target server. It can be implemented by calling hardware interfaces provided by the operating system or by directly accessing processor registers. In practical applications, this instruction ensures the acquisition of accurate CPU core count and NUMA node distribution information, thus providing a reliable basis for subsequent calculations of the number of service threads. The purpose of introducing this instruction is to avoid manual estimation or configuration omissions, and to improve the accuracy of hardware topology characteristic data.

[0074] In this embodiment, the network device detection command is an automated query tool. It interacts with the network interface card (NIC) driver to obtain the NIC model identifier and further searches a NIC capability lookup table to identify whether the NIC supports specific protocols (such as RDMA) and its performance parameters such as queue depth. This command can be implemented based on standard network management protocols (such as SNMP) or a dedicated driver interface, aiming to ensure that network binding policies are generated based on actual hardware capabilities rather than relying on general configurations. The storage device detection command is a technical means specifically designed to obtain the specifications and scale of storage resources. It can be implemented by calling storage management tools or directly reading storage controller information. This command can specifically obtain disk type and quantity data, providing an accurate basis for calculating the target number of object storage devices and preventing resource allocation imbalances due to missing disk information.

[0075] In this embodiment, the above steps ensure the comprehensiveness and accuracy of hardware topology feature data through a standardized hardware detection instruction execution process. First, processor detection instructions are executed to obtain CPU core count and NUMA node distribution information. This process directly extracts computing resource topology details from the hardware level, avoiding errors that may arise from manual intervention. Second, network device detection instructions are executed to obtain network interface card (NIC) type and queue depth data. By automatically querying NIC models and queue parameters, network device performance characteristics are identified, enabling network binding policy generation to be optimized based on actual hardware capabilities. Third, storage device detection instructions are executed to obtain disk type and quantity data, providing a foundation for the rational allocation of target storage resources. Finally, by integrating the above multi-dimensional hardware parameters, a complete server hardware view is constructed, significantly reducing the risk of configuration conflicts in the subsequent parameter generation rule engine.

[0076] Building upon this, the aforementioned scheme works closely with the hardware topology feature data input parameter generation rule engine to ensure that the generated Lustre deployment parameter set accurately reflects the actual hardware capabilities of the target server. This systematic hardware detection method not only solves the deployment parameter generation problem caused by data acquisition defects but also lays a solid foundation for the subsequent generation of optimized configuration data, thereby effectively improving the efficiency and reliability of Lustre storage system deployment.

[0077] In one feasible implementation, the steps of executing network device detection commands to obtain network interface card (NIC) type and queue depth data include: querying the NIC model identifier through the NIC driver interface; retrieving the NIC capability lookup table based on the NIC model identifier; and activating the NIC's RoCE mode and obtaining queue depth data when RDMA protocol support is detected.

[0078] In this embodiment, the network interface card (NIC) model identifier refers to the encoded information used to uniquely identify the NIC hardware model, which can be read through the standardized interface provided by the NIC driver. In practical applications, this identifier is usually stored in the NIC firmware in string or numeric form, directly reflecting the basic characteristics of the hardware. The purpose of introducing the NIC model identifier is to provide an accurate basic data source for subsequent protocol support status determination, thereby avoiding errors that may be caused by manual input. The NIC capability lookup table can be a predefined database used to store the mapping relationship between different NIC models and their corresponding functional characteristics. For example, this lookup table can be implemented in the form of key-value pairs, where the key is the NIC model identifier and the value is the corresponding protocol support status and other hardware capability descriptions. In this way, discrete hardware characteristics can be transformed into structured information, significantly improving query efficiency and accuracy.

[0079] In this embodiment, the RDMA protocol support status detection is performed based on the matching results between the network interface card (NIC) model identifier and the NIC capability lookup table. When NIC support for the RDMA protocol is detected, the system automatically activates the NIC's RoCE mode and collects queue depth data in this mode. This process ensures that queue depth data is acquired in an optimized state, thus providing a reliable basis for subsequent network performance tuning of the Lustre system.

[0080] In this embodiment, the solution achieves automated data acquisition and optimization through a series of ordered steps. First, the model identifier is obtained directly from the hardware layer using the network card driver interface, ensuring the authenticity and reliability of the data source. Then, by searching the network card capability lookup table, the functional characteristics of the network card are quickly located, especially its support status for the RDMA protocol. This design effectively solves the inefficiency and error-prone problems caused by relying on manual data searching in traditional methods. Finally, after confirming RDMA protocol support, RoCE mode is dynamically enabled and queue depth data is collected. This mechanism not only improves the accuracy of data acquisition but also provides crucial support for network layer performance optimization of the Lustre system.

[0081] Building upon this foundation, the aforementioned technical solution is closely integrated with the overall process of hardware topology feature data acquisition. By accurately identifying network interface card (NIC) protocol characteristics, the integrity of the hardware topology feature data is ensured, thus laying the groundwork for generating a reliable Lustre deployment parameter set. This design significantly improves the stability of cluster operation and network throughput efficiency, resolving the deployment parameter deviation problem caused by protocol incompatibility.

[0082] In one feasible implementation, the step of inputting the hardware topology feature data into a parameter generation rule engine to generate a Lustre deployment parameter set includes: calculating the number of service threads based on the number of CPU cores in the hardware topology feature data; calculating the target number of object storage based on the number of disks in the hardware topology feature data; generating a network binding policy based on the network interface card type in the hardware topology feature data; and combining the number of service threads, the target number of object storage, and the network binding policy to generate the Lustre deployment parameter set.

[0083] In this embodiment, hardware topology feature data refers to the data set collected from the target server that reflects its hardware configuration and performance characteristics. This data can be obtained using various detection methods, such as processor detection commands, network device detection commands, and storage device detection commands. The number of service threads refers to the number of worker threads used to handle concurrent requests in the Lustre system, aiming to improve the system's parallel processing capability through dynamic matching with the number of CPU cores. The number of object storage targets refers to the number of target units used to store data in the Lustre file system, aiming to optimize storage I / O performance through a reasonable correspondence with the number of disks. The network bonding strategy refers to a network configuration scheme tailored to the characteristics of different network interface card (NIC) types, aiming to improve communication efficiency and reliability by adapting to the NIC hardware characteristics.

[0084] In this embodiment, the number of CPU cores refers to the number of available physical or logical processor cores in the target server. It serves as a core indicator of hardware parallel processing capability, and the number of service threads is determined through a calculation formula or a preset mapping table. The number of disks refers to the number of available physical storage devices in the target server. This can be counted by statistically analyzing disk identifiers or querying the storage management interface, and the target number of object storage devices is set accordingly. The network interface card (NIC) type refers to the model of the network interface card in the target server and the protocol characteristics it supports. This can be identified by querying the driver interface or retrieving a capability lookup table, and a corresponding network bonding strategy is generated based on characteristics such as whether it supports the RDMA protocol.

[0085] In this embodiment, deployment parameters are dynamically generated by deeply associating specific dimensions of hardware topology feature data, effectively solving the problem of parameter configuration being disconnected from the hardware environment. First, the number of service threads is calculated based on the number of CPU cores in the hardware topology feature data. Using the number of CPU cores as a core indicator of hardware parallel processing capability ensures that the number of service threads is strictly aligned with actual computing resources, avoiding scheduling overhead or resource idleness caused by thread overload, thereby optimizing service processing efficiency. Second, the target number of object storage is calculated based on the number of disks in the hardware topology feature data. Based on the storage I / O potential reflected by the number of disks, the target number of object storage is matched to the scale of physical storage devices, improving the parallelism and balance of data distribution and reducing storage access bottlenecks. Third, a network binding strategy is generated based on the network interface card (NIC) type in the hardware topology feature data. Network configurations are customized for NIC type differences, ensuring that the binding strategy is highly compatible with network hardware characteristics, enhancing communication reliability and bandwidth utilization. Finally, a Lustre deployment parameter set is generated by combining the number of service threads, the target number of object storage, and the network binding strategy. This integrates key parameters at the computing, storage, and network levels, ensuring the inherent consistency of the configurations of each system component and avoiding deployment failures or performance degradation caused by parameter conflicts. This automates and improves the efficiency of Lustre storage system deployment, significantly enhancing the rationality of system resource allocation, the stability of I / O performance, and the efficiency of network communication, thus providing reliable deployment assurance for large-scale data storage and processing scenarios.

[0086] In one feasible implementation, the step of inputting the structured configuration data and the Lustre deployment parameter set into a conflict rule base for verification to generate optimized configuration data includes: loading a conflict rule base containing a matching threshold for network bandwidth and the number of target object storage; extracting the number of target object storage from the Lustre deployment parameter set; comparing the network bandwidth data in the structured configuration data with the number of target object storage; and adjusting the number of target object storage when the number of target object storage exceeds the matching threshold to output the optimized configuration data.

[0087] In this embodiment, the conflict rule base refers to a data set used to store preset rules. It can be implemented using relational databases, NoSQL databases, or memory caching technologies, aiming to provide flexible and efficient rule management capabilities. The matching threshold is a quantitative standard set based on the performance dependency between network bandwidth and the target number of object storage locations. It can be determined through experimental testing, historical data analysis, or theoretical modeling, aiming to ensure stable system operation within hardware capability limits. Furthermore, adjusting the target number of object storage locations is a process of automatically correcting unreasonable configurations based on preset rules, aiming to avoid resource waste or performance degradation caused by parameter conflicts.

[0088] In this embodiment, the scheme defines a quantified matching standard by loading a conflict rule base containing matching thresholds for network bandwidth and the target number of object storage. This enables the verification process to accurately identify hardware capability boundaries, thus avoiding the ambiguity of a general rule base. Next, by extracting the target number of object storage from the Lustre deployment parameter set, it ensures that the verification basis originates from automatically generated hardware topology feature data, rather than subjective settings, thereby enhancing parameter reliability. Based on this, comparing network bandwidth data with the target number of object storage in the structured configuration data achieves a dynamic correlation assessment between user needs and hardware limitations, demonstrating automatic alignment between configuration data and hardware capabilities. Finally, when the target number of object storage exceeds the matching threshold, the system automatically adjusts this number, instantly optimizing unreasonable configurations. This ensures the stability of the deployment process and the reliability of system operation, solving the problem of the lack of an automatic matching mechanism between network bandwidth and the target number of object storage, significantly reducing the need for manual intervention, and improving deployment efficiency and overall system performance.

[0089] In one feasible implementation, the step of constructing a cluster deployment instruction sequence based on the optimized configuration data and the Lustre deployment parameter set includes: merging network address parameters in the optimized configuration data with network binding policies in the Lustre deployment parameter set; generating a package distribution instruction sequence based on the merging result; generating a cluster configuration instruction sequence based on the target number of object storage in the Lustre deployment parameter set; and combining the package distribution instruction sequence and the cluster configuration instruction sequence in execution order to generate the cluster deployment instruction sequence.

[0090] In this embodiment, the network address parameter refers to the logical address information used to identify network devices, which can be implemented using IPv4 or IPv6 address formats. Its purpose is to ensure accurate positioning in network communication. The network binding strategy can be a set of rules defining the working mode of a network interface, specifically implemented through a link aggregation control protocol or manual configuration. Its purpose is to improve the reliability of network transmission and bandwidth utilization. The target number of object storage refers to the total number of independently addressable storage units in a distributed storage system. In practical applications, it can be dynamically adjusted according to hardware resources to optimize the allocation efficiency of storage resources.

[0091] In this embodiment, the solution integrates network configuration and storage resource information to construct a coordinated and consistent cluster deployment instruction sequence. First, network address parameters and network binding strategies are deeply integrated. This integration mechanism ensures the consistency of network configuration information and effectively avoids network setting conflicts caused by isolated parameters. Then, a software package distribution instruction sequence is generated based on the integrated network status. This approach allows distribution instructions to accurately match the actual network environment, significantly improving the accuracy and efficiency of software package distribution. Simultaneously, the generation of the cluster configuration instruction sequence is guided by the target number of object storage units. This quantitative configuration method ensures dynamic adaptation of storage resources and hardware capabilities. Finally, by scientifically arranging the instruction execution process, the software package distribution and cluster configuration steps are logically connected. This execution order design avoids resource contention and step conflicts, ensuring the overall reliability of the deployment operation. Thus, this application solves the problems of a lack of effective integration mechanism between network address parameters and network binding strategies, and the failure to use the target number of object storage units to guide the generation of cluster configuration instruction sequences. It achieves parameter uniformity and execution rationality in the deployment instruction sequence, thereby improving the deployment success rate and operational performance of the Lustre storage system.

[0092] In one feasible implementation, the steps of creating a system state snapshot and recording the association between the snapshot and the instruction include: identifying the current system storage volume status; creating a logical volume snapshot using a storage volume management tool; and recording the association between the logical volume snapshot and the current instruction.

[0093] In this embodiment, identifying the current system storage volume status refers to detecting and confirming the real-time operating status of the storage volume before performing a snapshot operation. The purpose is to ensure that the storage volume is in a stable state when the snapshot is triggered, thereby avoiding the generation of invalid snapshots due to abnormal volume status. In practical applications, this function can be achieved by querying the I / O activity status of the storage volume or checking mount point information. Creating logical volume snapshots through storage volume management tools allows for snapshot operations to be completed by calling the functional modules of the storage volume management tool using standardized interfaces. This aims to improve the compatibility and stability of snapshot generation. For example, LVM (Logical Volume Management) tools or other storage management frameworks that support snapshot functionality can be used. The specific implementation method can be selected based on the underlying storage architecture. Recording the association between the logical volume snapshot and the current instruction refers to dynamically binding the generated logical volume snapshot with the specific instruction that triggered the snapshot. The purpose is to provide accurate status positioning for possible subsequent system recovery. In practical applications, this function can be achieved by adding information such as timestamps and instruction identifiers to the metadata.

[0094] In this embodiment, the above technical solution effectively solves the problem of unreliable state recovery during deployment by defining the specific operation process for creating and associating system state snapshots. First, the operation of identifying the current system storage volume state ensures the stability of the system environment when a snapshot is triggered, providing a reliable foundation for subsequent recovery. Second, the operation of creating logical volume snapshots through storage volume management tools not only improves the compatibility of snapshot generation but also avoids deviations that may be caused by manual operations, significantly enhancing the repeatability of snapshot generation. Finally, the operation of recording the association between logical volume snapshots and current instructions achieves dynamic binding between snapshots and instruction contexts, enabling accurate location of the historical state point of the corresponding instruction in deployment interruption scenarios, thereby avoiding state misalignment problems during the recovery process.

[0095] Building upon this, the aforementioned technical solution, combined with the execution mechanism of a transactional deployment engine, further enhances the ability to quickly roll back in the event of deployment failure. By creating a system state snapshot and recording the relationships before each instruction is executed, it ensures that the most recent historical state point can be accurately located in the event of any instruction failure, thereby achieving efficient and accurate system recovery. Simultaneously, this mechanism also provides a clear basis for clearing intermediate data files, reducing the risk of data inconsistency.

[0096] In one feasible implementation, the step of creating a logical volume snapshot using a storage volume management tool includes: sending a snapshot creation instruction containing a storage volume identifier to the storage volume management tool; and executing the snapshot creation instruction to generate a logical volume snapshot.

[0097] In this embodiment, the storage volume identifier refers to a string or numeric code used to uniquely identify a storage volume. It can be implemented using a UUID, device path, or user-defined label. The purpose of introducing the storage volume identifier is to ensure accurate location of the target storage volume in a multi-volume environment, avoiding operational errors caused by ambiguous volume identification. The snapshot creation command can be a standardized command format, typically including information such as operation type, target object, and execution parameters. The aim is to reduce issues such as missing parameters or incorrect sequence caused by manual intervention by standardizing the command content.

[0098] In this embodiment, the above-described solution sends a snapshot creation command containing a storage volume identifier to the storage volume management tool. The uniqueness of the identifier ensures that the command only applies to the specified storage volume, effectively solving the problem of snapshot object confusion in complex hardware topology scenarios. Subsequently, the snapshot creation command is executed to generate a logical volume snapshot. This process directly binds command execution and snapshot generation, eliminating timing deviations that may be introduced by manual intervention and ensuring the timeliness and completeness of system state capture. Furthermore, the design of this process also considers integration with deployment failure rollback mechanisms, providing a reliable foundation for rapid system state recovery. Thus, when facing complex hardware environments, this not only improves the reliability of logical volume snapshot creation but also significantly enhances the fault tolerance of the deployment process, thereby reducing the risk of deployment interruption. Simultaneously, this solution, combined with system state snapshot recording and fault rollback mechanisms, further optimizes the stability and efficiency of the overall deployment process.

[0099] In one feasible implementation, the method further includes: when an instruction execution failure is detected, locating the nearest logical volume snapshot based on the association relationship; restoring the system state recorded by the logical volume snapshot; and clearing the intermediate data files generated by the instruction execution failure.

[0100] In this embodiment, the association relationship refers to the mapping relationship between snapshots and instructions that have been accurately recorded before instruction execution. This can be implemented using an index structure based on the instruction execution sequence context, aiming to significantly improve the accuracy and efficiency of snapshot retrieval. A logical volume snapshot can be a state image of the storage system at a specific moment. It can be generated by storage volume management tools and can completely retain key state information during deployment, aiming to ensure the system rolls back to a consistent and predictable baseline. Intermediate data files refer to temporary files generated during instruction execution, which may include configuration caches, log fragments, or partially written data blocks. Clearing these files aims to prevent residual data from conflicting with subsequent operations or polluting the environment.

[0101] In this embodiment, the technical solution effectively solves the problem of inconsistent system states caused by command failures during deployment by constructing a closed-loop fault recovery mechanism. First, upon detecting command execution failure, the system immediately triggers a recovery process to prevent fault propagation and ensure timely response. Then, based on pre-established associations, the system can quickly locate the nearest logical volume snapshot. Since the associations are directly based on the context of the command execution sequence, snapshot retrieval can be efficiently completed without relying on timestamps or global searches. Next, the system restores the state to the baseline recorded in the logical volume snapshot, thereby eliminating intermediate state remnants caused by failed commands and ensuring the system returns to a stable and consistent state. Finally, intermediate data files generated during the failure process are actively cleaned up to maintain the purity of the deployment environment. These steps work together to not only avoid the tedious manual recovery operations but also ensure the continuity and repeatability of the deployment process, fundamentally reducing the operational risks caused by deployment failures.

[0102] Furthermore, this technical solution is closely integrated with the aforementioned steps of creating system state snapshots and recording correlations. By leveraging the precise mapping between snapshots and instructions, it further improves the efficiency and reliability of fault recovery. This design not only enhances the automation level of the deployment process but also demonstrates higher fault tolerance in complex hardware environments and diverse user needs, thereby significantly improving the deployment experience of Lustre storage systems.

[0103] In the embodiments of this application, the Lustre storage system deployment method uses technologies such as automated hardware detection agent, parameter generation rule engine, conflict rule base verification and transactional deployment engine to achieve standardization and intelligence of the entire Lustre storage system deployment process. This effectively avoids data omissions and misjudgments caused by manual operation, ensures accurate matching between configuration parameters and hardware capabilities, and can prevent configuration conflicts in real time. It has the advantages of efficient deployment process, accurate configuration, preventable conflicts, and recoverable state, thus improving the deployment efficiency and operational stability of the Lustre storage system.

[0104] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the deployment method of the Lustre storage system in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0105] This application also provides a Lustre storage system deployment system, see reference. Figure 2 The Lustre storage system deployment system includes: a memory 10, a processor 20, and a Lustre storage system deployment program stored on the memory 10 and executable on the processor 20. The Lustre storage system deployment program is configured to implement the steps of the Lustre storage system deployment method.

[0106] The Lustre storage system deployment system provided in this application, employing the Lustre storage system deployment method in the above embodiments, can improve the deployment efficiency and operational stability of the Lustre storage system. Compared with the prior art, the beneficial effects of the Lustre storage system deployment system provided in this application are the same as those of the Lustre storage system deployment method provided in the above embodiments, and other technical features of the Lustre storage system deployment system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0107] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0108] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. All equivalent structural transformations made under the technical concept of this application using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included within the scope of patent protection of this application.

Claims

1. A lustre storage system deployment method characterized by, The method includes: Hardware topology feature data of the target server is collected through a hardware detection agent; The hardware topology feature data is input into the parameter generation rule engine to generate the Lustre deployment parameter set. Obtain the deployment template file filled in by the user, and parse the deployment template file to obtain structured configuration data; The structured configuration data and the Lustre deployment parameter set are input into the conflict rule base for verification to generate optimized configuration data; A cluster deployment instruction sequence is constructed based on the optimized configuration data and the Lustre deployment parameter set; The cluster deployment instruction sequence is executed through a transactional deployment engine, and a deployment report is output; wherein, a system state snapshot is created before executing each instruction and the association between the snapshot and the instruction is recorded; The steps of inputting the structured configuration data and the Lustre deployment parameter set into the conflict rule base for verification and generating optimized configuration data include: Load a conflict rule base containing thresholds for matching network bandwidth and the target number of object storage items; Extract the target number of objects stored in the Lustre deployment parameter set; Compare the network bandwidth data in the structured configuration data with the target number of objects to be stored; When the target number of objects to be stored exceeds the matching threshold, the target number of objects to be stored is adjusted to output the optimized configuration data; The steps for constructing a cluster deployment instruction sequence based on the optimized configuration data and the Lustre deployment parameter set include: Merge the network address parameters in the optimized configuration data with the network binding strategy in the Lustre deployment parameter set; Generate a software package distribution instruction sequence based on the merge results; Generate a cluster configuration instruction sequence based on the target number of object storage in the Lustre deployment parameter set; The cluster deployment instruction sequence is generated by combining the package distribution instruction sequence and the cluster configuration instruction sequence in the execution order.

2. The Lustre storage system deployment method as described in claim 1, characterized in that, The step of collecting hardware topology feature data of the target server through a hardware detection agent includes: Execute processor detection instructions to obtain information on the number of CPU cores and the distribution of NUMA nodes; Execute network device detection commands to obtain network interface card type and queue depth data; Execute storage device detection commands to obtain disk type and quantity data; The hardware topology feature data is generated by integrating the CPU core count and NUMA node distribution information, the network card type and queue depth data, and the disk type and quantity data.

3. The Lustre storage system deployment method as described in claim 2, characterized in that, The steps for executing network device detection commands to obtain network interface card (NIC) type and queue depth data include: Query the network card model identifier through the network card driver interface; Retrieve the network card capability lookup table based on the network card model identifier; When RDMA protocol support is detected, activate the network card's RoCE mode and obtain queue depth data.

4. The Lustre storage system deployment method as described in claim 2, characterized in that, The steps of inputting the hardware topology feature data into the parameter generation rule engine to generate the Lustre deployment parameter set include: The number of service threads is calculated based on the number of CPU cores in the hardware topology feature data. Calculate the target number of object storage based on the number of disks in the hardware topology feature data; A network bonding strategy is generated based on the network interface card (NIC) type in the hardware topology feature data. The Lustre deployment parameter set is generated by combining the number of service threads, the number of object storage targets, and the network binding strategy.

5. The Lustre storage system deployment method as described in claim 1, characterized in that, The steps for creating a system state snapshot and recording the association between the snapshot and the instructions include: Identify the current system storage volume status; Create logical volume snapshots using storage volume management tools; Record the association between the logical volume snapshot and the current instruction.

6. The Lustre storage system deployment method as described in claim 5, characterized in that, The steps to create a logical volume snapshot using a storage volume management tool include: Send a snapshot creation command containing the storage volume identifier to the storage volume management tool; Execute the snapshot creation instruction to generate a logical volume snapshot.

7. The Lustre storage system deployment method as described in claim 1, characterized in that, The method further includes: When an instruction execution failure is detected, the nearest logical volume snapshot is located based on the aforementioned association. Restore the system to the state recorded in the logical volume snapshot; Clear intermediate data files generated when command execution fails.

8. A Lustre storage system deployment system, characterized in that, The Lustre storage system deployment system includes: a memory, a processor, and a Lustre storage system deployment program stored on the memory and executable on the processor, the Lustre storage system deployment program being configured to implement the steps of the Lustre storage system deployment method as described in any one of claims 1 to 7.

Images