Disk replacement method for distributed storage cluster and electronic device

Abstract

The application discloses a disk replacement method for a distributed storage cluster and electronic equipment, and relates to the technical field of computer storage. The method comprises the following steps: in response to a disk replacement instruction, monitoring whether the distributed storage cluster is in a healthy state; in the case that the distributed storage cluster is in the healthy state, obtaining object storage device configuration information of a target storage pool, and outputting disk replacement operation prompt information, wherein the target storage pool comprises a hard disk to be replaced; when it is detected that the disk replacement operation is completed, checking whether there is misplaced data in the distributed storage cluster; if there is misplaced data, performing a preset recovery operation based on the object storage device configuration information to eliminate the misplaced data; and performing a preset reconstruction operation after the misplaced data is eliminated, wherein the reconstruction operation comprises writing data stored in the hard disk to be replaced into a newly replaced hard disk. Through the technical scheme provided by the application, the disk replacement operation of the distributed storage cluster can be efficiently, stably and safely realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer storage, and in particular to a distributed storage cluster disk replacement method and an electronic device. BACKGROUND

[0002] With the development of new generation information technologies such as cloud computing, big data, and artificial intelligence, distributed storage clusters have gradually become the mainstream solution in the field of data storage due to their high scalability, high fault tolerance, and elastic resource allocation.

[0003] In a distributed storage cluster, the selection and configuration of hard disks as the core storage medium will directly affect the performance of the distributed storage cluster. Among them, the hard disk will have a situation of expiration or bad disk during use, at which time the hard disk needs to be replaced to prolong the service life of the distributed storage cluster. However, how to efficiently, stably, and safely complete the disk replacement operation of the distributed storage cluster under the condition that the storage pool is at a high water level is a technical problem that needs to be solved at present. SUMMARY

[0004] The present application provides a distributed storage cluster disk replacement method and an electronic device, which can efficiently, stably, and safely complete the disk replacement operation of the distributed storage cluster under the condition that the storage pool is at a high water level.

[0005] The present application provides a distributed storage cluster disk replacement method, which comprises:

[0006] In response to a disk replacement instruction, it is monitored whether the distributed storage cluster is in a healthy state; the disk replacement instruction includes the name of a target storage pool, and the storage water level of the target storage pool is greater than or equal to a preset water level threshold.

[0007] In the case that the distributed storage cluster is in a healthy state, the object storage device configuration information of the target storage pool is obtained, and a disk replacement operation prompt information is output; the target storage pool includes a hard disk to be replaced, and the object storage device configuration information includes at least one of the following: topology structure information of the target storage pool, object storage device mapping information, placement group mapping rule information, and placement group distribution information.

[0008] When it is detected that the disk replacement operation is completed, it is checked whether there is misplaced data in the distributed storage cluster.

[0009] If there is misplaced data, a preset recovery operation is performed based on the object storage device configuration information to eliminate the misplaced data; and after the misplaced data is eliminated, a preset reconstruction operation is performed, which includes writing the data stored in the hard disk to be replaced into a newly replaced hard disk.

[0010] The present application also provides a distributed storage cluster disk replacement device, which comprises:

[0011] The monitoring module is configured to monitor whether the distributed storage cluster is in a healthy state in response to a disk replacement instruction, wherein the disk replacement instruction comprises a name of a target storage pool, and a storage level of the target storage pool is greater than or equal to a preset water level threshold.

[0012] The obtaining module is configured to obtain object storage device configuration information of the target storage pool and output a disk replacement operation prompt information when the distributed storage cluster is in the healthy state, wherein the target storage pool comprises a hard disk to be replaced, and the object storage device configuration information comprises at least one of the following: topology structure information of the target storage pool, object storage device mapping information, placement group mapping rule information, and placement group distribution information.

[0013] The checking module is configured to check whether there is misplaced data in the distributed storage cluster when it is detected that the disk replacement operation is completed.

[0014] The recovery module is configured to, if there is misplaced data, execute a preset recovery operation based on the object storage device configuration information to eliminate the misplaced data, and execute a preset reconstruction operation after the misplaced data is eliminated, wherein the reconstruction operation comprises writing data stored in the hard disk to be replaced into a newly replaced hard disk.

[0015] The application further provides an electronic device, comprising a memory configured to store a computer program, and a processor configured to execute the computer program to implement the steps of any of the distributed storage cluster disk replacement methods.

[0016] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of any of the distributed storage cluster disk replacement methods.

[0017] The application further provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the steps of any of the distributed storage cluster disk replacement methods.

[0018] The distributed storage cluster disk replacement method and the electronic device provided by the application, before the disk replacement operation is performed, whether the distributed storage cluster is in a healthy state is monitored, in the case that the distributed storage cluster is in the healthy state, the object storage device configuration information of a target storage pool (the storage level is greater than or equal to a preset water level threshold) is acquired, and disk replacement operation prompt information is output, which can ensure that the disk replacement operation is performed in a safe environment, and by acquiring the object storage device configuration information of the target storage pool, subsequent data recovery can be ensured; after the disk replacement operation is completed, if there is misplaced data in the distributed storage cluster, the preset recovery operation can be performed based on the object storage device configuration information, the misplaced data is automatically eliminated, and the consistency of data is ensured, so that the influence of the disk replacement operation on the performance of the distributed storage cluster is reduced, and the disk replacement of the distributed storage cluster is efficiently, stably and safely completed. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 A flowchart of a distributed storage cluster disk replacement method provided in an embodiment of the application;

[0021] Figure 2 Another flowchart of a distributed storage cluster disk replacement method provided in an embodiment of the application;

[0022] Figure 3 A structure diagram of a distributed storage cluster disk replacement device provided in an embodiment of the application;

[0023] Figure 4 A structure diagram of an electronic device provided in an embodiment of the application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the application will be described clearly and completely in combination with the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0025] It should be noted that in the description of the present application, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0026] Some data involved in the embodiments of the present application are explained as follows:

[0027] Object Storage Device (OSD): The most basic storage unit in a distributed storage cluster, responsible for data storage, replication, recovery and load balancing, is a key component to ensure data high availability, consistency and cluster performance.

[0028] Controlled Replication Under Scalable Hashing (CRUSH) algorithm: A controllable, scalable, distributed replica data placement algorithm, mainly used in distributed storage clusters, determines the storage location of data in the cluster through calculation, realizes efficient data distribution, load balancing and fault tolerance management.

[0029] Placement Group (PG): The core logical unit for data distribution, load balancing and fault tolerance management in a distributed storage cluster, its design significantly reduces cluster complexity and improves scalability by grouping objects for management. As an intermediate layer between objects and OSDs, PGs perform the following key functions:

[0030] (1) Object aggregation: Grouping management of massive objects (such as millions) in a storage pool, avoiding maintaining metadata (such as location information) for each object individually, thereby reducing computational overhead. For example, a storage pool containing 1 million objects can be divided into 1000 PGs, each managing about 1000 objects.

[0031] (2) Data distribution: Map PGs to OSDs through the CRUSH algorithm to achieve uniform distribution of data in the cluster.

[0032] (3) Load balancing: Dynamically adjust the mapping relationship between PGs and OSDs to avoid overloading a single OSD. For example, when the disk usage of an OSD exceeds a threshold, the storage cluster will migrate some PGs to other idle OSDs.

[0033] With the rapid development and deep integration of new generation information technologies such as cloud computing, big data, artificial intelligence, the data generated by various business applications is rapidly growing. In this context of big data era, distributed storage clusters have gradually become the mainstream solution in the field of mass data storage due to their high scalability, high fault tolerance, and elastic resource allocation, and their application scenarios are increasingly widespread, covering many industries such as Internet services.

[0034] With the continuous deployment and deepening application of distributed storage architecture, the amount of storage data carried by storage clusters is constantly breaking records. This explosive growth of data directly leads to a sharp increase in the number of hard disk nodes required in distributed storage clusters, forming a super-large-scale storage array composed of thousands of hard disks. In this context, the stability, reliability, and persistence of hard disks as the basic carrier for data persistence storage have become key factors to ensure the high availability of the entire distributed storage cluster.

[0035] In the fine operation and management of distributed storage clusters, the high water level state of the storage pool as a key capacity warning indicator (usually defined as the actual capacity utilization of the storage pool reaching or exceeding the preset threshold, for example, 95%) triggers the storage cluster to enter the capacity sensitive period. However, the traditional hard disk replacement strategy does not fully consider the special constraints in the high water level scenario during the design phase, resulting in the following multi-dimensional technical problems in actual operation:

[0036] Capacity dynamic over-limit: When the storage pool is in a high water level state, the traditional disk replacement operation needs to perform data migration pre-allocation space verification. Due to the fact that the remaining available capacity has approached the safety boundary, the intermediate copies or temporary metadata generated during the data migration process may instantly exceed the physical capacity limit, triggering the capacity hard isolation mechanism of the storage controller. This situation will directly lead to the denial of service of new write requests, or force the cluster to automatically start the data elimination strategy based on priority, thereby causing a sharp increase in write delay or even temporary interruption of critical business data.

[0037] Data migration efficiency reduction: Traditional disk replacement operations usually rely on PG rebalancing algorithms, which will forcibly trigger the cross-node data redistribution of all PGs in the storage pool. Under high water level conditions, the input / output (I / O) channels of the storage cluster are already in a high load running state, and at this time, PG rebalancing will compete with business I / O for resources, resulting in a nonlinear increase in I / O response time. In extreme cases, it may cause a sharp increase in the depth of the I / O request queue, causing a cliff-like drop in the overall throughput of the storage cluster.

[0038] Data reconstruction consistency out of control: in the high water level state, the distribution density of PG data shards among storage nodes is significantly improved, and the PG recovery process after the traditional disk replacement operation lacks an intelligent distribution verification mechanism. When a new hard disk is connected, if the PG mapping relationship recovery algorithm fails to accurately match the original data topology, a large number of data shards will be incorrectly placed on non-optimal nodes. This distribution misplacement will trigger a secondary reconstruction effect - the storage cluster needs to perform additional cross-node data correction operations, not only prolonging the data recovery time, but also possibly causing link bandwidth exhaustion due to multiple data transfers.

[0039] The above technical problems will systematically threaten the reliability of the storage cluster, specifically manifested as: high risk of data loss, unstable cluster state, and long recovery time, which is difficult to meet the efficient and safe disk replacement needs.

[0040] In addition, if problems occur during the traditional disk replacement process, manual adjustment of PG data distribution and manual control of data migration volume are usually required to avoid large data migration. However, this approach not only has high labor costs, but also is prone to human error and has poor fault tolerance.

[0041] Based on the above, how to efficiently, stably, and safely complete the disk replacement operation of the distributed storage in the high water level state is a technical problem that needs to be solved by the technical personnel in the field at present.

[0042] To solve the above technical problems, the embodiment of the present application provides a distributed storage cluster disk replacement method. Before performing the disk replacement operation, the method first monitors whether the distributed storage cluster is in a healthy state. If the distributed storage cluster is in a healthy state, the object storage device configuration information of the target storage pool (the storage water level is greater than or equal to the preset water level threshold) is obtained to ensure that the disk replacement operation is performed in a safe environment. After the disk replacement operation is completed, if there is misplacement data in the distributed storage cluster, the pre-set recovery operation can be performed based on the object storage device configuration information to automatically eliminate the misplacement data, thereby reducing the impact of the disk replacement operation on the performance of the distributed storage cluster and efficiently, stably, and safely completing the disk replacement of the distributed storage cluster.

[0043] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0044] Referring to Figure 1 , Figure 1 The flowchart of the distributed storage cluster disk replacement method provided in the embodiment of the present application. In some embodiments, the distributed storage cluster disk replacement method includes:

[0045] S101, in response to the disk replacement instruction, monitoring whether the distributed storage cluster is in a healthy state. If yes, continue to perform S102; if no, exit the current disk replacement process.

[0046] Optionally, the disk replacement instruction includes the name of the target storage pool. The storage level of the target storage pool is greater than or equal to a preset water level threshold (such as 95%).

[0047] In some embodiments, the state indicators of the storage cluster nodes can be periodically collected through the monitoring interface of the distributed storage cluster, including but not limited to: storage node online state, network link bandwidth utilization, I / O request response time, data replica synchronization state and other key parameters.

[0048] By comprehensively analyzing the above-mentioned state indicators, it can be determined whether the distributed storage cluster is in a healthy state. For example, when all key parameters are within the preset threshold range, it can be determined that the distributed storage cluster is in a maintainable healthy state.

[0049] S102, obtaining object storage device configuration information of the target storage pool, and outputting disk replacement operation prompt information; wherein the target storage pool includes a hard disk to be replaced.

[0050] In some embodiments, when it is confirmed that the distributed storage cluster is in a maintainable healthy state, the OSD configuration information of the target storage pool can be obtained through the object storage management interface.

[0051] For example, the OSD configuration information can include topology structure information, OSD mapping information, PG mapping rule information, PG distribution information, etc. of the target storage pool, which is not limited in the embodiments of the present application.

[0052] In some embodiments, after obtaining the object storage device configuration information of the target storage pool, the management console can output visual disk replacement operation prompt information to the maintenance personnel. The disk replacement operation prompt information can include the position identification of the device to be replaced, data migration path description and safety operation precautions. The maintenance personnel can replace the hard disk to be replaced according to the disk replacement operation prompt information.

[0053] S103, when detecting that the disk replacement operation is completed, checking whether there is misplaced data in the distributed storage cluster. If yes, continue to perform S104; if no, perform S105.

[0054] In some embodiments, the state change of the hard disk tray can be monitored in real time through the hardware management interface, and combined with the alarm information in the storage log, if it is confirmed that the old hard disk has been physically removed and the new hard disk has completed the power-on identification, it can be determined that the disk replacement operation is completed.

[0055] Upon completion of the disk replacement operation, a data consistency check is performed to determine whether there is misplaced data in the distributed storage cluster by comparing the storage pool metadata records with the actual physical layout.

[0056] For example, the following three misplaced scenarios can be identified:

[0057] Orphaned shard: metadata record exists but physical device is missing.

[0058] Ghost shard: physical device exists but no corresponding metadata record.

[0059] Location mismatch: actual storage location of the shard does not match the metadata record.

[0060] S104, based on the above object storage device configuration information, a pre-set recovery operation is performed to eliminate the misplaced data.

[0061] In some embodiments, if there is misplaced data, the previously obtained OSD configuration information can be imported, and the original configuration of the target storage pool is restored using the OSD configuration information, thereby eliminating the misplaced data.

[0062] After performing the recovery operation, the data consistency check can be performed again to determine whether there is misplaced data in the distributed storage cluster. When it is determined that there is no misplaced data in the distributed storage cluster, step S105 is performed.

[0063] S105, a pre-set reconstruction operation is performed, which includes writing data stored in the hard disk to be replaced to the newly replaced hard disk.

[0064] In some embodiments, when it is determined that there is no misplaced data in the distributed storage cluster, a data reconstruction engine can be started, and based on the current OSD configuration information and the real-time state of the cluster, a phased and interruptible efficient data migration is performed to write data stored in the hard disk to be replaced to the newly replaced hard disk.

[0065] The disk replacement method provided in the embodiments of the present application can ensure that the disk replacement operation is performed in a safe environment, and can provide guarantee for subsequent data recovery by obtaining the object storage device configuration information of the target storage pool; after the disk replacement operation is completed, if there is misplaced data in the distributed storage cluster, the preset recovery operation can be performed based on the object storage device configuration information to automatically eliminate the misplaced data and ensure data consistency, thereby reducing the impact of the disk replacement operation on the performance of the distributed storage cluster, and efficiently, stably and safely completing the disk replacement of the distributed storage cluster.

[0066] In some embodiments, the step S101 of monitoring whether the distributed storage cluster is in a healthy state comprises:

[0067] The OSD list associated with the target storage pool is obtained, and the state of each OSD in the OSD list is detected; when the state of each OSD in the OSD list is normal, the running parameter of the distributed storage cluster is obtained; and based on the running parameter, it is determined whether the distributed storage cluster is in a healthy state.

[0068] If there is an OSD with an abnormal state in the OSD list, it can be determined that the distributed storage cluster is not in a healthy state.

[0069] For example, the OSD list associated with the target storage pool can be queried by using a management tool of the distributed storage cluster. The obtained OSD list is traversed, and state checking is performed on each OSD, for example, the state information of each OSD is obtained by using a command provided by the distributed storage cluster, the state information of the OSD is checked, and it is judged whether it is normal; if any OSD with an abnormal state is found in the OSD list, the information of the OSD is immediately recorded, and the cluster is marked as an unhealthy state.

[0070] After it is confirmed that all OSDs are in a normal state, the overall running parameter of the cluster is obtained, for example, the health state of the cluster, the data distribution, the usage rate, the I / O performance index, and the like. Based on the running parameter, it is determined whether the distributed storage cluster is in a healthy state.

[0071] Optionally, the running parameter comprises a state parameter of the distributed storage cluster and the number of blocked interfaces; and the determination of whether the distributed storage cluster is in a healthy state based on the running parameter comprises:

[0072] When the health state field is included in the state parameter and the number of blocked interfaces is less than the alarm number threshold, it is determined that the distributed storage cluster is in a healthy state.

[0073] Exemplarily, the health status field of the cluster (e.g., HEALTH_OK, HEALTH_WARN, HEALTH_ERR) can be checked. If the cluster is in the health status and the number of blocked interfaces is less than the alarm number threshold, it can be determined that the distributed storage cluster is in the health status. If the health status of the cluster is in the health warning state or the failure state, or the number of blocked interfaces is greater than or equal to the alarm number threshold, it is determined that the storage cluster is not in the health status.

[0074] In some embodiments, the OSD list associated with the target storage pool can be acquired, the OSD state is verified to be normal, and whether the storage cluster running health status and whether there is a serious I / O block exist are judged. If the cluster health status check passes, the disk replacement process is continued, otherwise the process is exited.

[0075] In the embodiment of the application, by monitoring the health status of the distributed storage cluster in real time, it can be ensured that the disk replacement operation is performed in a safe environment.

[0076] In some embodiments, the OSD configuration information of the target storage pool includes:

[0077] Based on the name of the target storage pool, a directory for storing configuration files in the disk replacement process is established; the OSD configuration information of the target storage pool is acquired, and the acquired OSD configuration information is saved to the above directory according to the current timestamp.

[0078] In some embodiments, the distributed storage cluster can receive user input parameters when the script is executed, generate and save the OSD configuration information of the target storage pool according to the timestamp.

[0079] Optionally, the OSD configuration information includes at least one of the following information:

[0080] The topology structure information (Crushmap) of the target storage pool, the topology structure information including the hierarchical relationship of the hard disks in the target storage pool; the OSD mapping information (OSDmap) generated based on the topology structure information; the PG mapping rule information (Upmap); and the PG distribution information (PG Dump).

[0081] In some embodiments, before replacing the hard disk, the OSD configuration information of the target storage pool can be acquired, including generating the Crushmap of the target storage pool to save the hierarchical relationship of the hard disks, generating the Upmap file by combining the OSDmap acquired by using the OSD mapping information tool (osdmaptool), and acquiring the PG Dump to solidify the data distribution logic before disk replacement.

[0082] In some embodiments, the topology structure information of the target storage pool can be generated according to the identification information of the hard disks in the target storage pool.

[0083] Optionally, the identification information of the hard disks can include a hard disk serial number, a device path, slot information, an interface type, etc. According to the identification information of each hard disk in the target storage pool, the connection relationship between the hard disks in the target storage pool is determined, and then the topology structure information of the target storage pool is generated.

[0084] In some embodiments, OSD mapping information of the target storage pool can be generated according to the identification information of the OSDs in the target storage pool.

[0085] For example, by analyzing the unique identification of each OSD in the storage pool, and combining the physical / logical attribution relationship (such as the host to which the OSD belongs, the rack, the hard disk type, etc.), a structured mapping table is generated to clearly indicate the position and attributes of each OSD in the storage pool.

[0086] In some embodiments, placement group mapping rule information of the target storage pool can be generated according to the OSD mapping information.

[0087] For example, by analyzing the OSD mapping information (such as the hierarchical attribution, weight, and state of the OSD), and combining the replica strategy of the storage pool and the CRUSH algorithm rule, a rule set is generated to clearly indicate which OSDs each placement group should be mapped to, so as to achieve high availability and balanced distribution of data.

[0088] In some embodiments, placement group distribution information of the target storage pool can be generated according to the identification information of the placement groups in the target storage pool.

[0089] For example, by analyzing the unique identification of each placement group in the target storage pool, and combining the OSD mapping information and the CRUSH algorithm rule, a structured distribution table is generated to clearly indicate the combination of the OSDs to which each PG is currently actually mapped and the physical distribution (such as the host and rack to which the PG belongs).

[0090] In the embodiments of the present application, the Upmap optimization mapping rule is generated by using the OSD mapping information tool, and the crushmap and PG Dump information are exported to solidify the data distribution logic before disk replacement, to provide a reliable data basis for subsequent recovery operations, avoid complex problems caused by data migration and distribution changes in the high water level, and ensure the safety and consistency of the disk replacement process.

[0091] In some embodiments, the above checking whether there is misplaced data in the distributed storage cluster includes:

[0092] determine whether the distributed storage cluster is in a stable state, and perform consistency checking on the distributed storage cluster to determine whether there is misplaced data in the distributed storage cluster when it is determined that the distributed storage cluster is in the stable state.

[0093] In some embodiments, after detecting that the user has replaced the hard disk, peering can be performed, and after the peering is completed, it is determined whether the distributed storage cluster is in a stable state, and when it is determined that the distributed storage cluster is in the stable state, consistency checking is performed on the distributed storage cluster to determine whether there is misplaced data.

[0094] In the embodiments of the present application, when it is determined that the distributed storage cluster is in a stable state, consistency checking is performed on the distributed storage cluster, which can guarantee the accuracy of the consistency checking result.

[0095] In some embodiments, if there is no misplaced data, the cluster reconstruction operation can be started; if it is detected that there is misplaced data in the storage cluster, a preset recovery operation is performed. The recovery operation includes: updating the configuration parameters of the target storage pool based on the object storage device configuration information; and updating the distribution position of the data stored in the distributed storage cluster based on the updated configuration parameters of the target storage pool.

[0096] In some embodiments, if it is detected that there is misplaced data in the storage cluster, the OSD configuration information saved before the disk replacement can be re-imported to restore the target storage pool to the original configuration, and the cluster reconstruction operation is started again after waiting for the consistency checking to be completed and the misplaced data to disappear.

[0097] In the embodiments of the present application, the recovery operation is performed based on the OSD configuration information, which avoids the reconstruction failure or performance degradation problem caused by improper data distribution in the high water level case, and improves the recovery efficiency and reliability after the disk replacement.

[0098] In some embodiments, if it is detected that there is misplaced data in the storage cluster, the newly generated OSD configuration information after the disk replacement can also be backed up, and when the OSD configuration information saved before the disk replacement cannot make the misplaced data disappear, the OSD configuration information generated after the disk replacement can also be restored again, which ensures the continuity and stability of the storage cluster service.

[0099] In some embodiments, during the execution of the reconstruction operation, it is monitored whether there is an alarm message or a blocked interface in the distributed storage cluster; when there is an alarm message or a blocked interface in the distributed storage cluster, the reconstruction operation is suspended, and identification information of the alarm message or the blocked interface is output; when it is detected that the alarm message is eliminated or the blocked interface is restored to normal, the reconstruction operation is continued.

[0100] Exemplarily, after the storage cluster starts normal reconstruction, the storage reconstruction progress and the cluster health status can be monitored in real time until the reconstruction is completed. If an alarm or I / O blockage occurs during the reconstruction, the reconstruction is suspended in time, and the user is prompted to check. After the alarm is eliminated, the reconstruction operation is continued until the reconstruction is completed.

[0101] In the embodiments of the present application, by automatically detecting and processing mispositioned data, the problem of reconstruction failure or performance decline caused by improper data distribution under high water level is avoided, and the recovery efficiency and reliability after disk replacement are improved.

[0102] Referring to Figure 2 , Figure 2 Another flowchart of a distributed storage cluster disk replacement method provided in the embodiments of the present application is shown. In some embodiments, the distributed storage cluster disk replacement method includes:

[0103] S201, in response to a disk replacement instruction, whether the distributed storage cluster is in a healthy state is monitored. If yes, S202 is continued to be executed.

[0104] It can be understood that, since the present application is a tool for replacing disks of a distributed storage cluster under high water level, a script program can be deployed on the nodes of the distributed storage, and the script program is added with executable permission by using a "chmod+x" command.

[0105] In some embodiments, the user can trigger the execution of the script program, and input a target storage pool name and pre-disk replacement operation parameters. The storage cluster checks the input parameters to ensure the correctness of the parameters, then acquires an OSD list corresponding to the target storage pool and checks whether the OSD list is in a normal state, and monitors whether the number of I / O blocked blocks is greater than a threshold value. If the cluster is in a healthy state and the number of I / O blocked blocks is less than an alarm threshold value, it can be determined that the state of the storage cluster is a healthy state.

[0106] S202, object storage device configuration information of the target storage pool is acquired, and disk replacement operation prompt information is output.

[0107] In some embodiments, a directory for storing various configuration files and logs during disk replacement can be automatically created according to the target storage pool name, so as to facilitate the recording and management of the disk replacement process. Then, current Crushmap, osdmap, Upmap and PG Dump configuration information are acquired and saved with a current timestamp, which is used for subsequent data consistency check and recovery operation. Finally, the user is prompted that the disk replacement operation can be started.

[0108] In some embodiments, after the user manually replaces the physical hard disk according to the prompt, the user can input the storage pool name and post-disk replacement operation parameters.

[0109] S203, determining whether there is misplaced data when the disk replacement operation is completed.

[0110] In some embodiments, after the consistency check (peering) is completed, the storage cluster can perform data misplaced detection, scan misplaced objects, and if there is no misplaced data, directly perform S204, otherwise perform S206.

[0111] S204, performing the preset reconstruction operation.

[0112] S205, monitoring the execution of the reconstruction operation.

[0113] In some embodiments, the storage cluster reconstruction progress and the storage cluster health status can be monitored in real time until the reconstruction operation is completed.

[0114] In some embodiments, after the storage cluster starts the reconstruction operation, the storage cluster reconstruction progress and the cluster health status can be monitored in real time until the reconstruction is completed. If the storage state produces an alarm or I / O congestion during the reconstruction process, the reconstruction operation can be paused and the user can be prompted to check. After the alarm is eliminated, the reconstruction operation will continue to be executed until the reconstruction operation is completed.

[0115] S206, importing the obtained object storage device configuration information.

[0116] In some embodiments, if it is detected that the storage cluster has misplaced data, the configuration information such as Crushmap, osdmap and Upmap generated after the disk replacement can be backed up first, then the old object storage device configuration information saved by the information collection module before the disk replacement is automatically imported, and the consistency check is waited again.

[0117] S207, checking whether the misplaced data disappears. If yes, S204 is performed.

[0118] The present application can ensure that the disk replacement operation is performed in a safe environment by monitoring the cluster health status and I / O blocking in real time. At the same time, the data misplaced problem can be automatically detected and processed, avoiding reconstruction failure or performance degradation caused by improper data distribution, and significantly improving the stability and reliability of the distributed storage cluster under high water level conditions.

[0119] In this embodiment, addressing the challenge of replacing hard drives in distributed storage clusters under high water level conditions, key configuration information is obtained before drive replacement to ensure subsequent recovery; automatic detection and handling of data misalignment issues after drive replacement ensures data consistency. This embodiment not only simplifies the operation and maintenance process and reduces the risk of manual intervention, but also ensures business continuity and stability through real-time monitoring and dynamic adjustment mechanisms, providing reliable technical support for drive replacement operations in distributed storage clusters and improving the availability and operational efficiency of distributed storage clusters.

[0120] It should be noted that the method provided in this application can be extended to any industry that relies on large-scale, high-performance storage infrastructure, especially in fields with high requirements for data continuity and security, such as cloud data centers, financial industry storage systems, and telecommunications operator storage systems. It can ensure the efficient maintenance of large-scale storage infrastructure, reduce service time, and improve service quality.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that the distributed storage cluster disk replacement method described in the above embodiments can be implemented by means of software plus necessary general-purpose hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0122] This application also provides a distributed storage cluster disk-swapping device in its embodiments. For example... Figure 3 As shown, Figure 3 This is a schematic diagram of a distributed storage cluster disk switching device provided in an embodiment of this application. The aforementioned distributed storage cluster disk switching device 30 includes:

[0123] The monitoring module 301 is used to monitor whether the distributed storage cluster is in a healthy state in response to the disk replacement command; the disk replacement command includes the name of the target storage pool and the storage level of the target storage pool is greater than or equal to the preset water level threshold.

[0124] The acquisition module 302 is used to acquire the object storage device configuration information of the target storage pool when the distributed storage cluster is in a healthy state, and output disk replacement operation prompt information; the target storage pool includes the hard disk to be replaced; the object storage device configuration information includes at least one of the following: the topology information of the target storage pool, the object storage device mapping information, the placement group mapping rule information, and the placement group distribution information.

[0125] The inspection module 303 is used to check whether there is misaligned data in the distributed storage cluster when the disk swapping operation is detected to be completed.

[0126] The recovery module 304 is configured to, if there is misplacement data, perform a preset recovery operation based on the object storage device configuration information to eliminate the misplacement data, and perform a preset reconstruction operation after the misplacement data is eliminated, the reconstruction operation including writing data stored in the to-be-replaced hard disk to the newly replaced hard disk.

[0127] In some embodiments, the monitoring module 301 is specifically configured to:

[0128] The monitoring module 301 is specifically configured to:

[0129] In some embodiments, the running parameter includes a state parameter and a number of blocked interfaces; and the monitoring module 301 is specifically configured to:

[0130] When the health state field is included in the state parameter and the number of blocked interfaces is less than an alarm number threshold, the distributed storage cluster is determined to be in a healthy state.

[0131] In some embodiments, the obtaining module 302 is further configured to:

[0132] The obtaining module 302 is specifically configured to:

[0133] The obtaining module 302 is specifically configured to:

[0134] The obtaining module 302 is specifically configured to:

[0135] In some embodiments, the recovery module 304 is specifically configured to:

[0136] The recovery module 304 is specifically configured to:

[0137] In some embodiments, the obtaining module 302 is further configured to:

[0138] When detecting that there is misplaced data in the distributed storage cluster, obtain object storage device configuration information of the target storage pool after the disk replacement operation is completed; and backup the object storage device configuration information of the target storage pool after the disk replacement operation is completed.

[0139] In some embodiments, the checking module 303 is configured to:

[0140] detect whether the distributed storage cluster is in a stable state; and when it is determined that the distributed storage cluster is in the stable state, perform a consistency check on the distributed storage cluster to determine whether there is misplaced data in the distributed storage cluster.

[0141] In some embodiments, the recovery module 304 is further configured to:

[0142] If there is no misplaced data in the distributed storage cluster, perform a reconstruction operation.

[0143] In some embodiments, the recovery module 304 is further configured to:

[0144] During the execution of the reconstruction operation, monitor whether there is an alarm message or a blocked interface in the distributed storage cluster; when there is an alarm message or a blocked interface in the distributed storage cluster, pause the execution of the reconstruction operation, and output identification information of the alarm message or the blocked interface; when it is detected that the alarm message is eliminated or the blocked interface is restored to normal, continue to execute the reconstruction operation.

[0145] It can be understood that the above description of features in the distributed storage cluster disk replacement device corresponds to the above description of features in the distributed storage cluster disk replacement method, which will not be repeated here.

[0146] The distributed storage cluster disk replacement device provided by the embodiments of the present application can monitor whether the distributed storage cluster is in a healthy state before performing a disk replacement operation, obtain object storage device configuration information of a target storage pool when the distributed storage cluster is in a healthy state, and perform a preset recovery operation based on the object storage device configuration information when misplaced data exists in the distributed storage cluster after the disk replacement operation is completed, so as to automatically eliminate the misplaced data, thereby reducing the influence of the disk replacement operation on the performance of the distributed storage cluster, and efficiently, stably and safely completing the disk replacement of the distributed storage cluster.

[0147] Figure 4 FIG. 1 is a structural schematic diagram of an electronic device provided in an embodiment of the present application. As shown in FIG. 1, the electronic device includes a processor 100, a memory 200 and a communication interface 300. Figure 4As shown, the electronic device 40 provided by the embodiment includes at least one processor 401 and a memory 402. Optionally, the electronic device 40 further includes a communication component 403. Wherein the processor 401, the memory 402 and the communication component 403 are connected through a bus.

[0148] In the implementation process, the at least one processor 401 executes the computer execution instructions stored in the memory 402, so that the at least one processor 401 executes the above-mentioned distributed storage cluster disk replacement method embodiment.

[0149] The specific implementation process of the processor 401 can refer to the above-mentioned distributed storage cluster disk replacement method embodiment, which has similar implementation principles and technical effects, and will not be described here in detail.

[0150] In the above-mentioned embodiments, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC) and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like. The steps of the distributed storage cluster disk replacement method disclosed in the application can be directly embodied as the execution of the hardware processor, or executed by the combination of hardware and software modules in the processor.

[0151] The memory can contain a random access memory (RAM), and can also include a non-volatile memory (NVM), for example, at least one disk memory.

[0152] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus or an extended industry standard architecture (EISA) bus and the like. The bus can be divided into an address bus, a data bus, a control bus and the like. For the convenience of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0153] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, wherein the computer program is arranged to execute the steps in any of the above distributed storage cluster disk replacement method embodiments when running.

[0154] In an example embodiment, the above computer readable storage medium can include, but is not limited to, a U disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media capable of storing a computer program.

[0155] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in any of the above distributed storage cluster disk replacement method embodiments.

[0156] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0157] The above describes the technical solutions provided by the present application in detail. The principles and implementation manners of the present application are described by using specific examples, and the above example descriptions are only applicable to help understand the technical solutions and core ideas of the present application. It should be noted that, for the ordinary skilled person in the technical field, some improvements and modifications can be made to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A disk-swapping method for a distributed storage cluster, characterized in that, The method includes: In response to a disk swapping command, the system monitors whether the distributed storage cluster is in a healthy state; the disk swapping command includes the name of the target storage pool, and the storage level of the target storage pool is greater than or equal to a preset water level threshold. When the distributed storage cluster is in a healthy state, the object storage device configuration information of the target storage pool is obtained, and a disk replacement operation prompt is output; the target storage pool includes the hard disk to be replaced, and the object storage device configuration information includes at least one of the following: the topology information of the target storage pool, object storage device mapping information, placement group mapping rule information, and placement group distribution information; When the disk swapping operation is detected as complete, check whether there is misaligned data in the distributed storage cluster; If misaligned data exists, a preset recovery operation is performed based on the object storage device configuration information to eliminate the misaligned data; and after eliminating the misaligned data, a preset reconstruction operation is performed, the reconstruction operation including writing the data stored on the hard drive to be replaced into the newly replaced hard drive; The method further includes: Based on the identification information of the hard drives in the target storage pool, the topology information of the target storage pool is generated, and the topology information includes the hierarchical relationship of the hard drives in the target storage pool; Based on the identification information of the object storage devices in the target storage pool, generate object storage device mapping information for the target storage pool; Based on the object storage device mapping information, generate the placement group mapping rule information for the target storage pool; Based on the identification information of the placement groups in the target storage pool, the placement group distribution information of the target storage pool is generated.

2. The method according to claim 1, characterized in that, Monitoring whether the distributed storage cluster is in a healthy state includes: Obtain the list of object storage devices associated with the target storage pool, and check whether the status of the object storage devices in the list is normal; When the object storage devices in the object storage device list are in normal status, obtain the operating parameters of the distributed storage cluster; Based on the aforementioned operating parameters, it is determined whether the distributed storage cluster is in a healthy state.

3. The method according to claim 2, characterized in that, The operating parameters include status parameters and the number of blocked interfaces; determining whether the distributed storage cluster is in a healthy state based on the operating parameters includes: When the status parameters include a health status field and the number of blocked interfaces is less than the alarm number threshold, the distributed storage cluster is determined to be in a healthy state.

4. The method according to claim 1, characterized in that, The process of obtaining the object storage device configuration information of the target storage pool includes: Based on the name of the target storage pool, a directory is created to store the configuration files used in the disk swapping process; Obtain the object storage device configuration information of the target storage pool, and save the obtained object storage device configuration information to the directory according to the current timestamp.

5. The method according to claim 1, characterized in that, The step of performing a preset recovery operation based on the object storage device configuration information includes: Based on the object storage device configuration information, update the configuration parameters of the target storage pool; Based on the updated configuration parameters of the target storage pool, the distribution location of the data stored in the distributed storage cluster is updated.

6. The method according to claim 1 or 5, characterized in that, The method further includes: When misaligned data is detected in the distributed storage cluster, the object storage device configuration information of the target storage pool is obtained after the disk swapping operation is completed. The object storage device configuration information of the target storage pool is backed up after the disk swapping operation is completed.

7. The method according to claim 1, characterized in that, The step of checking whether misplaced data exists in the distributed storage cluster includes: Detect whether the distributed storage cluster is in a stable state; Once the distributed storage cluster is determined to be in a stable state, a consistency check is performed on the distributed storage cluster to determine whether there is misaligned data in the distributed storage cluster.

8. The method according to claim 1, characterized in that, The method further includes: If there is no misaligned data in the distributed storage cluster, then the reconstruction operation is performed.

9. The method according to claim 1 or 8, characterized in that, The method further includes: During the refactoring process, monitor the distributed storage cluster for any alarm messages or blocked interfaces. When the distributed storage cluster has an alarm message or a blocked interface, the reconstruction operation is paused, and the identification information of the alarm message or the blocked interface is output. The refactoring operation continues when the alarm message is detected to be cleared or the blocked interface returns to normal.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the disk swapping method for a distributed storage cluster as described in any one of claims 1 to 9 when executing the computer program.

Images