Method, device and storage medium for processing disk kick in cluster

Abstract

The present application relates to a method, device, and storage medium for processing a disk failure in a cluster, wherein the method comprises: recording error information returned by the cluster corresponding to a target disk in the cluster, wherein the error information includes information for characterizing data read and write errors in the disk and information for characterizing metadata call errors in the disk; setting the target disk as a disk to be failed based on the error information; and performing a failed disk failure process on the disk to be failed when all replicas in the cluster are safe. This application solves the problem of low data processing reliability in the prior art of using peripheral scripts to process failed disks in a cluster.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a method, device, and storage medium for processing a disk kick in a cluster. Background Art

[0002] All data in a cluster is stored on disks. To ensure the security and reliability of cluster data, failed disks must be regularly scanned and removed for repair. Currently, failed disks in a cluster are handled by external scripts that collect kernel error information and then remove the disks. This means that any failed disks in a cluster currently require external scripts to handle, but the data processed by external scripts is unreliable. Furthermore, many failed replicas discovered by processes within the cluster require manual processing, complicating the operations and maintenance system and increasing the risk of data loss. Summary of the Invention

[0003] The present application provides a method, device and storage medium for processing a disk kick in a cluster, which solves the problem in the prior art of low data processing reliability of processing bad disks in a cluster through peripheral scripts.

[0004] In the first aspect, the present application provides a method for kicking a disk in a cluster, comprising: recording error information returned by the cluster corresponding to a target disk in the cluster, wherein the error information includes information for characterizing data read and write errors in the disk and information for characterizing metadata call errors in the disk; setting the target disk as the disk to be kicked based on the error information; and kicking the disk to be kicked when all replicas in the cluster are safe.

[0005] In the second aspect, the present application provides a disk kicking processing device in a cluster, including: a recording module for recording error information returned by the cluster corresponding to the target disk in the cluster, wherein the error information includes information for characterizing data read and write errors in the disk and information for characterizing metadata call errors in the disk; a setting module for setting the target disk as the disk to be kicked based on the error information; a kicking module for kicking the disk to be kicked when all copies in the cluster are safe.

[0006] In a third aspect, an electronic device is provided, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the steps of the disk kick processing method in the cluster described in any embodiment of the first aspect when executing the program stored in the memory.

[0007] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for processing a disk kick in a cluster as described in any embodiment of the first aspect are implemented.

[0008] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0009] The method provided in the embodiment of the present application can record the error information corresponding to the target disk in the cluster, set the target disk as the disk to be kicked, and kick the disk to be kicked when the data in the disk to be kicked is safe. It realizes automatic kicking of the disk under the condition that all copies in the cluster are safe based on the error information returned from the cluster, making data processing safer and more reliable than processing based on peripheral scripts, and solving the problem of low data processing reliability in the prior art of processing bad disks in the cluster through peripheral scripts. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0011] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0012] Figure 1 This is a flow chart of a method for processing a disk kick in a cluster provided in an embodiment of the present application;

[0013] Figure 2 This is a second flow chart of a method for processing a disk kick in a cluster provided in an embodiment of the present application;

[0014] Figure 3 This is one of the structural diagrams of the kick processing device in the cluster provided in an embodiment of the present application;

[0015] Figure 4 This is a second structural diagram of the kick disk processing device in the cluster provided by an embodiment of the present application;

[0016] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0018] First, the relevant terms in the embodiments of the present application are explained;

[0019] Cluster: Consists of multiple DataServers, providing services to the entire cluster.

[0020] DataServer: Data storage node machine, one service is deployed on one machine;

[0021] Block: A large file is a block, which is composed of small files. Each block has a unique BlockId identifier;

[0022] Object: A data block is composed of multiple objects;

[0023] NameServer: the central control node for all DataServers in the cluster;

[0024] Disk: data storage medium;

[0025] Fsck: A module that scans all information in the cluster.

[0026] Figure 1 A flowchart of a method for processing a disk kick in a cluster provided by an embodiment of the present application is shown as follows: Figure 1 As shown, the present application provides a method for processing a disk kick in a cluster, the method comprising:

[0027] Step 102: Record the error information returned by the cluster and corresponding to the target disk in the cluster, wherein the error information includes information indicating data read and write errors in the disk and information indicating metadata call errors in the disk.

[0028] In the present embodiment, disk data read and write errors may refer to input / output errors occurring during pread (atomic read of data from a file with an offset), pwrite (atomically write of data to a file with an offset), read, and write system calls. Furthermore, disk metadata call errors may refer to file system operation errors, such as errors in calls to open, rename, mkdir, and fsync.

[0029] Step 104: setting the target disk as the disk to be kicked based on the error information;

[0030] Step 106: When all replicas in the cluster are safe, the disk to be kicked is kicked.

[0031] In a specific example, steps 102 to 106 may be as follows: by recording data read and write errors and metadata system call errors occurring in the cluster, determining whether the disk can be dropped (unloaded), regularly scanning the status of all disks on the DataServer in the cluster, automatically kicking the disks that can be dropped, and sending the disk removal command to the NameServer. The NameServer then processes the disk accordingly. Only when the cluster Fsck information, i.e., the data is complete and safe, will it be sent to the DataServer. The disk is then set to unavailable, thereby achieving automatic disk kicking without relying on external scripts.

[0032] That is to say, in an embodiment of the present application, the target disk can be set as a disk to be kicked by recording the error information corresponding to the target disk in the cluster, and the disk to be kicked can be kicked when all copies in the cluster are safe. This realizes automatic kicking of the disk under the condition of satisfying data security based on the error information returned from the cluster, making data processing safer and more reliable than processing based on peripheral scripts, and solving the problem of low data processing reliability in the prior art of using peripheral scripts to process bad disks in the cluster.

[0033] In an optional implementation of the embodiment of the present application, the method of setting the target disk as a disk to be kicked based on the error information involved in step 104 includes at least one of the following methods:

[0034] Mode 1) When the error information indicates that the number of bad blocks with read and write errors in the target disk exceeds a first preset threshold, the target disk is set as a disk to be kicked;

[0035] Among them, the first preset threshold can be set accordingly as needed, and its value is not limited in the embodiment of the present application.

[0036] Method 2) When the error information indicates that a metadata call error occurs in the target disk, the target disk is set as the disk to be kicked.

[0037] Through the above-mentioned methods 1) and 2), it can be known that in the embodiment of the present application, whether to set the target disk as the disk to be kicked can be determined by the number of bad blocks and metadata call errors. In the case where the number of bad blocks exceeds the first preset threshold, setting the target disk as the disk to be kicked can avoid setting the disk as the disk to be kicked as soon as a bad block appears. The disk can be set as the disk to be kicked after the number of bad blocks accumulates to a certain level. The metadata call error is a relatively serious error. Therefore, in the case of a metadata call error, the target disk is set as the disk to be kicked. That is, in the embodiment of the present application, the appropriate time to set the disk to be kicked is given for different error situations, avoiding misoperation while taking into account the situation where the disk needs to be repaired.

[0038] In another optional implementation of the embodiment of the present application, the method of performing the kicking process on the disk to be kicked involved in the above step 106 may further include:

[0039] Step 11, determining the time of the last disk kick processing operation;

[0040] Step 12: After a preset time interval from the last disk kicking operation, the disk to be kicked is set to be unavailable.

[0041] It can be seen from the above steps 11 and 12 that there is a preset time interval between this kicking disk processing operation and the previous kicking disk processing operation. That is to say, in the embodiment of the present application, the speed of kicking the disk is limited, that is, the disk is not set to be unavailable immediately after being set as the disk to be kicked; by limiting the speed of kicking the disk, it can be further ensured that the cluster has a certain recovery capability.

[0042] In an optional implementation manner of the embodiment of the present application, after the target disk is set as the disk to be kicked based on the error information, as shown in FIG. Figure 2 As shown, the method steps of the embodiment of the present application may also include:

[0043] Step 202: Obtain disk information corresponding to the disk in the cluster; wherein the disk information is used to indicate the type of the disk corresponding to the disk information and the amount of replica data stored on the disk;

[0044] Step 204: Summarize the disk information of all disks in the cluster to obtain the data volume corresponding to each disk type;

[0045] Step 206 : Determine whether all replicas in the cluster are safe based on the replica data volume corresponding to each disk type and the preset storage data volume in the cluster.

[0046] From the above steps 202 to 206, it can be seen that before kicking the disk, it is necessary to determine whether all replicas in the cluster are safe, that is, the kicking process must be performed in a safe state to ensure data security.

[0047] Furthermore, the method of obtaining disk information corresponding to the disk in the cluster involved in the above step 202 may further include:

[0048] Step 21: determining a correspondence between each sub-thread in the plurality of sub-threads and a disk in the cluster, wherein the cluster includes a plurality of disks and each sub-thread corresponds to at least one disk;

[0049] Step 22: Obtain disk information corresponding to each disk by concurrently executing multiple sub-threads according to the corresponding relationship.

[0050] After determining the correspondence between the child threads and the disks through steps 21 and 22, all child threads can be executed concurrently according to the correspondence, and each child thread processes the disk corresponding to it to obtain disk information. Since the disk information can be information that can indicate the type and data volume of the disk corresponding to the disk information, in a specific example, the disk type can be three copies or an erasure copy, and the data volume can be 1MB or 2MB, etc.

[0051] For example, when there are sub-threads I, II, and III, and sub-thread I corresponds to disks 0-9999, sub-thread II corresponds to disks 10000-19999, and sub-thread III corresponds to disks 20000-29999, the following operations are performed simultaneously: sub-thread I processes disks 0-9999 in sequence to obtain disk information of each disk in disks 0-9999; sub-thread II processes disks 10000-19999 to obtain disk information of each disk in disks 10000-19999; sub-thread III processes disks 20000-29999 to obtain disk information of each disk in disks 20000-29999.

[0052] Furthermore, the method involved in step 206 above of determining whether all replicas in the cluster are safe based on the replica data volume corresponding to each disk type and the preset storage data volume in the cluster may further include:

[0053] Step 31: When the amount of replica data corresponding to each disk type is three times the preset storage data amount, determine that all replicas in the cluster are safe;

[0054] Step 32: When the amount of replica data corresponding to each disk type is less than three times the preset storage data amount, it is determined that there is a replica with damaged data in the cluster.

[0055] As can be seen from steps 31 and 32 above, after obtaining the data volume corresponding to each disk type, judgment information can be obtained based on the preset storage data volume in the cluster. The preset storage data volume can be the amount of original data to be stored. In other words, after obtaining the data volume corresponding to each disk type and the preset storage data volume in the cluster, it is possible to determine whether all replicas in the cluster are safe.

[0056] For example, when three copies are used to store the original data, the sum of the data volumes corresponding to the three copy types (i.e., the total amount of sub-data of the three copy types) should be three times the preset storage data volume. Therefore, when the total amount of sub-data of the three copy types is three times the preset storage data volume, it indicates that all copies in the cluster are safe and reliable, and there is no missing data copy; when the total amount of sub-data of the three copy types is less than three times the preset storage data volume, it indicates that there is missing or damaged data copy in the cluster.

[0057] In another optional implementation of the embodiment of the present application, the method of the embodiment of the present application may further include:

[0058] Step 108, determining the replica with data corruption in the target disk, and adding the replica to the queue;

[0059] Step 110 : Deleting the copies in the queue in sequence according to the order in which the copies were added to the queue, wherein the number of copies in the queue is less than a second preset threshold.

[0060] It should be noted that data corruption can occur in various system processes, such as data read, write, and delete, erasure coding (EC), compaction (garbage collection), recovery, and fsck (cluster health check). This means that data corruption can occur due to hardware errors, such as disk hardware errors or silent memory errors. In other words, data written to a replica can be inconsistent when read. Furthermore, automatically deleting replicas in the queue in the order they were added avoids deleting multiple replicas simultaneously. This eliminates the need for manual intervention and allows for rapid detection and resolution of corrupted replicas, ensuring faster and more secure cluster data processing.

[0061] Corresponding to the above Figure 1The embodiment of the present application further provides a disk kick processing device in a cluster, such as Figure 3 As shown, the device includes:

[0062] A recording module 32 is configured to record error information returned by the cluster and corresponding to a target disk in the cluster, wherein the error information includes information indicating data read and write errors in the disk and information indicating metadata call errors in the disk;

[0063] A setting module 34, configured to set a target disk as a disk to be kicked based on the error information;

[0064] The kicking module 36 is used to kick the disk to be kicked when all replicas in the cluster are safe.

[0065] Through the device of the embodiment of the present application, the target disk can be set as the disk to be kicked by recording the error information corresponding to the target disk in the cluster, and the disk to be kicked can be kicked when the data in the disk to be kicked is safe. This realizes automatic kicking of the disk under the condition that all copies in the cluster are safe based on the error information returned from the cluster, making data processing safer and more reliable than processing based on peripheral scripts, and solving the problem of low data processing reliability in the prior art of processing bad disks in the cluster through peripheral scripts.

[0066] Optionally, in an embodiment of the present application, the setting module may further include at least one of the following: a first setting unit, used to set the target disk as the disk to be kicked when the error information indicates that the number of bad blocks with read and write errors in the target disk exceeds a first preset threshold; a second setting unit, used to set the target disk as the disk to be kicked when the error information indicates that a metadata call error occurs in the target disk.

[0067] Through the above-mentioned first setting unit and second setting unit, in the embodiment of the present application, whether to set the target disk as the disk to be kicked can be determined by the number of bad blocks and metadata call errors. In the case where the number of bad blocks exceeds the first preset threshold, setting the target disk as the disk to be kicked can avoid setting the disk as the disk to be kicked as soon as a bad block appears. The disk can be set as the disk to be kicked after the number of bad blocks accumulates to a certain level. Metadata call errors are relatively serious errors. Therefore, in the case of metadata call errors, the target disk is set as the disk to be kicked. That is, in the embodiment of the present application, appropriate times to set the disk to be kicked are given for different error situations, avoiding misoperation while taking into account the situation where the disk needs to be repaired.

[0068] Optionally, the kick disk module 36 in the embodiment of the present application may further include: a determination unit for determining the time of the last kick disk processing operation; a third setting unit for setting the disk to be kicked as unavailable after a preset time interval from the time of the last kick disk processing operation.

[0069] It can be seen that in the embodiment of the present application, there is a preset time interval between the current kicking disk processing operation and the previous kicking disk processing operation. That is to say, in the embodiment of the present application, the speed of kicking the disk is limited, that is, the disk is not set to be unavailable immediately after being set as the disk to be kicked; by limiting the speed of kicking the disk, it can be further ensured that the cluster has a certain recovery ability.

[0070] exist Figure 3 On the basis of Figure 4 The device of the embodiment of the present application further includes:

[0071] An acquisition module 42 is configured to acquire disk information corresponding to a disk in the cluster; wherein the disk information is used to indicate the type of disk corresponding to the disk information and the amount of replica data stored on the disk;

[0072] A first processing module 44 is configured to aggregate disk information of all disks in the cluster to obtain a data volume corresponding to each disk type;

[0073] The determination module 46 is configured to determine whether all replicas in the cluster are safe based on the amount of replica data corresponding to each disk type and the preset amount of storage data in the cluster.

[0074] Optionally, the acquisition module 42 in the embodiment of the present application further includes: a first determination unit, used to determine the correspondence between each sub-thread of multiple sub-threads and the disk in the cluster, wherein the cluster includes multiple disks and each sub-thread corresponds to at least one disk; an acquisition unit, used to obtain disk information corresponding to each disk by concurrently executing multiple sub-threads according to the correspondence.

[0075] Optionally, the determination module 46 in the embodiment of the present application may further include: a second determination unit, used to determine that all replicas in the cluster are safe when the amount of replica data corresponding to each disk type is three times the preset storage data amount; a third determination unit, used to determine that there are replicas with damaged data in the cluster when the amount of replica data corresponding to each disk type is less than three times the preset storage data amount.

[0076] The device in the embodiment of the present application may also include: a second processing module, which is used to add the copy with damaged data to the queue when it is determined that there is a copy with damaged data in the target disk; a deletion module, which is used to delete the copies in the queue in sequence according to the order in which the copies are added to the queue, wherein the number of copies in the queue is less than a second preset threshold.

[0077] It should be noted that data corruption can occur in various system processes, such as data read, write, and delete, erasure coding (EC), compaction (garbage collection), recovery, and fsck (cluster health check). This means that data corruption can occur due to hardware errors, such as disk hardware errors or silent memory errors. In other words, data written to a replica can be inconsistent when read. Furthermore, automatically deleting replicas in the queue in the order they were added avoids deleting multiple replicas simultaneously. This eliminates the need for manual intervention and allows for rapid detection and resolution of corrupted replicas, ensuring faster and more secure cluster data processing.

[0078] like Figure 5 As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0079] Memory 113, for storing computer programs;

[0080] In one embodiment of the present application, the processor 111 is used to execute the program stored in the memory 113 to implement the disk kick processing method in the cluster provided by any of the aforementioned method embodiments, and its function is similar and will not be repeated here.

[0081] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method for processing a disk kick in a cluster provided in any one of the aforementioned method embodiments are implemented.

[0082] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0083] The foregoing description is intended only to provide specific embodiments of the present invention, which will enable those skilled in the art to understand and implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown herein, but is intended to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for processing a disk kick in a cluster, characterized in that: include: Recording error information returned by the cluster and corresponding to the target disk in the cluster, wherein the error information includes information used to characterize data read and write errors in the disk and information used to characterize metadata call errors in the disk; Setting the target disk as a disk to be kicked based on the error information; When all replicas in the cluster are safe, kicking the disk to be kicked; Among them, after setting the target disk as the disk to be kicked based on the error information, the method also includes: obtaining disk information corresponding to the disk in the cluster; wherein, the disk information is used to indicate the type of the disk corresponding to the disk information and the amount of replica data stored on the disk; summing up the disk information of all disks in the cluster to obtain the amount of data corresponding to each disk type; and determining whether all replicas in the cluster are safe based on the amount of replica data corresponding to each disk type and the preset storage data amount in the cluster.

2. The method according to claim 1, characterized in that The step of setting the target disk as a disk to be kicked based on the error information includes at least one of the following: When the error information indicates that the number of bad blocks with read and write errors in the target disk exceeds a first preset threshold, setting the target disk as the disk to be kicked; In a case where the error information indicates that a metadata call error occurs in the target disk, the target disk is set as the disk to be kicked.

3. The method according to claim 1, characterized in that The step of performing a kicking process on the disk to be kicked includes: Determine the time of the last kick operation; After a preset time interval from the last disk kicking operation, the disk to be kicked is set to be unavailable.

4. The method according to claim 1, wherein The obtaining of disk information corresponding to the disk in the cluster includes: Determine a correspondence between each of the plurality of sub-threads and a disk in the cluster, wherein the cluster includes a plurality of the disks and each of the sub-threads corresponds to at least one of the disks; According to the corresponding relationship, the disk information corresponding to each of the disks is obtained by concurrently executing the multiple sub-threads.

5. The method according to claim 1, wherein The determining whether all replicas in the cluster are safe based on the replica data volume corresponding to each disk type and the preset storage data volume in the cluster includes: When the amount of replica data corresponding to each disk type is three times the preset storage data amount, determining that all replicas in the cluster are safe; When the replica data volume corresponding to each disk type is less than three times the preset storage data volume, it is determined that a replica with damaged data exists in the cluster.

6. The method according to claim 5, characterized in that The method further comprises: When determining that a copy with damaged data exists in the target disk, adding the copy with damaged data to a queue; The copies in the queue are deleted in sequence according to the order in which the copies are added to the queue, wherein the number of copies in the queue is less than a second preset threshold.

7. A disk kick processing device in a cluster, characterized in that: include: a recording module, configured to record error information returned by the cluster and corresponding to a target disk in the cluster, wherein the error information includes information indicating data read and write errors in the disk and information indicating metadata call errors in the disk; A setting module, configured to set the target disk as a disk to be kicked based on the error information; A disk kicking module is used to kick the disk to be kicked when all replicas in the cluster are safe; The device further includes: an acquisition module for acquiring disk information corresponding to a disk in the cluster; wherein the disk information is used to indicate the type of the disk corresponding to the disk information and the amount of replica data stored on the disk; The first processing module is used to aggregate disk information of all disks in the cluster to obtain the data volume corresponding to each disk type; The determination module is used to determine whether all replicas in the cluster are safe based on the replica data volume corresponding to each disk type and the preset storage data volume in the cluster.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 6 when executing a program stored in a memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images