Fault processing method and computing device

Abstract

The invention discloses a fault processing method and computing equipment, and relates to the technical field of computers. The method is applied to the distributed storage system, the distributed storage system comprises at least one storage node, the storage node comprises at least one disk, and the disk comprises one or more magnetic arms; the method comprises the following steps: acquiring fault information; based on the fault information, fault types are determined, and the fault types comprise a disk fault and a magnetic arm fault; and based on the fault type, reconstructing at least one fault logic unit in the fault disk. According to the method, the fault type of the fault disk can be effectively distinguished, that is, whether the whole disk fault or the partial magnetic arm fault of the multi-magnetic-arm disk can be distinguished, so that the range of data reconstruction can be determined, the conditions of data loss and even EC degradation are avoided, and the reliability of a distributed storage system is guaranteed.

Description

Technical Field

[0001] This application relates to the field of computer technologies, and in particular, to a fault handling method and a computing device. Background Art

[0002] In a distributed storage system, in order to improve the reliability of the storage system, the erasure code (EC) algorithm / technology is usually adopted to ensure the reliability of disk data and node data.

[0003] The erasure code is a coding technology with the characteristics of high fault tolerance and high storage efficiency, that is, a data sharding redundancy protection mechanism. In a distributed storage system adopting the erasure code, data will be stored in the form of an EC check group. That is, when writing data, a piece of data is sliced into multiple data shards, and multiple check shards are generated based on the multiple data shards (the multiple data shards and multiple check shards corresponding to a piece of data together constitute an EC check group), and the multiple data shards and multiple check shards are distributed and stored in the logical units of the storage nodes. When a shard (data shard or check shard) is lost due to a logical unit failure, the lost shard can be reconstructed to ensure the reliability of the data.

[0004] In the current technology, the common method for handling disk failures is as follows: when there is a failure in a logical unit in a distributed storage system, the data of the faulty logical unit is reconstructed, and the faulty disk corresponding to the faulty logical unit is replaced. For a single-actuator disk, it presents a logical unit to the upper-layer operating system. When the logical unit fails, the corresponding single-actuator disk fails. Therefore, after reconstructing the data of the faulty logical unit, only the single-actuator disk needs to be replaced. However, for a multi-actuator disk, it presents at least two logical units to the upper-layer operating system. When some logical units in the multi-actuator disk fail, after reconstructing the data of the faulty logical unit and replacing the multi-actuator disk, the data of the remaining logical units of the multi-actuator disk will be lost.

[0005] It can be seen from this that the method for handling disk failures in the current technology cannot guarantee the reliability of the data in the distributed storage system, and may cause data loss and even EC degradation. Summary of the Invention

[0006] A fault handling method and a computing device provided by this application are applicable to a distributed storage system including a multi-actuator disk, and can guarantee the reliability of the data in the distributed storage system, and avoid data loss and even EC degradation.

[0007] To achieve the above object, this application adopts the following technical solutions:

[0008] In a first aspect, an embodiment of the present application provides a fault handling method, which is applied to a distributed storage system. The distributed storage system includes at least one storage node, the storage node includes at least one disk, and the disk includes one or more magnetic arms. The method includes: obtaining fault information; based on the fault information, determining a fault type, where the fault type includes disk fault and magnetic arm fault; based on the fault type, reconstructing at least one faulty logical unit in the faulty disk. The embodiment of the present application can effectively distinguish the fault type of the faulty disk, that is, it can distinguish whether the multi-magnetic-arm disk has a whole disk fault or a partial magnetic arm fault, so as to determine the scope of data reconstruction, avoid data loss and even EC degradation, thereby ensuring the reliability of the distributed storage system.

[0009] In a possible implementation manner, when the fault type is a disk fault, reconstruct the data stored in all logical units in the faulty disk.

[0010] In a possible implementation manner, when the fault type is a disk fault, after reconstructing the data stored in all logical units in the faulty disk, set the disk replacement flag corresponding to the faulty disk to a first disk replacement identifier, where the first disk replacement identifier is used to indicate the replacement of the faulty disk; in response to the disk replacement flag corresponding to the faulty disk being the first disk replacement identifier, perform a disk replacement operation on the faulty disk. The distributed storage system automatically performs a disk replacement operation on the faulty disk in response to the disk replacement flag corresponding to the faulty disk being the first disk replacement identifier.

[0011] In a possible implementation manner, when the fault type is a magnetic arm fault, reconstruct the data stored in the faulty logical units in the faulty disk.

[0012] In a possible implementation manner, when the fault type is a magnetic arm fault, after reconstructing the data stored in the faulty logical units in the faulty disk, set the disk replacement flag corresponding to the faulty disk to a second disk replacement identifier, where the second disk replacement identifier is used to indicate not to replace the faulty disk; and / or set the health status corresponding to the faulty disk to a sub-healthy status, where the health status is used to indicate the current status of the disk. Set the disk replacement flag corresponding to the faulty disk to the second disk replacement identifier, thereby indicating not to replace the faulty disk, avoiding data loss and even EC degradation; set the health status corresponding to the faulty disk to the sub-healthy status, which is used to indicate the current status of the faulty disk, that is, it means that there are still some non-fatal problems with the disk currently, but no disk replacement is required.

[0013] In a possible implementation, after reconstructing at least one faulty logical unit in the faulty disk based on the fault type, when the unusable storage capacity in the distributed storage system is greater than the first preset storage capacity, the distributed storage system is expanded. When the unusable storage capacity in the distributed storage system is greater than the first preset storage capacity, it indicates that the usable storage capacity in the distributed storage system is too small, resulting in a significant decline in the performance of the distributed storage system and even a performance bottleneck, unable to meet the user's requirements for the distributed storage system. Therefore, the distributed storage system is expanded to meet the user's requirements for the distributed storage system (such as performance requirements and storage capacity requirements).

[0014] In a possible implementation, based on the fault information, it is determined whether the proportion of the faulty logical units in the faulty disk among all the logical units in the faulty disk is not less than the preset proportion; if so, the fault type is disk fault; if not, the fault type is arm fault. When the proportion of the faulty logical units in the faulty disk among all the corresponding logical units is not less than the preset proportion, it indicates that the proportion of the faulty arms of the disk among all the arms is not less than the preset proportion. Therefore, the fault type of the disk is disk fault; otherwise, the fault type is arm fault.

[0015] In a possible implementation, the identifiers of the faulty logical units included in the fault information are traversed; based on the identifiers of the faulty logical units and the stored annotation information, the corresponding faulty disk is determined; based on the identifier of the faulty disk and the annotation information, all the logical units belonging to the faulty logical units are determined; based on the fault information, it is determined whether the proportion of the faulty logical units in the faulty disk among all the logical units in the faulty disk is not less than the preset proportion. Through the stored annotation information, the faulty disk and all the logical units in the faulty disk can be quickly located, so as to accurately determine whether the proportion of the faulty logical units in the faulty disk among all the logical units in the faulty disk is not less than the preset proportion.

[0016] In a possible implementation, the distributed storage system performs a rebalancing operation on its own storage capacity to ensure that the storage capacity usage of each storage node and each disk is balanced, that is, to ensure that the usage rates of all storage nodes or disks are similar, and to avoid overloading of some storage nodes or disks. Thereby optimizing performance, improving resource utilization, and enhancing the stability and reliability of the distributed storage system.

[0017] Second aspect, an embodiment of the present application provides a fault handling device, which is applied to a distributed storage system. The distributed storage system includes at least one storage node, the storage node includes at least one disk, and the disk includes one or more magnetic arms. The fault handling device includes: an information acquisition module, configured to acquire fault information; a type determination module, configured to determine a fault type based on the fault information, where the fault type includes disk faults and magnetic arm faults; and a data reconstruction module, configured to reconstruct at least one faulty logical unit in the faulty disk based on the fault type. The embodiment of the present application can effectively distinguish the fault types of faulty disks, that is, it can distinguish whether the multi-magnetic-arm disk has a whole disk fault or a partial magnetic arm fault, so as to determine the scope of data reconstruction, avoid data loss and even EC degradation, thereby ensuring the reliability of the distributed storage system.

[0018] In a possible implementation, the data reconstruction module includes: a faulty disk reconstruction module and a faulty magnetic arm reconstruction module. The faulty disk reconstruction module is configured to reconstruct the data stored in all logical units in the faulty disk based on the fault type being a disk fault; the faulty disk reconstruction module is configured to reconstruct the data stored in the faulty logical units in the faulty disk based on the fault type being a disk fault.

[0019] Third aspect, an embodiment of the present application provides a distributed storage system, which includes at least one storage node, the storage node includes at least one disk, and the disk includes one or more magnetic arms; the distributed storage system is used to implement the fault handling method in the first aspect and its various possible implementations.

[0020] Fourth aspect, an embodiment of the present application provides a computing device, including: a processor and a memory; a storage system software runs on the processor, and the storage system software is used to manage and maintain the distributed storage system; the memory is used to store computer program instructions; the processor is used to execute the computer program instructions stored in the memory to implement the fault handling method in the first aspect and its various possible implementations.

[0021] Fifth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions run on a computing device, the computing device is enabled to implement the fault handling method in the first aspect and its various possible implementations.

[0022] Sixth aspect, the present application provides a computer program product, which includes computer program instructions. When the computer program instructions run on a computing device, the computing device is enabled to implement the fault handling method in the first aspect and its various possible implementations. Description of the Drawings

[0023] Figure 1 It is an example diagram of the data storage scenario of a distributed storage system provided by an embodiment of the present application;

[0024] Figure 2 It is another example diagram of the data storage scenario of a distributed storage system provided by an embodiment of the present application;

[0025] Figure 3 It is a schematic diagram of the composition of a data storage system architecture provided by an embodiment of the present application;

[0026] Figure 4A It is a schematic flowchart of a fault handling method provided by an embodiment of the present application;

[0027] Figure 4B It is another schematic flowchart of a fault handling method provided by an embodiment of the present application;

[0028] Figure 5 It is a schematic diagram of the structure of a distributed storage system provided by an embodiment of the present application;

[0029] Figure 6 It is a schematic flowchart of a fault handling method corresponding to a disk fault provided by an embodiment of the present application;

[0030] Figure 7 It is a schematic flowchart of a fault handling method corresponding to a magnetic arm fault provided by an embodiment of the present application;

[0031] Figure 8 It is an application example diagram of a fault handling method provided by an embodiment of the present application;

[0032] Figure 9 It is another schematic flowchart of a fault handling method provided by an embodiment of the present application;

[0033] Figure 10 It is another schematic flowchart of a fault handling method provided by an embodiment of the present application;

[0034] Figure 11 It is another schematic flowchart of a fault handling method provided by an embodiment of the present application;

[0035] Figure 12 It is another application example diagram of a fault handling method provided by an embodiment of the present application;

[0036] Figure 13 It is another schematic flowchart of a fault handling method provided by an embodiment of the present application. Detailed implementation manners

[0037] The terms "first", "second", "third", etc. in the description, claims and drawings of this application are used to distinguish different objects, rather than to limit a specific order.

[0038] In the embodiments of this application, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0039] For the sake of clear and concise description of the following embodiments, a brief introduction to the related technologies is given first:

[0040] With the rapid development of technologies such as High-Performance Computing (HPC), big data, and Artificial Intelligence (AI), massive amounts of data are impacting various industries with an unprecedented growth trend. In order to effectively manage and maintain this data, users have an increasing demand for data storage capacity and an increasingly high requirement for data processing efficiency. In this context, it has spurred the development of disk technology towards large capacity and high performance. Among them, multi-actuator disks can balance disk capacity and performance, and the data read / write efficiency is multiple times that of traditional single-actuator disks. Therefore, multi-actuator disks are increasingly widely used in distributed storage systems.

[0041] The actuator arm is a mechanical component in a hard disk drive (HDD) that connects the read / write head to the drive motor and is responsible for accurately positioning the read / write head onto the track on the disk surface (the platter surface).

[0042] A single-actuator disk refers to a hard disk drive (HDD) that uses a single actuator arm (or a single actuator) to read and write data, that is, a single actuator arm serves all the tracks on all the platter surfaces. Specifically, a single-actuator disk internally has one actuator arm, one or more platters, and one or more heads (read / write heads). Among them, both sides of a platter can store data. Correspondingly, there can be one head on each of its upper and lower surfaces for reading and writing, and a platter can be divided into multiple sectors. By swinging a single actuator arm, the head can be positioned to the target sector on the platter (i.e., the sector where data needs to be written or read), and then data can be read from or written to the target sector through the head.

[0043] A multi-actuator disk is a hardware drive that uses multiple actuators (or multiple arms) to read and write data. That is, in a multi-actuator disk, multiple arms respectively serve all the tracks on multiple disk surfaces. Specifically, a multi-actuator disk internally has multiple arms, multiple disks, and multiple heads. The disks inside a multi-actuator disk can be divided into multiple groups, and each arm can be responsible for reading and writing data on a group of disks.

[0044] As can be seen from the above, since a multi-actuator disk has multiple arms performing data reading and writing concurrently, while a single-actuator disk has only one arm for data reading and writing, the reading and writing efficiency of a multi-actuator disk is much higher than that of a single-actuator disk.

[0045] Furthermore, it can be understood that since a single-actuator disk has only one arm for reading and writing data, a single-actuator disk generally presents a single logical unit to the upper-layer operating system (OS). A logical unit can also be referred to as a logical volume / logical disk (LUN / Disk), that is, for a single-actuator disk: logical unit (LUN / Disk) = 1:1. Since a multi-actuator disk has multiple arms (e.g., 2 arms) for reading and writing data, and multiple arms can work simultaneously, a multi-actuator disk generally presents multiple logical units (LUN / Disk) to the upper-layer operating system. That is, for a multi-actuator disk: logical unit (LUN / Disk) = 1:E (E is an integer greater than 1). It should be understood that the number of logical units presented by a multi-actuator disk to the upper-layer operating system is equal to the number of its arms. For example, a multi-actuator disk with 2 arms presents 2 logical units to the upper-layer operating system, that is, E = 2; a multi-actuator disk with 3 arms presents 3 logical units to the upper-layer operating system, that is, E = 3.

[0046] A distributed storage system is a system that distributes data across multiple storage nodes in a computer network. In a distributed storage system, storage nodes are usually built based on general-purpose servers, and in order to improve the reliability of the distributed storage system, erasure code algorithms / techniques are generally used to ensure the reliability of disk data and node data.

[0047] Erasure Code (EC) is a coding technology with the characteristics of high fault tolerance and high storage efficiency, that is, a data sharding redundancy protection mechanism, which realizes data redundancy protection by calculating check shards. Specifically, when a distributed storage system writes data, the data is usually split into N data shards, and M check shards are calculated based on the N data shards through the EC algorithm. Subsequently, the N data shards and the M check shards are distributed and stored in logical units. In the case where a shard (data shard or check shard) is lost due to a logical unit failure, the original data (i.e., the lost shard) can be restored through any N of the N+M shards. Therefore, if the number of lost shards in the distributed storage system is less than or equal to M, the lost shards can be restored based on the non-lost shards. Here, N is an integer greater than 0, and M is an integer greater than 0.

[0048] In practical applications, users can configure different EC redundancy ratios according to the number of storage nodes. When the number of storage nodes ≥ (N+M), an N+M redundancy configuration is adopted. When adopting the N+M redundancy ratio, the N+M shards (i.e., N data shards and M check shards) are respectively stored in the logical units on N different storage nodes. Therefore, the N+M redundancy ratio can allow M logical unit failures or M storage node failures without data loss. When ((N+M) / M) ≤ the number of storage nodes < (N+M), redundancy ratios such as N+M:1 or N+M:2 can be adopted. When adopting redundancy ratios such as N+M:1 or N+M:2, the N+M shards can be dispersed and stored on all storage nodes, and each storage node stores at least one shard. It should be understood that the N+M:1 redundancy ratio can allow M logical unit failures or 1 storage node failure without data loss, and the N+M:2 redundancy ratio can allow M logical unit failures or 2 storage node failures without data loss. Similarly, the N+M:F redundancy ratio can allow M logical unit failures or F storage node failures without data loss. F is an integer greater than 0.

[0049] Furthermore, compared with replication technology and others, the EC technology has a higher capacity utilization rate. For example, when EC adopts a redundancy of N+2:1, when n takes the value of 1, its capacity utilization rate is approximately 66%, while when using double replication (i.e., each piece of data is stored twice), its capacity utilization rate is 50%.

[0050] For the convenience of understanding, the following combines Figure 1 and Figure 2 to introduce, by way of example, the data storage scenario of a distributed storage system.

[0051] The following combines Figure 1, taking N = 4, M = 2, and the number of storage nodes being 6 as an example, the scenario where the number of storage nodes in the distributed storage system is greater than or equal to (N + M) is described.

[0052] As Figure 1 shown in (a) of , each storage node includes 4 logical units. When writing data, the data is sliced into 4 data shards, and 2 parity shards are obtained based on the 4 data shards through the EC algorithm. Since the number of shards is equal to the number of storage nodes, the N + M (4 + 2) redundancy ratio is adopted, that is, one shard can be stored in each storage node, that is, one logical unit in each storage node is selected to store the shard. For example: as Figure 1 shown, the 4 data shards are respectively stored in storage nodes 2, 3, 4, and 6, and the 2 parity shards are respectively stored in storage nodes 1 and 5. This can ensure that when 2 single-arm disks or 2 storage nodes fail simultaneously in the distributed storage system, the data is not lost.

[0053] Furthermore, when the logical unit 101 storing the data shard in storage node 2 in (a) of Figure 1 fails, resulting in the loss of the stored data shard. The current technology's handling method for disk failures is as Figure 1 shown in (b) of . According to the reconstruction strategy (generally divided into delayed reconstruction and immediate reconstruction), the data shard stored in the logical unit 101 in storage node 2 is reconstructed through the remaining shards and stored in other logical units in storage node 2; the disk corresponding to the failed logical unit is replaced, so as to ensure the availability and data reliability of the distributed storage system.

[0054] Next, in combination with Figure 2 , taking N = 4, M = 2, and the number of storage nodes being 3 as an example, the scenario where the number of storage nodes in the distributed storage system is greater than or equal to ((N + M) / M) and less than (N + M) is described.

[0055] As Figure 2As shown in (a) therein, each storage node includes 4 logical units. When writing data, the data is sliced into 4 data shards, and 2 parity shards are obtained through the EC algorithm based on the 4 data shards. Since the number of shards is greater than the number of storage nodes, and the number of storage nodes is greater than (N + M) / M = 3, an N + M:1 (4 + 2:1) redundancy ratio is adopted. At this time, two logical units can be selected in each storage node to store shards, that is, each storage node stores two shards. For example: one of the 4 data shards is stored in a logical unit in storage node 1, two data shards are respectively stored in two logical units in storage node 2, and one data shard is stored in a logical unit in storage node 3; the two parity shards are respectively stored in a logical unit in storage node 1 and storage node 3. This can ensure that data is not lost when 2 logical units or 1 storage node fail simultaneously in the distributed storage system.

[0056] Further, when, as Figure 2 shown in (a) therein, the logical unit 201 storing the data shard in storage node 1 fails, resulting in the loss of the stored data shard. The processing method for disk failure in the current technology is as Figure 2 shown in (b) therein. According to the reconstruction strategy, the data shard stored in the failed logical unit 201 in storage node 1 is reconstructed through the remaining shards and stored in other logical units in storage node 1; the disk corresponding to the failed logical unit is replaced, so as to ensure the availability and data reliability of the distributed storage system.

[0057] In the related technology, the processing method for disk failure is to reconstruct the data stored in the failed logical unit and replace the disk corresponding to the failed logical unit. However, when only some of the logical units corresponding to the multi-arm disk fail, in the current technology, the data stored in the failed logical unit is reconstructed, while the data in the non-failed logical units corresponding to the multi-arm disk is not reconstructed. At this time, replacing the multi-arm disk will cause the data in the remaining non-failed logical units to be lost; further, the remaining non-failed logical units may store data shards or parity shards, so replacing the multi-arm disk will cause the data shards or parity shards in the EC to be lost, resulting in Erasure Coding Degradation.

[0058] An embodiment of the present application provides a fault handling method, including: obtaining fault information, where the fault information includes the identifier of a fault logic unit; determining a fault type based on the fault information, where the fault type includes a disk fault and an arm fault; and reconstructing at least one fault logic unit in the faulty disk based on the fault type. The embodiment of the present application can effectively distinguish the fault types of faulty disks, that is, it can distinguish whether a multi-arm disk has a whole-disk fault or a partial-arm fault, so as to determine the scope of data reconstruction, avoid data loss and even EC degradation, and thus ensure the reliability of the distributed storage system.

[0059] Embodiment 1:

[0060] The following Figures 3 - 7 is used to introduce in detail a fault handling method provided by an embodiment of the present application.

[0061] First, in combination with Figure 3 the system architecture to which a fault handling method provided by an embodiment of the present application is applied is described.

[0062] As Figure 3 shown, the data storage system architecture includes: a computing device 310, a network device (Network Equipment) 320, and a distributed storage system 330.

[0063] Among them, the computing device 310 can also be referred to as a client device. The computing device 310 can be a tablet computer, a laptop computer, a desktop server, etc. The computing device 310 includes a client 311. That is, when a storage client software is installed on the computing device 310, the computing device 310 (its client 311) is responsible for reading and writing data. For example: when writing data, the client 311 included in the computing device 310 can split the written data into N data shards, and then obtain M parity shards through the EC algorithm based on the N data shards. Subsequently, the client 311 on the computing device 310 stores the N data shards and the M parity shards into the distributed storage system 330 through the network device 320 respectively.

[0064] Specifically, when the client 311 on the computing device 310 needs to write data, the client 311 included in the computing device 310 can obtain the initialized disk information, that is, obtain the disk type and the logical unit corresponding to each disk; select candidate logical units according to the disk type; at the same time, the client 311 divides the data into N data shards, and then obtains M parity shards through the EC algorithm based on the N data shards, that is, the client 311 obtains N data shards and M parity shards; according to the EC redundancy ratio, select the target logical unit from the candidate logical units, and write the N data shards and M parity shards into the corresponding target logical units through the network device 320, so as to realize that the client 311 writes data into the distributed storage system 330.

[0065] Exemplarily: as Figure 3 shown, taking M = 4 and N = 2 as an example, when the client 311 on the computing device 310 needs to write data, the data is divided into 4 data shards, and 2 parity shards are obtained through the EC algorithm based on the 4 data shards, obtaining 4 data shards and 2 parity shards. According to the EC redundancy ratio (N + M: 1), determine the target logical unit, and write the 4 data shards and 2 parity shards into the corresponding target logical units respectively, so as to realize that the client 311 writes data into the distributed storage system 330.

[0066] Among them, the network device 320 is connected to the computing device 310 and is also connected to the distributed storage system 330. Through the network device 320, communication (i.e., data interaction) between the computing device 310 and the distributed storage system 330 is realized. Further, each storage node (i.e., storage node 331, storage node 332, and storage node 333) in the distributed storage system 330 is respectively connected to the network device 320, so that each storage node in the distributed storage system 330 can also communicate (i.e., data interaction) through the network device 320.

[0067] The network device 320 refers to various hardware and software components used to build, manage, and maintain a computer network, which plays a crucial role in data communication and ensures that data can be reliably transmitted between different nodes in the network. Exemplarily, the network device 320 can be a switch, routers, etc.

[0068] Among them, the distributed storage system 330 includes multiple storage nodes, such as Figure 3As shown in the figure, the distributed storage system includes 3 storage nodes, namely storage node 331, storage node 332, and storage node 333. In a possible implementation, each storage node can be one or more servers, such as blade servers, high-density servers, rack servers, cabinet servers, etc. As Figure 3 shown, taking each storage node as a computing device as an example. Each storage node in the distributed storage system 330 includes one or more multi-arm disks, and each multi-arm disk corresponds to multiple logical units (LUN / Disk), as Figure 3 shown, each node of the distributed storage system 330 includes two dual-arm disks, and each dual-arm disk corresponds to two logical units.

[0069] Furthermore, the servers corresponding to the respective storage nodes included in the distributed storage system 330 form a server cluster of the distributed storage system. A storage system software is deployed on the server cluster, that is, all servers in the server cluster are deployed with / running the storage system software, and the storage system software is used to manage and maintain the distributed storage system, for example: used to handle faults (data reconstruction) in the distributed storage system. It can be understood that the distributed storage system 330 realizes the management and maintenance of itself through the storage system software deployed on its corresponding server cluster.

[0070] In a possible implementation, the distributed storage system 330 may further include a coordination node dedicated to managing and maintaining the distributed storage system, and the server to which the coordination node belongs is also deployed with the storage system software.

[0071] It should be noted that, as Figure 3 shown, the distributed storage system 330 is only an exemplary illustration and does not constitute a limitation thereto. For example: the distributed storage system 330 may further include 2, 4, 5, etc. storage nodes; each storage node may include multi-arm disks and single-arm disks; the multi-arm disk may also be a multi-arm disk including 3 arms, 4 arms, etc., and the present application does not make specific limitations. And Figure 3 the system architecture shown is also only an exemplary illustration and does not constitute a limitation thereto. For example: the data storage system framework may include more or fewer devices or modules than Figure 3 shown, and the present application does not make specific limitations.

[0072] Next, in combination with Figures 4A - 7 , a fault handling method provided by an embodiment of the present application will be introduced in detail. This method is applied to a distributed storage system including multi-arm disks, and the computing device has written shards (N data shards and M parity shards) into the distributed storage system.

[0073] It should be noted that the fault handling method can be executed by the computing device (running the storage system software) belonging to any storage node in the distributed storage system; it can also be executed by the computing devices (running the storage system software) belonging to all / part of the storage nodes in the distributed storage system; in addition, it can also be executed by the coordination node (running the storage system software) dedicated to managing and maintaining the distributed storage system in the distributed storage system.

[0074] As Figure 4A shown, a fault handling method provided by an embodiment of the present application includes the following steps:

[0075] S410. Obtain fault information.

[0076] Among them, the fault information is used to indicate the situation where the logical unit storing the shard (data shard and / or parity shard) fails. Specifically, the fault information includes: the identifier of the faulty logical unit.

[0077] In a possible implementation manner, the storage system software deployed in the distributed storage system will regularly perform a health check to monitor the status of the logical units in each storage node, including: checking data consistency / integrity, etc. Exemplarily: The distributed storage system detects data integrity by comparing redundant data (N data shards and M parity shards), so as to detect the status of the logical units in each storage node. For example: When the distributed storage system compares redundant data and finds that one of the parity shards is missing, it determines that the logical unit storing the parity shard fails, and then reports the fault information carrying the identifier of the faulty logical unit.

[0078] In another possible implementation manner, the storage nodes in the distributed storage system maintain a heartbeat connection with the distributed storage system through the heartbeat mechanism. Specifically, multiple storage nodes regularly send heartbeat messages to the storage system software running in the server cluster, so that the storage system software can obtain whether the current working status is normal, so as to ensure an effective connection mechanism between multiple storage nodes and the server cluster. Exemplarily, the storage node reports fault information through the heartbeat message, and the fault information includes the identifier of the faulty logical unit, and may also include the situation where the faulty logical unit fails, etc.

[0079] It should be noted that in addition to the above two ways of obtaining the fault information reported by the storage node, the fault information can also be obtained through performance monitoring, that is, the distributed storage system continuously monitors the performance indicators of the logical unit, etc. The present application does not make specific limitations.

[0080] S420. Determine the fault type based on the fault information.

[0081] Among them, the fault types include: disk fault and magnetic arm fault.

[0082] A disk failure is a type in which the proportion of the failed logical unit corresponding to the disk in all the logical units corresponding to the disk is not less than a preset proportion, that is, for the same disk, when the failed logical unit / all logical units is not less than the preset proportion, the failure type of the disk is a disk failure, where the preset proportion belongs to (0, 1]. Specifically, for a single-arm disk, since there is only one arm in the single-arm disk, that is, the single-arm disk corresponds to only one logical unit. When the logical unit of the single-arm disk to which it belongs fails, it means that the only arm of the single-arm disk fails, that is, the proportion of the failed logical unit of the single-arm disk in all logical units is 100% (1), and this proportion must not be less than the preset proportion. Therefore, the failure type of the single-arm disk is a disk failure; for a multi-arm disk, since the multi-arm disk includes multiple arms, that is, the multi-arm disk corresponds to multiple logical units. Exemplarily, assuming that the preset proportion is 70% (0.7), for a multi-arm disk with 5 arms, when there are 4 failed logical units in the multi-arm disk, the proportion of the failed logical units in the 5 logical units corresponding to the multi-arm disk is 80%, which is not less than the preset proportion of 70%. Therefore, the failure type of this multi-arm disk is a disk failure.

[0083] Furthermore, since the preset proportion is a positive number not greater than 1, when all the logical units corresponding to the disk fail, the failure type of the disk must be a disk failure.

[0084] An arm failure is a type in which the proportion of the failed logical unit corresponding to the disk in all the logical units corresponding to the disk is less than the preset proportion, that is, for the same disk, when the failed logical unit / all logical units is less than the preset proportion, the failure type of the disk is an arm failure. Specifically, for a single-arm disk, there is no arm failure because the single-arm disk corresponds to only one logical unit. Therefore, when one logical unit fails, all the logical units of the single-arm disk fail (the failed logical unit / all logical units is 100%), that is, the failure type of the single-arm disk is a disk failure; for a multi-arm disk, since the multi-arm disk includes multiple arms, that is, the multi-arm disk corresponds to multiple logical units. When the proportion of the failed logical units in all logical units is less than the preset proportion, the failure type of the multi-arm disk is an arm failure. Exemplarily, assuming that the preset proportion is 70% (0.7), for a multi-arm disk with 5 arms, when there are 3 failed logical units in the multi-arm disk, the proportion of the failed logical units in the 5 logical units corresponding to the multi-arm disk is 60%, which is less than the preset proportion of 70%. Therefore, the failure type of this multi-arm disk is an arm failure.

[0085] In a possible implementation, the preset proportion is 1 (i.e., 100%). When all the logical units corresponding to a disk fail, the failure type of the disk is disk failure; when some of the logical units corresponding to the disk fail, the failure type of the disk is arm failure.

[0086] In a possible implementation, based on the failure information and the stored annotation information, determine the failure type of the faulty disk.

[0087] Among them, the annotation information is used to indicate all the logical units belonging to the same disk. Exemplarily, the annotation information includes the correspondence between the disk and all its corresponding logical units. And the annotation information is obtained and stored during the initialization phase of the distributed storage system.

[0088] In a possible implementation, the annotation information is obtained in the following way:

[0089] When each storage node of the distributed storage system is powered on, the distributed storage system can initialize the disks in each storage node and obtain the disk information of each storage node. The disk information includes: the identifier of the logical unit, the identifier of the disk (for example: disk serial number, World Wide Name, etc.), the disk type, etc. Based on the identifier of the disk, annotate all the logical units belonging to the same disk to obtain the annotation information and store it in the distributed storage system itself, for example: a specific database.

[0090] Among them, the identifier of the logical unit is used to uniquely identify the logical unit. Exemplarily, the identifier of the logical unit is the Logical Unit Number (LUN). LUN is a unique identifier for a logical unit / logical volume, and it represents a storage resource that can be recognized and used by the operating system.

[0091] Among them, the identifier of the disk is used to uniquely identify the disk. Exemplarily, the identifier of the disk is the Serial Number (SN). SN is a unique identifier assigned by the manufacturer to each physical disk and is usually recorded in the disk firmware; the identifier of the disk can also be the World Wide Name (WWN), and WWN is a globally unique identifier.

[0092] Among them, the disk types include: single-arm disks and multi-arm disks. Further, the disk types can be refined into single-arm disks, double-arm disks, triple-arm disks, etc.

[0093] Specifically, when each storage node of the distributed storage system is powered on, the disks of each storage node will perform initialization operations. During the initialization process, the disks will report disk-related information (such as disk identifiers, disk types, etc.) to the upper-layer operating system. On the operating systems of each storage node (corresponding servers), logical units (LUN / Disk) corresponding to the disks can be generated. And since each storage node belongs to the same distributed storage system, each storage node can obtain or synchronize the information of the logical units of other storage nodes to obtain the disk information of each storage node.

[0094] In a possible implementation, the distributed storage system (i.e., the servers corresponding to each storage node) can also call each disk interface to obtain or read the information of all disks in the distributed storage system (i.e., the disk information of each storage node).

[0095] In a possible implementation, the structure of the disk information is "identifier of the logical unit - identifier of the corresponding disk - corresponding disk type", for example: identifier of logical unit a1 - identifier of disk A to which logical unit a1 belongs - disk type of disk A.

[0096] It should be noted that the disk type can be directly obtained during the initialization process of the distributed storage system, or it can be the disk type comprehensively obtained based on the identifier of the logical unit and the identifier of the disk during the initialization process of the distributed storage system. For example: the obtained disk information includes: "LUN1 - SN30", "LUN2 - SN30", "LUN3 - SN31", "LUN4 - SN30", then it is determined that the disk with the identifier SN30 includes three logical units, which are the logical units with the identifiers LUN1, LUN2, and LUN4 respectively, so it is determined that the disk type of the disk with the identifier SN30 is a multi-arm disk (or three-arm disk); the disk with the identifier SN31 includes one logical unit, which is the logical unit with the identifier LUN3, so it is determined that the disk type of the disk with the identifier SN31 is a single-arm disk.

[0097] Specifically, after obtaining the disk information of each storage node, based on the identifier of the disk in the disk information, all logical units belonging to the same disk are marked, that is, the identifiers of all logical units belonging to the same disk are recorded, so as to obtain the marking information.

[0098] In a possible implementation, after obtaining the disk information of each storage node, traverse / check the identifiers of the disks in the disk information and their corresponding disk types. If the disk type is a multi-arm disk, based on the identifier of the disk corresponding to the multi-arm disk, label all the logical units belonging to the same multi-arm disk, that is, record the identifiers of all the logical units belonging to the same multi-arm disk, so as to obtain the labeling information. If the disk type is a single-arm disk, there is no need to label the logical units belonging to the single-arm disk again, because a single-arm disk only includes one logical unit, so the obtained disk information already clearly indicates the single-arm disk and its corresponding logical unit, and there is no need for repeated labeling, which simplifies the operation process and avoids unnecessary resource waste during the initialization of the distributed storage system.

[0099] For the convenience of understanding, the following combines Figure 5 , and gives an example to introduce how to obtain the labeling information.

[0100] As Figure 5 shown, the disk information obtained by the distributed storage system is: "Disk1-WWN / SN1-multi-arm disk", "Disk2-WWN / SN1-multi-arm disk", "Disk3-WWN / SN2-multi-arm disk", "Disk4-WWN / SN2-multi-arm disk", "Disk5-WWN / SN3-multi-arm disk", "Disk6-WWN / SN3-multi-arm disk".

[0101] Then, according to the identifiers of the disks in the disk information, label all the logical units belonging to the same disk. Specifically, according to the identifier of the disk "WWN / SN1", determine and label the logical units "Disk1" and "Disk2" corresponding to "WWN / SN1"; according to the identifier of the disk "WWN / SN2", determine and label the logical units "Disk3" and "Disk4" corresponding to "WWN / SN2"; according to the identifier of the disk "WWN / SN3", determine and label the logical units "Disk5" and "Disk6" corresponding to "WWN / SN3". The final obtained labeling information includes: WWN / SN1->(Disk1, Disk2), WWN / SN2->(Disk3, Disk4), WWN / SN3->(Disk5, Disk6), which is used to indicate all the logical units belonging to the same disk.

[0102] Specifically, based on the fault information and the stored annotation information, determine the fault type of the disk, including: traversing the identifiers of the fault logical units included in the fault information; based on the identifiers of the fault logical units and the stored annotation information, determine the corresponding faulty disks; based on the identifiers of the faulty disks and the annotation information, determine all the logical units belonging to the faulty disks; based on the fault information, determine whether the proportion of the fault logical units in the faulty disks among all the logical units in the faulty disks is not less than a preset proportion; when the proportion of the fault logical units in the faulty disks among all the corresponding logical units is not less than the preset proportion, the fault type of the faulty disk is a disk fault; when the proportion of the fault logical units in the faulty disks among all the corresponding logical units is less than the preset proportion, the fault type of the faulty disk is an arm fault.

[0103] That is, for the identifier of the currently traversed fault logical unit, determine the identifier of the disk to which the fault logical unit belongs, and the disk indicated by the identifier of the belonging disk is the faulty disk; based on the identifier of the faulty disk, obtain all the logical unit identifiers belonging to the faulty disk from the stored annotation information, and based on the fault information, determine whether the proportion of the fault logical units belonging to the faulty disk among all the corresponding logical units of the faulty disk is not less than the preset proportion.

[0104] Exemplarily, taking the preset proportion as 1 and the annotation information as: SN40 -> (LUN1, LUN2, LUN4), SN41 -> (LUN3) as an example, assume the fault information includes: LUN2 and LUN4. Traverse the identifiers of the fault logical units included in the fault information. For LUN2, based on LUN2, determine the identifier SN40 of the disk corresponding to LUN2, that is, the disk indicated by SN40 is the faulty disk; based on SN40, obtain the identifiers LUN1, LUN2, LUN4 of all the logical units corresponding to SN40 from the annotation information, and based on the fault information, determine that among all the logical units belonging to the disk with the identifier SN40, there are logical unit identifiers LUN2 and LUN4 with logical unit faults, that is, two logical unit faults. Then, the proportion of the fault logical units of the disk with the identifier SN40 among all the corresponding logical units is 2 / 3, which is not greater than the preset proportion 1. So, the fault type of the disk with the identifier SN40 is an arm fault, and it is the fault of the arm corresponding to the logical units with the identifiers LUN2 and LUN4 in the disk with the identifier SN40; the judgment process for LUN4 is the same as that for LUN2, and the result obtained is that the fault type of the disk with the identifier SN40 is an arm fault.

[0105] S430. Reconstruct at least one fault logical unit in the faulty disk based on the fault type.

[0106] Specifically, when the fault type is a disk fault, reconstruct all logical units in the faulty disk; when the fault type is an arm fault, reconstruct all faulty logical units in the faulty disk.

[0107] For ease of understanding, the following specifically introduces the implementation of a fault handling method provided by an embodiment of the present application in conjunction with Figure 4B the following content.

[0108] As Figure 4B shown, a fault handling method provided by an embodiment of the present application includes the following steps:

[0109] S401. Obtain fault information.

[0110] S402. Determine the fault type based on the fault information.

[0111] It should be noted that S401 and S402 are the same as S410 and S420 above. Therefore, for the specific implementation manners of S401 and S402, please refer to S410 and S420 above, and details will not be elaborated here.

[0112] When the fault type is a disk fault, then perform S403.

[0113] When the fault type is an arm fault, then perform S404.

[0114] S403. Reconstruct the data stored in all logical units in the faulty disk.

[0115] When the fault type of the faulty disk is a disk fault, it indicates that the proportion of faulty logical units in all corresponding logical units is not less than a preset proportion. Therefore, reconstruct the data of the faulty disk based on the remaining shards (the remaining data shards and parity shards), that is, reconstruct the data (data shards or parity shards) stored in all logical units in the faulty disk.

[0116] In a possible implementation manner, through the EC algorithm, recalculate the shards stored in all logical units in the faulty disk based on the remaining data shards and parity shards, and copy them to the idle logical units of the remaining non-faulty disks.

[0117] In a possible implementation manner, when reconstructing the data stored in all logical units in the faulty disk, support setting the reconstruction configuration of the distributed storage system. Among them, the reconstruction configuration includes reconstruction speed, current limiting function, etc. For example: set the reconstruction speed of the distributed storage system to high speed, that is, set the distributed storage system to support high-speed reconstruction: set the distributed storage system to support the current limiting function, and set the current limiting conditions.

[0118] Specifically, set the reconstruction speed of the distributed storage system, that is, complete the reconstruction of shards (data) according to certain reconstruction speed requirements. Exemplarily, set the reconstruction speed of the distributed storage system to high speed, that is, set the distributed storage system to support high-speed reconstruction. At this time, the distributed storage system selects to complete the reconstruction as fast as possible and utilizes all available resources to accelerate the reconstruction process of data (shards).

[0119] Specifically, set the distributed storage system to support the flow-limiting function and set the flow-limiting conditions to prevent the reconstruction process from having too much impact on other services of the distributed storage system. Exemplarily, the flow-limiting conditions include: maximum bandwidth, maximum input / output (I / O) number, etc. Among them, setting the maximum bandwidth means setting the maximum bandwidth allowed to be used during the reconstruction process to prevent the reconstruction process from occupying too much network resources; setting the maximum input / output number means setting the maximum number of I / O operations allowed to be executed per second or per minute during the reconstruction process to avoid putting too much pressure on the distributed storage system.

[0120] In a possible implementation manner, when the fault type is a disk fault, after reconstructing the data stored in all logical units in the faulty disk, it further includes: setting the disk replacement flag corresponding to the faulty disk to the first disk replacement identifier; in response to the disk replacement flag corresponding to the faulty disk being the first disk replacement identifier, performing a disk replacement operation on the faulty disk.

[0121] For ease of understanding, the following combines Figure 6 to introduce in detail the fault handling method for a disk with a disk fault as the fault type. As Figure 6 shown, it includes the following steps:

[0122] S601. Reconstruct the data stored in all logical units in the faulty disk.

[0123] S602. Set the disk replacement flag corresponding to the faulty disk to the first disk replacement identifier.

[0124] Among them, the disk replacement flag corresponding to the faulty disk is used to indicate whether the faulty disk needs to be replaced after the loss / fault shards (data) in the faulty disk are reconstructed. When the disk replacement identifier corresponding to the faulty disk is the first disk replacement identifier, it means that the faulty disk needs to be replaced after the lost / faulty shards (data) in the faulty disk are reconstructed; when the disk replacement identifier corresponding to the faulty disk is the second disk replacement identifier, it means that the faulty disk does not need to be replaced after the faulty / lost shards (data) in the faulty disk are reconstructed.

[0125] In a possible implementation manner, each disk in the distributed storage system has its corresponding disk replacement flag, and the disk and the disk replacement flag are stored in the database in a one-to-one correspondence.

[0126] Exemplarily, set the disk replacement flag corresponding to the faulty disk to "Y", where "Y" is the first disk replacement identifier.

[0127] S603. In response to the disk replacement flag corresponding to the faulty disk being the first disk replacement identifier, perform a disk replacement operation on the faulty disk.

[0128] In a possible implementation, in response to the disk replacement flag corresponding to the faulty disk being the first disk replacement identifier, the distributed storage system automatically performs a disk replacement operation on the faulty disk. Exemplarily, in response to the disk replacement flag corresponding to the faulty disk being the first disk replacement identifier, the distributed storage system automatically selects a suitable new disk from the backup disks according to a preset rule and replaces the faulty disk, thereby completing the disk replacement operation on the faulty disk.

[0129] Generally, the distributed storage system pre-configures a certain number of backup disks so that they can be immediately put into use when needed. And, the suitable new disk is generally a disk in the backup disks that has the same structure as the faulty disk.

[0130] In another possible implementation, in response to the disk replacement identifier corresponding to the faulty disk being the first disk replacement identifier, the distributed storage system triggers a disk replacement instruction; so that the user can perform a disk replacement operation on the faulty disk based on the disk replacement instruction.

[0131] Among them, the disk replacement instruction is used to prompt / guide the user to replace the faulty disk.

[0132] Specifically, the user removes the faulty disk from the distributed storage system based on the disk replacement instruction and inserts a replacement disk with the same structure as the faulty disk, thereby completing the disk replacement operation on the faulty disk.

[0133] In a possible implementation, the distributed storage system can display the disk replacement instruction through the servers corresponding to each storage node to prompt / guide the user to replace the faulty disk; in another possible implementation, in response to the disk replacement identifier corresponding to the faulty disk being the first disk replacement identifier, the distributed storage system triggers the indicator light of the faulty disk to flash at a preset frequency to trigger the disk replacement instruction, thereby prompting / guiding the user to replace the faulty disk.

[0134] The above combination Figure 6 has introduced in detail the fault handling method for disks with the fault type of disk fault. Next, continue to introduce Figure 4B a fault handling method provided by an embodiment of the present application.

[0135] S404. Reconstruct the data stored in the faulty logical unit of the faulty disk.

[0136] When the failure type of the failed disk is an arm failure, it indicates that the proportion of the failed logical unit in the failed disk (failed multi-arm disk) among all the corresponding logical units is less than a preset proportion. Therefore, the data (data shard or parity shard) stored in the failed logical unit is reconstructed based on the remaining shards (remaining data shards and parity shards), that is, the data of the failed arm is reconstructed.

[0137] In a possible implementation, through the EC algorithm, based on the remaining data shards and parity shards, the shards stored in the failed logical unit in the failed disk are recalculated and copied to the logical units in the remaining non-failed spaces.

[0138] In a possible implementation, when reconstructing the data stored in the failed logical unit in the failed disk, setting the reconstruction configuration of the distributed storage system is supported.

[0139] In a possible implementation, when the failure type is an arm failure, after reconstructing the data stored in the failed logical unit in the failed disk, it further includes: setting the disk change flag corresponding to the failed disk to the second disk change identifier, and / or setting the health status corresponding to the failed disk to the sub-healthy state; expanding the distributed storage system based on the unusable storage capacity in the distributed storage system.

[0140] For ease of understanding, the following Figure 7 introduces in detail the fault handling method for a disk with an arm failure type. As Figure 7 shown, it includes the following steps:

[0141] S701. Reconstruct the data stored in the failed logical unit in the failed disk.

[0142] S702. Set the disk change flag corresponding to the failed disk to the second disk change identifier.

[0143] Among them, setting the disk change flag corresponding to the failed disk to the second disk change identifier is used to indicate that after the lost / failed shards (data) in the failed disk are reconstructed, there is no need to replace the failed disk.

[0144] Exemplarily, set the disk change flag corresponding to the failed disk to "N", and "N" is the second disk change identifier.

[0145] S703. Set the health status corresponding to the failed disk to the sub-healthy state.

[0146] Among them, the health status corresponding to the disk is used to indicate the current state of the disk. Specifically, the health status corresponding to the disk includes: normal state, sub-healthy state, and failed state.

[0147] The "Healthy" state indicates that the disk is operating well, all performance is within the preset range, and no errors or abnormalities are detected; the "Failed" state indicates that the disk can no longer work properly and cannot continue to provide reliable services. The "Sub-healthy" or "Degraded" state is a state between the normal state and the failed state, indicating that there are some non-fatal problems with the disk.

[0148] It should be noted that in the embodiments of this application, the cases where the health state of the disk corresponds to the normal state and the failed state are not considered, and only the case where the health state of the failed disk corresponds to the sub-healthy state is considered.

[0149] In a possible implementation, the health state corresponding to the disk is empty and sub-healthy, or the health state corresponding to the disk is sub-healthy and not sub-healthy.

[0150] In a possible implementation, each disk in the distributed storage system has its corresponding health state, and the disk and the health state are stored in the database in a one-to-one correspondence. That is, the disk, the disk replacement flag, and the health state are stored in the database in a corresponding manner.

[0151] For the convenience of understanding, the following combines Figure 8 to separately introduce the processing method for the fault type of disk fault and the processing method for the fault type of arm fault by way of examples.

[0152] As Figure 8 shown in (a) and (b) in

[0153] As Figure 8As shown in (a) therein, take the dual-arm disk 820 as the failed disk, and the failure type of the dual-arm disk 820 is disk failure, that is, both the logical units 821 and 822 corresponding to the dual-arm disk 820 fail. The data (shards) stored in the logical units 821 and 822 are all lost. Then, according to the remaining data shards and parity shards (i.e., 3 data shards and 1 parity shard), the data shards in the logical unit 821 and the parity shard in the logical unit 822 are reconstructed, and they are respectively saved to the logical unit 812 of the dual-arm disk 810 and the logical unit 832 of the dual-arm disk 830. When the data is reconstructed, the disk replacement flag of the dual-arm disk 820 is set to the first disk replacement identifier (i.e., "Y"). In response to the disk replacement flag of the dual-arm disk 820 being the first disk replacement identifier, a disk replacement operation for the dual-arm disk 820 is performed.

[0154] As Figure 8 described in (b) therein, take the dual-arm disk 820 as the failed disk, and the failure type of the dual-arm disk 820 is arm failure, that is, the logical unit 822 corresponding to the dual-arm disk 820 fails. The parity shard stored in the logical unit 822 is lost. Then, according to the remaining data shards and parity shards (i.e., 4 data shards and 1 parity shard), the parity shard in the logical unit 822 is reconstructed, and it is saved to the logical unit 812 of the dual-arm disk 810. When the data is reconstructed, the disk replacement flag of the dual-arm disk 820 is set to the second disk replacement identifier (i.e., "N"), and the health status of the dual-arm disk 820 is set to the sub-healthy state.

[0155] S704. Determine whether the unusable storage capacity in the distributed storage system is greater than the first preset storage capacity.

[0156] Since when the failed disk (failed multi-arm disk) has an arm failure, that is, some logical units in the failed (multi-arm) disk fail, but there are still other logical units that do not fail and can be used normally, only the data stored in the failed logical units is reconstructed, and no disk replacement operation is performed on the failed disk with an arm failure. Since no disk replacement operation is performed on the failed (multi-arm) disk with an arm failure, there are still failed logical units in the distributed storage system that cannot be used. Then, the distributed storage system needs to undertake all tasks through the remaining storage capacity (storage resources). The larger the total storage capacity of the failed logical units in the distributed storage, that is, the larger the unusable storage capacity, the smaller the usable storage capacity in the distributed storage system. Correspondingly, the more tasks (IO operations) that the unit storage capacity in the distributed storage system needs to undertake, resulting in a decline in the performance of the distributed storage system, and even an overall performance bottleneck due to overload.

[0157] When the unusable storage capacity in the distributed storage system is greater than the first preset storage capacity, it indicates that the unusable storage capacity in the distributed storage system exceeds the threshold, and the usable storage capacity in the distributed storage system is too small, resulting in a significant decline in the performance of the distributed storage system, and even a performance bottleneck, unable to meet the user's requirements for the distributed storage system; when the unusable storage capacity in the distributed storage system is not greater than the first preset storage capacity, it indicates that the unusable storage capacity in the distributed storage system does not exceed the threshold, and the usable storage capacity in the distributed storage system can meet the user's requirements for the distributed storage system.

[0158] In a possible implementation, traverse the faulty disks with a sub-healthy status in the distributed storage system, and determine whether the total storage capacity of the faulty logical units in the faulty disks with a sub-healthy status is greater than the first preset storage capacity. Among them, the total storage capacity of the faulty logical units is the unusable storage capacity.

[0159] Specifically, traverse the faulty disks with a sub-healthy status in the distributed storage system, obtain the total storage capacity of the faulty logical units in the faulty disks with a sub-healthy status, and determine whether the total storage capacity of the faulty logical units is greater than the first preset storage capacity.

[0160] In another possible implementation, traverse the faulty disks with a disk replacement flag of the second disk replacement identifier in the distributed storage system, and determine whether the total storage capacity of the faulty logical units in the faulty disks with a disk replacement flag of the second disk replacement identifier is greater than the first preset storage capacity. Among them, the total storage capacity of the faulty logical units is the unusable storage capacity.

[0161] Specifically, traverse the faulty disks with a disk replacement flag of the second disk replacement identifier in the distributed storage system, obtain the total storage capacity of the faulty logical units in the faulty disks with a disk replacement flag of the second disk replacement identifier, and determine whether the total storage capacity of the faulty logical units is greater than the first preset storage capacity.

[0162] In a possible implementation, the first preset storage capacity is obtained based on the first preset ratio and the total storage capacity of the distributed storage system. Exemplarily: the first preset storage capacity is 5% of the total storage capacity of the distributed storage system, where "5%" is the first preset ratio.

[0163] When the unusable storage capacity in the distributed storage system is not greater than the first preset storage capacity, the fault handling method provided in the embodiment of the present application can continue to be performed.

[0164] When the unusable storage capacity in the distributed storage system is greater than the first preset storage capacity, then perform S705.

[0165] S705. Expand the distributed storage system.

[0166] Specifically, when the unusable storage capacity in the distributed storage system is greater than the first preset storage capacity, it indicates that the available storage capacity in the distributed storage system is too small, resulting in a significant decline in the performance of the distributed storage system and even a performance bottleneck, unable to meet the user's requirements for the distributed storage system. Therefore, it is necessary to expand the distributed storage system to meet the user's requirements for the distributed storage system (such as performance requirements and storage capacity requirements).

[0167] Among them, the expansion methods include: disk expansion and storage node expansion. That is, perform disk expansion on the distributed storage system or perform storage node expansion on the distributed storage system. Among them, disk expansion refers to adding additional disks to the existing storage nodes to increase the storage capacity of a single storage node, thereby increasing the storage capacity of the distributed storage system; storage node expansion refers to adding new storage nodes to the distributed storage system, thereby increasing the storage capacity of the distributed storage system.

[0168] In a possible implementation, specifically expanding the disk of the distributed storage system is as follows: To ensure the balance of capacity and performance, disks are added to each storage node in the distributed storage system, and the number, type, and storage capacity of the disks added to each storage node are the same. Exemplarily, assume that the distributed storage system includes 3 storage nodes, and a dual-arm disk with a storage capacity of 17T is added to each of the 3 storage nodes in the distributed storage system.

[0169] In a possible implementation, specifically expanding the storage nodes of the distributed storage system is as follows: To ensure the balance of capacity and performance, at least one storage node is added to the distributed storage system, and the newly added storage nodes are the same as the original storage nodes in the distributed storage system, that is, the number of disks, disk types, and disk capacities included in the newly added storage nodes are the same as those of the original storage nodes.

[0170] In a possible implementation, when expanding the distributed storage system, the expanded storage capacity is greater than the second preset storage capacity. The second preset storage capacity is obtained based on the second preset ratio and the total storage capacity of the distributed storage system.

[0171] Among them, the second preset storage capacity is greater than or equal to the first preset storage capacity, so as to ensure that the expanded distributed storage system can meet the user's requirements for the distributed storage system.

[0172] It should be noted that in Figure 7In the possible implementation shown, taking "setting the disk replacement flag of the faulty disk to the second disk replacement identifier and setting the health status corresponding to the faulty disk to sub-healthy" as an example, it is also possible to only "set the disk replacement flag of the faulty disk to the second disk replacement identifier", and then "determine the unusable storage capacity in the distributed storage system based on the second disk replacement identifier"; it is also possible to only "set the health status corresponding to the faulty disk to sub-healthy", and then "indicate that there is no need to replace the faulty disk based on the sub-healthy status".

[0173] The above combination Figure 7 Details the fault handling method for disks with the fault type of arm fault. In a possible implementation, after reconstructing at least one faulty logical unit in the faulty disk, it is also possible to perform a balancing operation on the distributed storage system. Next, continue to combine Figure 4B Introduce a fault handling method provided by an embodiment of the present application.

[0174] S405. Perform a balancing operation on the storage capacity of the distributed storage system.

[0175] The distributed storage system performs a rebalancing operation on its own storage capacity to ensure that the storage capacity usage of each storage node and each disk reaches equilibrium, that is, to ensure that the usage rates of all storage nodes or disks are similar and avoid overloading of some storage nodes or disks. Thereby optimizing performance, improving resource utilization, and enhancing the stability and reliability of the distributed storage system.

[0176] Exemplarily, assume that the distributed storage system is Ceph. Then the Ceph distributed storage system uses the Controlled Replication Under Scalable Hashing (CRUSH) algorithm to determine how data is distributed to each Object Storage Daemon (OSD), so as to achieve dynamic adjustment of data distribution according to the actual usage conditions of storage nodes and disks, that is, to achieve the balancing operation. In Ceph, an OSD usually corresponds to a disk or a logical unit.

[0177] An embodiment of the present application provides a fault handling method, including: obtaining fault information; determining a fault type based on the fault information; reconstructing at least one faulty logical unit in the faulty disk based on the fault type; and expanding the storage capacity of the distributed storage system. In the embodiment of the present application, when the fault type is a disk fault, the data stored in all logical units in the faulty disk is reconstructed; when the fault type is an arm fault, the faulty logical unit in the faulty disk is reconstructed. Therefore, the embodiment of the present application can effectively distinguish the fault types of the faulty disks, that is, it can distinguish whether the multi-arm disk has a whole disk fault or a partial arm fault, so as to determine the scope of data reconstruction, avoid data loss and even EC degradation, and thus ensure the reliability of the distributed storage system.

[0178] Further, for a faulty disk with an arm fault type, it is determined whether the unusable storage capacity in the distributed storage system is greater than a first preset storage capacity. If the unusable storage capacity in the distributed storage system is greater than the first preset storage capacity, the distributed storage system is expanded to meet the user's requirements for the distributed storage system (such as performance requirements, storage capacity requirements, etc.).

[0179] The above-mentioned embodiment 1 details a fault handling method provided by the embodiment of the present application. The fault handling method introduced in embodiment 1 does not consider specific application scenarios. Next, through embodiment 2, different application scenarios are specifically described, and another fault handling method provided by the embodiment of the present application is described.

[0180] Embodiment 2:

[0181] Next, in combination with Figures 9 - 13 , a detailed introduction is made to another fault handling method provided by the embodiment of the present application in different application scenarios.

[0182] Before introducing another fault handling method provided by the embodiment of the present application, three application scenarios are first introduced.

[0183] Among them, the three application scenarios are: a performance-first scenario, a cost-first scenario, and a reliability-first scenario.

[0184] The performance-first scenario is to prioritize ensuring the performance of the distributed storage system, targeting HPC, big data online analysis and computing services, etc. In the performance-first scenario, it is required to ensure the high performance of the distributed storage system, that is, when a logical unit in the distributed storage system fails (a disk fails), the business performance of the distributed storage system (such as CPU performance, memory performance, network performance, etc.) is guaranteed to the greatest extent, and it is ensured that data reconstruction is carried out without affecting the business performance.

[0185] The cost - priority scenario is to prioritize ensuring the cost of the distributed storage system, targeting scenarios such as HPC, big data archiving and backup, etc. In the cost - priority scenario, it is required to ensure a relatively low cost of the distributed storage system. That is, when a logical unit in the distributed storage system fails (a disk fails), it can ensure that the cost of the distributed storage system is extremely low.

[0186] The reliability - priority scenario is to prioritize ensuring the reliability of the distributed storage system, targeting core assets such as HPC core data, drawings, etc. That is, in the reliability - priority scenario, it is required to ensure the reliability of the distributed storage system. That is, when a logical unit in the distributed storage system fails (a disk fails), it can ensure that the reliability of the distributed storage system is extremely high.

[0187] For the convenience of understanding, the following combines Figure 9 、 Figure 10 and Figure 11 to introduce the fault - handling methods provided by the embodiments of the present application in the performance - priority scenario, cost - priority scenario, and reliability - priority scenario respectively.

[0188] As Figure 9 shown, in the performance - priority scenario, the fault - handling method provided by the embodiments of the present application includes the following steps:

[0189] S901. Obtain fault information.

[0190] It should be noted that the above S901 is the same as S401 in Embodiment 1. For the specific implementation details of S901, please refer to S401 in Embodiment 1, and will not be elaborated here.

[0191] S902. Determine whether the business performance of the distributed storage system meets the preset conditions.

[0192] Among them, the business performance includes: the central processing unit (CPU) usage rate of the server corresponding to the storage node in the distributed storage system, the memory usage rate of the server corresponding to the storage node, the disk usage rate, the network latency, etc.

[0193] In a possible implementation, if the CPU usage rate, memory usage rate, disk usage rate, and network latency of the distributed storage system meet the corresponding preset requirements, it indicates that the business requirements of the distributed storage system meet the preset conditions; otherwise, it indicates that the business performance of the distributed storage system does not meet the preset conditions.

[0194] Exemplarily, when the CPU usage rate of the distributed storage system < 70%, the memory usage rate < 70%, the disk usage rate < 70%, and the network latency < 1ms, it is determined that the service performance of the distributed storage system meets the preset conditions; when one or more of the CPU usage rate, memory usage rate, disk usage rate, and network latency do not meet the preset requirements, for example: CPU usage rate > 70%, memory usage rate < 70%, disk usage rate < 70%, and network latency > 1ms (that is, the CPU usage rate and network latency do not meet the preset conditions), it is determined that the service performance of the distributed storage system does not meet the preset conditions.

[0195] When it is determined that the service performance of the distributed storage system does not meet the preset conditions, S902 is repeated, that is, the service performance of the distributed storage system is continuously monitored until it is determined that the service performance of the distributed storage system meets the preset conditions.

[0196] When it is determined that the service performance of the distributed storage system does not meet the preset conditions, it indicates that the current service of the distributed storage system is relatively busy. If data reconstruction is performed at this time, it may affect the current service performance. Due to the performance priority strategy, it is necessary to give priority to ensuring the service performance of the distributed storage system, so data reconstruction is not performed.

[0197] When it is determined that the service performance of the distributed storage system meets the preset conditions, S903 is performed.

[0198] S903: Determine the type of fault based on the fault information.

[0199] When it is determined that the service performance of the distributed storage system meets the preset conditions, it indicates that the current service of the distributed storage system is relatively not busy. If data reconstruction is performed at this time, it will not affect the current service performance, then data reconstruction continues.

[0200] When the type of fault is a disk fault, S904 is performed.

[0201] When the type of fault is an arm fault, S905 is performed.

[0202] S904: Reconstruct the data stored in all logical units of the faulty disk.

[0203] S905: Reconstruct the data stored in the faulty logical unit of the faulty disk.

[0204] S906: Perform a balancing operation on the storage capacity of the distributed storage system.

[0205] It should be noted that the above S903 - S906 is the same as S402 - S405 in the first embodiment. Therefore, for the specific implementation details of S903 - S906, please refer to S402 - S405 in the first embodiment, which will not be elaborated here.

[0206] A fault handling method provided by an embodiment of the present application. In a performance - priority scenario, when the distributed storage system obtains fault information, it is necessary to determine whether the business performance of the distributed storage system meets a preset condition. When the preset condition is met, at least one faulty logical unit in the faulty disk is reconstructed based on the fault type. In the embodiment of the present application, before data reconstruction, by judging the business performance of the distributed storage system and the preset condition, the priority of the business performance of the distributed storage system is ensured.

[0207] If the business performance of the distributed storage system meets the preset condition, it means that the business performance of the distributed storage system will not be affected by data reconstruction, ensuring the priority of the business performance of the distributed storage system. If the business performance of the distributed storage system does not meet the preset condition, the business performance of the distributed storage system is continuously monitored until it is determined that the business performance of the distributed storage system meets the preset condition. When it is determined that the business performance of the distributed storage system does not meet the preset condition, it means that the current business performance of the distributed storage system may be affected by data reconstruction. Therefore, in order to ensure the priority of the business performance of the distributed storage system, data reconstruction is not performed.

[0208] The above combination Figure 9 introduces a fault handling method provided by an embodiment of the present application in a performance - priority scenario. The following combination Figure 10 introduces a fault handling method provided by an embodiment of the present application in a cost - priority scenario.

[0209] As Figure 10 shown, in a cost - priority scenario, the fault handling method provided by an embodiment of the present application includes the following steps:

[0210] S1001. Obtain fault information.

[0211] S1002. Determine the fault type based on the fault information.

[0212] Specifically, it is judged whether the proportion of the faulty logical unit corresponding to the faulty disk in all the corresponding logical units is not less than a preset proportion; when the proportion of the faulty logical unit corresponding to the faulty disk in all the corresponding logical units is not less than the preset proportion, the fault type is determined to be a disk fault; when the proportion of the faulty logical unit corresponding to the faulty disk in all the corresponding logical units is less than the preset proportion, the fault type is determined to be an arm fault. In order to ensure that the cost of the distributed storage system is extremely low, the preset proportion is set to 1 (100%), that is, only when all the logical units of the disk are faulty is it a disk fault, otherwise it is an arm fault.

[0213] When the fault type is a disk fault, S1003 is performed.

[0214] When the fault type is an arm fault, then perform S1004.

[0215] S1003. Reconstruct the data stored in all logical units of the faulty disk.

[0216] S1004. Reconstruct the data stored in the faulty logical unit of the faulty disk.

[0217] S1005. Perform a balancing operation on the storage capacity of the distributed storage system.

[0218] It should be noted that the above S1001 - S1005 are the same as S401 - S405 in the first embodiment. Therefore, for the specific implementation details of S1001 - S1005, please refer to S401 - S405 in the first embodiment, which will not be elaborated here.

[0219] A fault handling method provided by an embodiment of the present application, in a cost - priority scenario, the distributed storage system obtains fault information. When determining the fault type based on the fault information, the preset ratio is set to 1, that is, when all logical units of the faulty disk are faulty due to a disk fault, subsequently reconstruct all logical units of the faulty disk; when some logical units of the faulty disk are faulty due to an arm fault, only reconstruct the faulty logical units of the faulty disk, and do not additionally reconstruct any non - faulty logical units, thereby ensuring the cost - priority of the distributed storage system.

[0220] The above combination Figure 10 introduced a fault handling method provided by an embodiment of the present application in a cost - priority scenario. The following combination Figure 11 introduces a fault handling method provided by an embodiment of the present application in a reliability - priority scenario.

[0221] As Figure 11 shown, in a reliability - priority scenario, the fault handling method provided by an embodiment of the present application includes the following steps:

[0222] S1101. Obtain fault information.

[0223] It should be noted that the above S1101 is the same as S401 in the first embodiment. For the specific implementation details of S1101, please refer to S401 in the first embodiment, which will not be elaborated here.

[0224] S1102. Determine the fault type based on the fault information.

[0225] When the fault type is a disk fault or an arm fault, perform S1103.

[0226] S1103. Reconstruct the data stored in all logical units of the faulty disk.

[0227] In the reliability - first scenario, whether it is a disk failure or an arm failure, the distributed storage system reconstructs the data stored in all logical units of the disk with a faulty logical unit (i.e., the faulty disk), so as to ensure the reliability of the distributed storage system.

[0228] Specifically, based on the remaining shards (the remaining data shards and parity shards) in the non - faulty disks, the data stored in all logical units of the faulty disk is reconstructed, whether it is the data stored in the faulty logical unit or the data stored in the non - faulty logical unit.

[0229] In a possible implementation, in the reliability - first scenario, regardless of whether the failure type is a disk failure or an arm failure, after reconstructing the data stored in all logical units of the faulty disk, the disk - replacement flag corresponding to the faulty disk is set to the first disk - replacement identifier; in response to the disk - replacement flag corresponding to the faulty disk being the first disk - replacement identifier, a disk - replacement operation for the faulty disk is performed.

[0230] S1104. Perform a balancing operation on the storage capacity of the distributed storage system.

[0231] It should be noted that the above S1103 and S1104 are the same as S403 and S405 in Embodiment 1. Therefore, for the specific implementation details of S1103 and S1104, please refer to S403 and S405 in Embodiment 1, and will not be elaborated here.

[0232] For the sake of easy understanding, the following combines Figure 12 to introduce, by way of example, the processing methods for a disk failure and an arm failure in the reliability - first scenario.

[0233] As Figure 12 shown in (a) and (b) thereof, the distributed storage system includes: three storage nodes, namely storage node 1, storage node 2, and storage node 3, and each storage node includes two dual - arm disks, and each dual - arm disk corresponds to 4 logical units. Among them, 4 (N = 4) data shards are respectively stored in logical unit 1211 of dual - arm disk 1210 in storage node 1, logical units 1241 and 1242 of dual - arm disk 1240 in storage node 2, and logical unit 1251 of dual - arm disk 1250 in storage node 3; 2 (M = 2) parity shards are respectively stored in logical unit 1222 of dual - arm disk 1220 in storage node 1 and logical unit 1262 of dual - arm disk 1260 in storage node 3.

[0234] As Figure 12As shown in (a) therein, take the dual-arm disk 1240 as the failed disk, and the failure type of the dual-arm disk 1240 is disk failure, that is, both the logical units 1241 and 1242 corresponding to the dual-arm disk 1240 fail as an example. If the data (shards) stored in the logical units 1241 and 1242 are all lost, then reconstruct the data shards in the logical unit 1241 and the data shards in the logical unit 1242 according to the remaining data shards and parity shards (i.e., 2 data shards and 2 parity shards), and save them to the logical unit 1212 of the dual-arm disk 1210 and the logical unit 1232 of the dual-arm disk 1230 respectively; when the data reconstruction is completed, set the disk replacement flag of the dual-arm disk 1240 to the first disk replacement identifier (i.e., "Y"); in response to the disk replacement flag of the dual-arm disk 1240 being the first disk replacement identifier, perform a disk replacement operation on the dual-arm disk 1240.

[0235] As Figure 12 described in (b) therein, take the dual-arm disk 1240 as the failed disk, and the failure type of the dual-arm disk 1240 is arm failure, that is, the logical unit 1242 corresponding to the dual-arm disk 1240 fails as an example. Reconstruct the data shards in the logical unit 1241 and the data shards in the logical unit 1242 according to the remaining data shards and parity shards (i.e., 2 data shards and 2 parity shards), and save them to the logical unit 1212 of the dual-arm disk 1210 and the logical unit 1232 of the dual-arm disk 1230 respectively; when the data reconstruction is completed, set the disk replacement flag of the dual-arm disk 1240 to the first disk replacement identifier (i.e., "Y"); in response to the disk replacement flag of the dual-arm disk 1240 being the first disk replacement identifier, perform a disk replacement operation on the dual-arm disk 1240.

[0236] A fault handling method provided by an embodiment of the present application. In a reliability-first scenario, when a distributed storage system obtains fault information, regardless of whether it is a disk fault or an arm fault, reconstruct the data stored in all logical units of the failed disk; in response to the completion of data reconstruction, perform a balancing operation on the storage capacity of the distributed storage system. In the embodiment of the present application, as long as there is a failed disk, regardless of whether it is a disk fault or an arm fault in the failed disk, reconstruct the data stored in all logical units of the failed disk, thereby ensuring the reliability of the distributed storage system.

[0237] The above respectively combines Figure 9 、 Figure 10 and Figure 11 to introduce the fault handling methods in a performance-first scenario, a cost-first scenario, and a reliability-first scenario respectively. Next, in combination with Figure 13 detail, introduce a fault handling method provided by an embodiment of the present application.

[0238] S1301. Obtain the target scenario input or selected by the user.

[0239] Among them, the target scenarios include: performance - priority scenario, cost - priority scenario, and reliability - priority scenario.

[0240] In a possible implementation, the server corresponding to the storage node in the distributed storage system can display a scenario selection interface through a display device, and the user inputs / selects the target scenario based on the displayed scenario selection interface.

[0241] S1302. Execute the corresponding fault - handling process based on the target scenario.

[0242] Specifically, when the target scenario is a performance - priority scenario, execute the fault - handling process corresponding to the performance - priority scenario, that is, the fault - handling process as Figure 9 shown. When the target scenario is a cost - priority scenario, execute the fault - handling process corresponding to the cost - priority scenario, that is, the fault - handling process as Figure 10 shown. When the target scenario is a reliability - priority scenario, execute the fault - handling process corresponding to the reliability - priority scenario, that is, the fault - handling process as Figure 11 shown.

[0243] In addition, the embodiment of the present application also provides a computing device. The server includes a processor and a memory. The processor is coupled to the memory, and the memory stores computer - executable instructions. When the processor executes the computer - executable instructions, it implements the fault - handling method in the above - mentioned embodiment.

[0244] The embodiment of the present application also provides a computer - readable storage medium. A computer program is stored on the computer - readable storage medium. When the computer program runs on a computer, the computer executes the fault - handling method in the above - mentioned embodiment.

[0245] Regarding the explanation and beneficial effects description of the relevant content in any of the above - provided computer - readable storage media, reference can be made to the corresponding embodiments above, and details are not described herein again.

[0246] The embodiments of the present application also provide a computer program product containing instructions. When the instructions run on a computer, the computer is enabled to execute any one of the fault handling methods in the above embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from a website, a computer, a server, or a data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by the computer or a data storage device such as a server or a data center that contains one or more media integrated therein. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as an SSD), etc.

[0247] It should be noted that the devices for storing computer instructions or computer programs provided in the embodiments of the present application, such as but not limited to, the above-mentioned memory, computer-readable storage medium, and communication chip, etc., are all non-transitory.

[0248] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more media integrated therein. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0249] Although the present application has been described in conjunction with various embodiments, however, in the process of implementing the claimed present application, those skilled in the art can understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "one" does not exclude a plurality. A single processor or other unit can implement several functions recited in the claims. Certain measures are recited in mutually different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0250] Although the present application has been described in conjunction with specific features and their embodiments, it is obvious that various modifications and combinations can be made without departing from the spirit and scope of the present application. Accordingly, the present specification and the drawings are merely exemplary illustrations of the present application defined by the appended claims, and are considered to have covered any and all modifications, variations, combinations, or equivalents within the scope of the present application. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these changes and modifications.

Claims

1. A fault handling method, applied to a distributed storage system, wherein the distributed storage system includes at least one storage node, the storage node includes at least one disk, and the disk includes one or more magnetic arms, the method comprising: Acquire fault information; wherein the fault information includes: an identifier of a faulty logic unit; Based on the fault information, determine the fault type; wherein the fault type includes: disk fault and magnetic arm fault; Based on the failure type, at least one failed logical unit in the failed disk is reconstructed.

2. The method according to claim 1, characterized in that: When the fault type is a disk fault, reconstructing at least one faulty logical unit in the faulty disk based on the fault type includes: Based on the fact that the failure type is a disk failure, data stored in all logical units in the failed disk are reconstructed.

3. The method according to claim 2, characterized in that After reconstructing the data stored in all logical units in the failed disk based on the failure type being a disk failure, the method further includes: Setting the disk replacement mark corresponding to the faulty disk as a first disk replacement mark; wherein the disk replacement mark corresponding to the faulty disk is used to indicate whether to replace the faulty disk, and the first disk replacement mark is used to indicate to replace the faulty disk; In response to the disk swap mark corresponding to the failed disk being the first disk swap mark, a disk swap operation is performed on the failed disk.

4. The method according to claim 1, when the fault type is a magnetic arm fault, reconstructing at least one faulty logical unit in the faulty disk based on the fault type comprises: Based on the fact that the fault type is a magnetic arm fault, data stored in a faulty logic unit in the faulty disk is reconstructed.

5. The method according to claim 4, characterized in that After reconstructing the data stored in the faulty logic unit in the faulty disk based on the fault type being a magnetic arm fault, the method further includes: Setting the disk replacement mark corresponding to the faulty disk as a second disk replacement mark; wherein the disk replacement mark corresponding to the faulty disk is used to indicate whether to replace the faulty disk, and the second disk replacement mark is used to indicate not to replace the faulty disk; And / or set the health status corresponding to the faulty disk to a sub-healthy state; wherein the health status is used to indicate the current state of the disk.

6. The method according to claim 4, characterized in that After reconstructing at least one failed logical unit in the failed disk based on the failure type, the method further includes: When the unusable storage capacity in the distributed storage system is greater than the first preset storage capacity, the distributed storage system is expanded.

7. The method according to claim 1, characterized in that The determining the fault type based on the fault information includes: Based on the fault information, determine whether the proportion of the faulty logical unit in the faulty disk to all the logical units in the faulty disk is not less than a preset proportion; wherein the preset proportion is a positive number less than or equal to 1; If yes, the fault type is disk failure; If not, the fault type is a magnetic arm fault.

8. The method according to claim 7, characterized in that The determining, based on the fault information, whether a proportion of the faulty logical unit in the faulty disk to all the logical units in the faulty disk is not less than a preset proportion includes: Traversing the identifiers of the faulty logic units included in the fault information; Determine the corresponding faulty disk based on the identification of the faulty logical unit and the stored labeling information; wherein the labeling information is used to indicate all logical units belonging to the same disk; Based on the identifier of the failed disk and the label information, determining all logical units belonging to the failed disk; Based on the fault information, it is determined whether a proportion of the faulty logical units in the faulty disk to all logical units in the faulty disk is not less than a preset proportion.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: A balancing operation is performed on the storage capacity of the distributed storage system.

10. A computing device, characterized in that: include: Processor and memory; Storage system software runs on the processor, and the storage system software is used to manage and maintain the distributed storage system; The memory is used to store computer program instructions; The processor is used to execute computer program instructions stored in the memory to implement the fault handling method described in any one of claims 1-9.

Images