Fault processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a fault processing method and device, computer equipment, a readable storage medium and a program product, and belongs to the technical field of information. The method is applied to an access layer of a distributed storage system. The method comprises the following steps: acquiring error information, and determining an error type and an error source of the error information; determining a fault level of the error information according to the error type; determining a fault object related to the error information according to the fault level and the error source; the fault object comprises at least one of a fault disk and a fault data set, the fault data set is a data set related to error information in a cache of a proxy service, and the proxy service is used for completing the function of an access layer; and adding the fault object into the fault list, and determining the survival duration of the fault object in the fault list, so that the proxy service carries out fault shielding on the fault object when carrying out data processing in the survival duration. By adopting the method, the availability of the distributed storage system can be improved.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a fault handling method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Ensuring high availability of distributed storage systems has always been a complex and critical task, so that the system can maintain stable operation and data accessibility in the face of various emergencies such as hardware failures, software errors, network problems, etc. To this end, traditional distributed storage systems use a variety of technical means, such as data redundancy, multi-copy storage, cross-region replication, and automatic fault recovery.

[0003] However, with traditional fault handling methods, the distributed storage system still has the problem of low availability after fault handling. Summary of the invention

[0004] Based on this, it is necessary to provide a fault handling method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the availability of a distributed storage system in response to the above technical problems.

[0005] In a first aspect, the present application provides a fault handling method, which is applied to an access layer of a distributed storage system; the method comprises:

[0006] Obtain error information, and determine the error type and error source of the error information;

[0007] Determining a fault level of the error information according to the error type;

[0008] Determine a fault object related to the error information according to the fault level and the error source; the fault object includes at least one of a faulty disk and a fault data set, the fault data set is a data set related to the error information in a cache of a proxy service, the proxy service is used to complete the function of the access layer;

[0009] The fault object is added to a fault list, and the survival time of the fault object in the fault list is determined, so that the proxy service performs fault shielding on the fault object when processing data within the survival time.

[0010] In one embodiment, determining the fault level of the error information according to the error type includes:

[0011] If the error type indicates that an error occurs in the data block, determining that the fault level of the error information is a data block level fault;

[0012] If the error type indicates that a disk error occurs, determining that the fault level of the error information is a disk level fault;

[0013] If the error type indicates that an error occurs in the service process, it is determined that the fault level of the error information is a process level fault.

[0014] In one embodiment, determining a fault object related to the error information according to the fault level and the error source includes:

[0015] If the fault level is a data block level fault, determining a first object related to the error information, the first object comprising a set of fault data related to the error information and having an error in the cache of the proxy service;

[0016] If the fault level is a disk-level fault, determining a second object related to the error information, the second object including a first disk where the error source is located, and a set of fault data related to the error information and associated with the first disk in the cache of the proxy service;

[0017] If the fault level is a process-level fault, a third object related to the error information is determined, and the third object includes a second disk under the same data storage service process as the error source, and a set of fault data related to the error information and associated with the second disk in the cache of the proxy service; the first object, the second object and the third object all belong to fault objects.

[0018] In one embodiment, the distributed storage system further includes a control layer; after adding the fault object to a fault list and determining the survival time of the fault object in the fault list so that the proxy service performs fault shielding on the fault object when processing data within the survival time, the system further includes:

[0019] In response to an external write request, checking whether the number of normal data groups in the data group set cached by the proxy service is sufficient;

[0020] If yes, determining a target data group that is not in the fault list from the data group set;

[0021] If not, exclude the faulty disks in the fault list, and request the control layer to allocate a target data group that is not affected by the error information;

[0022] A proxy write request is initiated to the target data group.

[0023] In one embodiment, the distributed storage system further includes a control layer and a storage layer; after adding the fault object to a fault list and determining the survival time of the fault object in the fault list so that the proxy service performs fault shielding on the fault object when processing data within the survival time, the system further includes:

[0024] In response to an external read request, checking whether a data group containing the to-be-read data corresponding to the external read request is hit in the cache of the proxy service;

[0025] If yes, selecting a target data block that is not in the fault list from the cache;

[0026] If not, request the control layer for designated data group information, add the designated data group information to the cache of the proxy service, and select a target data block that is not in the fault list from the cache;

[0027] A read request for the target data block is initiated to the storage layer.

[0028] In one embodiment, the distributed storage system further includes a control layer and a storage layer; the method further includes:

[0029] If the first fault object in the fault list triggers the fault handling function of the control layer, obtaining a notification of updating or deleting the fault object sent by the control layer; updating or deleting the first fault object in the fault list according to the notification to cooperate with the control layer in fault handling;

[0030] If the second fault object in the fault list does not trigger the fault handling function of the control layer, and the duration of the second fault object in the fault list reaches the survival duration, the second fault object is deleted from the fault list.

[0031] In a second aspect, the present application further provides a fault handling device, which is applied to an access layer of a distributed storage system; the device comprises:

[0032] An error type and error source determination module, used to obtain error information and determine the error type and error source of the error information;

[0033] A fault level determination module, used to determine the fault level of the error information according to the error type;

[0034] a fault object determination module, configured to determine a fault object related to the error information according to the fault level and the error source; the fault object includes at least one of a faulty disk and a fault data set, the fault data set being a data set related to the error information in a cache of a proxy service, the proxy service being configured to complete a function of the access layer;

[0035] The fault shielding module is used to add the fault object to the fault list and determine the survival time of the fault object in the fault list, so that the proxy service shields the fault object when processing data within the survival time.

[0036] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0037] Obtain error information, and determine the error type and error source of the error information;

[0038] Determining a fault level of the error information according to the error type;

[0039] Determine a fault object related to the error information according to the fault level and the error source; the fault object includes at least one of a faulty disk and a fault data set, the fault data set is a data set related to the error information in a cache of a proxy service, the proxy service is used to complete the function of the access layer;

[0040] The fault object is added to a fault list, and the survival time of the fault object in the fault list is determined, so that the proxy service performs fault shielding on the fault object when processing data within the survival time.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0042] Obtain error information, and determine the error type and error source of the error information;

[0043] Determining a fault level of the error information according to the error type;

[0044] Determine a fault object related to the error information according to the fault level and the error source; the fault object includes at least one of a faulty disk and a fault data set, the fault data set is a data set related to the error information in a cache of a proxy service, the proxy service is used to complete the function of the access layer;

[0045] The fault object is added to a fault list, and the survival time of the fault object in the fault list is determined, so that the proxy service performs fault shielding on the fault object when processing data within the survival time.

[0046] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0047] Obtain error information, and determine the error type and error source of the error information;

[0048] Determining a fault level of the error information according to the error type;

[0049] Determine a fault object related to the error information according to the fault level and the error source; the fault object includes at least one of a faulty disk and a fault data set, the fault data set is a data set related to the error information in a cache of a proxy service, the proxy service is used to complete the function of the access layer;

[0050] The fault object is added to a fault list, and the survival time of the fault object in the fault list is determined, so that the proxy service performs fault shielding on the fault object when processing data within the survival time.

[0051] The above-mentioned fault handling method, device, computer equipment, computer-readable storage medium and computer program product are applied to the access layer of a distributed storage system; error information is obtained, and the error type and error source of the error information are determined; the fault level of the error information is determined according to the error type; according to the fault level and the error source, a fault object related to the error information is determined, and the fault object includes at least one of a faulty disk and a fault data set, and the fault data set is a data set related to the error information in the cache of a proxy service, and the proxy service is used to complete the function of the access layer; then, the fault object is added to a fault list, and the survival time of the fault object in the fault list is determined, so that the proxy service performs fault shielding on the fault object when processing data within the survival time, that is, shielding the fault through the access layer of the distributed storage system, the fault impact can be intercepted at the location (access layer) that directly communicates with the outside, reducing the external perception of the distributed storage system (cluster) fault, and controlling the fault impact range within the distributed storage system (cluster) without spreading outward, so that the distributed storage system is not affected by local faults, thereby improving the overall availability of the distributed storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A schematic diagram of a flow chart of a fault handling method in an embodiment;

[0054] Figure 2 is an architecture diagram of a distributed storage system in one embodiment;

[0055] Figure 3 is a schematic diagram of a hierarchical structure of storage layer management in one embodiment;

[0056] Figure 4 A schematic diagram of a process flow of a fault shielding triggering stage in an embodiment;

[0057] Figure 5 A schematic diagram of a flow chart of a fault shielding validation phase of a write request in one embodiment;

[0058] Figure 6 A schematic diagram of a flow chart of a fault masking validation phase of a read request in an embodiment;

[0059] Figure 7 A schematic diagram of a process flow of a fault shielding and recovery phase in an embodiment;

[0060] Figure 8 is a structural block diagram of a fault handling device in an embodiment;

[0061] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0063] In one embodiment, Figure 1As shown, a fault handling method is provided. This embodiment uses the method applied to a computer device as an example for illustration. The computer device can be a terminal or a server. It can be understood that the method can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0064] In this embodiment, the method is applied to the access layer of a distributed storage system; the fault handling method includes the following steps:

[0065] Step S102, obtaining error information, and determining the error type and error source of the error information.

[0066] It is understood that the computer device includes a distributed storage system, such as Figure 2 As shown in the figure, the distributed storage system of the data includes an access layer, a control layer, and a storage layer. The access layer is responsible for receiving external requests, performing load balancing and security control, and forwarding requests to appropriate storage nodes; the control layer manages system metadata, monitors node health status, performs fault detection and recovery, and ensures data consistency; the storage layer actually stores and processes data, performs data replication and read and write operations, and ensures data persistence and availability. In terms of fault handling, the control layer plays a leading role. It has a global perspective to monitor the health status of the entire system, receives abnormal reports from other layers, and can quickly discover and respond to faults.

[0067] The access layer is the entrance to the entire storage cluster (distributed storage system). The service that completes the access layer function is called a proxy service. In the access layer, multiple proxy services can be horizontally expanded according to the number of concurrent external requests and the performance and scale of the entire cluster, and different proxy services run independently. The access layer is used to receive external requests, analyze and process them, and call storage layer services to complete various data operations. In object storage, this external request is the request from the object storage protocol gateway. In order to distinguish between the two types of requests, the requests received by the access layer from the outside are called external requests, and the requests sent by the access layer to the storage layer are called proxy requests. According to the operation type, the requests can also be divided into write requests, read requests, and delete requests. The access layer can contain multiple proxy service processes.

[0068] At the storage layer, the management structure has different implementations, and controlling the scope of the impact of failures is also a problem. For example, in some storage system implementations, a data storage service process only manages one physical disk, while some implementations manage multiple physical disks. Too many processes will increase the complexity of storage system management and maintenance and waste computer system resources; but too many disks managed by a single process will increase the complexity of process processing logic and the risk of the spread of failure impacts. The size division regulations for data blocks on each physical disk are also different, and data blocks with different states may exist on the same disk. All of the above require us to strengthen fault tolerance and redundancy design.

[0069] As the entrance to the entire storage cluster, the access layer directly receives various data operation requests from the object storage protocol gateway in the object storage. The impact of some faults may first be reflected in the processing of a single request, especially when the requests are dense. The access layer can perceive the occurrence of the fault earlier. At this time, even a short jitter may affect a large number of requests. Due to the existence of network delays and computing delays, there will always be delays and time differences in fault handling between different modules across processes or nodes, and the handling of some faults may be delayed. In addition, if the cluster itself is in a high-load state when a fault occurs, retries and other fault handling mechanisms may in turn cause some negative effects. Therefore, strengthening fault identification and shielding methods at the access layer is also beneficial to the stability of the entire cluster.

[0070] Optionally, during the fault mask triggering phase, the access layer (proxy service) collects error information from the error report or response to determine the error type and error source. The error information can be error information of a read request, a write request, or a delete request.

[0071] If the proxy service fails to send a request to the data storage service, the error type is determined based on the error message, and the source of the error is marked as the primary data block of the requested data group and its corresponding disk; if the proxy service successfully sends a request to the data storage service, and an exception occurs during the data storage service processing, the data storage service will carry the error type, error information, and error source in the returned response content.

[0072] The error type is an error type defined by the operating system, or an error type defined within the distributed storage system. The error type includes at least one of service not found, service connection refused, request timeout, disk I / O error, disk I / O submission timeout, data block compression, or other data block internal error.

[0073] The error sources are the data block where the error occurs and the disk where the data block is located.

[0074] Step S104: determining the fault level of the error information according to the error type.

[0075] The fault level can be set to different levels according to the impact range of the fault. The impact range of the fault is positively correlated with the fault level, that is, the larger the impact range of the fault is, the larger the fault level is.

[0076] Optionally, the fault level of the error information is determined according to the error type, including: if the error type indicates that a data block error occurs, then the fault level of the error information is determined to be a data block level failure; if the error type indicates that a disk error occurs, then the fault level of the error information is determined to be a disk level failure; if the error type indicates that a service process error occurs, then the fault level of the error information is determined to be a process level failure.

[0077] like Figure 3 As shown in the figure, the applicable scenario includes a common hierarchical structure of storage layer management: a physical node has at least one data storage service process, a data storage service process manages at least one physical disk, and a physical disk can have multiple data blocks. A data block is the basic unit for storage layer replica management, and a data block can contain multiple data. The combination of data blocks distributed on different nodes or disks that meet certain redundancy rules is called a data group.

[0078] according to Figure 3 Based on the hierarchical structure of the storage layer and the possible impact range of the error, the fault types of error information are divided into the following three levels:

[0079] Level 1: Data block level failure. For example, failures in data block compression or other internal data blocks. For write operations, this type of failure usually only affects the writing of a certain data group. Data blocks are distinguished by data block IDs, which are assigned by the control layer of the distributed storage system (cluster) and need to be unique within the cluster. A data group refers to a collection of replicas that meet certain redundancy rules, a logical grouping of storage objects.

[0080] Level 2: disk-level failure. For example, disk I / O errors, disk I / O submission timeouts, etc. This error usually affects the normal reading and writing of the entire disk. Disks are distinguished by disk IDs. The disk serial number can be used as the disk ID, or the disk ID can be assigned by the cluster. The disk ID is required to be at least unique in the distributed storage system (cluster) to identify a disk in the cluster. Each disk registers a piece of information in the service discovery module, and the registration content should include the disk ID.

[0081] Level 3: Process-level failure. For example, the service cannot be found, the service port refuses to connect, the request times out, etc. The occurrence of such errors often means that the entire data storage service process cannot provide services normally. Processes can be distinguished by socket addresses in service discovery. It is required that all services in the same process share a port, and the address information registered in the service discovery module is the same socket address. A process in the network can be identified by the socket address.

[0082] Optionally, if the fault level of the error information cannot be determined according to the error type, the fault level of the error information is defined according to the impact scope of the error information.

[0083] Step S106, determining the fault object related to the error information according to the fault level and the error source; the fault object includes at least one of a faulty disk and a fault data set, the fault data set is a data set related to the error information in the cache of the proxy service, and the proxy service is used to complete the function of the access layer.

[0084] The fault object is an object affected by the fault, including at least one of a faulty disk and a faulty data set, which is a data set affected by error information in the cache of the proxy service, and can be a faulty data group or a faulty data block.

[0085] Optionally, based on the fault level and the error source, a fault object related to the error information is determined, including: if the fault level is a data block level fault, a first object related to the error information is determined, the first object includes a set of fault data related to the error information and in the cache of the proxy service where the error occurs; if the fault level is a disk level fault, a second object related to the error information is determined, the second object includes the first disk where the error source is located, and a set of fault data related to the error information and associated with the first disk in the cache of the proxy service; if the fault level is a process level fault, a third object related to the error information is determined, the third object includes a second disk under the same data storage service process as the error source, and a set of fault data related to the error information and associated with the second disk in the cache of the proxy service; the first object, the second object and the third object all belong to fault objects.

[0086] For a write request, if the fault level is a data block level fault, the scope of influence is the current data group, so the first object includes the faulty data group in the proxy service cache that is related to the error information and has an error; if the fault level is a disk level fault, the scope of influence is the disk where the error source is located and the data group associated with the disk, so the second object includes the first disk where the error source is located, and the faulty data group in the proxy service cache that is related to the error information and associated with the first disk; if the fault level is a process level fault, the scope of influence is the disks under the same data storage service process as the error source and the data groups associated with these disks, so the third object includes the second disk under the same data storage service process as the error source, and the faulty data group in the proxy service cache that is related to the error information and associated with the second disk. The data group associated with a disk in the above content refers to a data block in the data group on the disk.

[0087] For read requests, if the failure level is a data block level failure, the affected scope is the current data block, so the first object includes the failed data block in the proxy service's cache that is related to the error information and where the error occurs; if the failure level is a disk level failure, the affected scope is the disk where the error source is located and the data blocks distributed on the disk, so the second object includes the first disk where the error source is located, and the failed data blocks in the proxy service's cache that are related to the error information and associated with the first disk; if the failure level is a process level failure, the affected scope is the disk under the same data storage service process as the error source and the data blocks distributed on the disk, so the third object includes the second disk under the same data storage service process as the error source, and the failed data blocks in the proxy service's cache that are related to the error information and associated with the second disk.

[0088] It is understandable that the characteristics of a write request are: writing is a change operation, which needs to be successfully written on all data blocks in a data group, but it does not require a specific data group. If writing cannot be successfully completed on a data group, other data groups can be replaced. The proxy service will cache the data group information set used for write requests in memory. The characteristics of a read request are: the data group where the data to be read is located is clear, but because reading is a non-changing operation and the data group is redundant, reading a certain number of data blocks that meet the redundancy requirements is considered a read. The proxy service will cache the data group information that has been recently read and accessed.

[0089] Therefore, it is particularly important to emphasize that what is sought for write requests is the data group, and what is sought for read requests is the data block. Due to the existence of redundant design, if only one data block in the data group is faulty, it will affect the writing of the entire data group, but will not affect the reading of the entire data group.

[0090] It is understandable that different fault levels are classified according to the source and type of errors, which makes it easier to estimate the importance and impact of the errors, promptly confirm which nodes, disks or data may be affected, and block access to these potentially affected content for a period of time. This can avoid repeated access and retries to nodes with known serious faults, reduce unnecessary network and computing overhead, improve the overall performance and efficiency of the system, and provide better conditions for cluster fault recovery.

[0091] Step S108, adding the fault object to the fault list, and determining the survival time of the fault object in the fault list, so that the proxy service performs fault shielding on the fault object when processing data within the survival time.

[0092] The fault list is a list used for fault shielding, and the fault list indicates that the fault objects added to the fault list are in a shielded state. The fault list is also called a blacklist.

[0093] The survival time is the length of time that the fault object is in the fault list. During this survival time, the fault object is in a fault shielding state and cannot be processed. The survival time can be set as needed. If the error message is a read request, the fault list is the read request fault list; if the error message is a write request, the fault list is the write request fault list; if the error message is a delete request, the fault list is the delete request fault list.

[0094] Optionally, the access layer (proxy service) adds the fault object to the fault list in the fault shielding module, and configures the survival time of the fault object in the fault list.

[0095] The survival time can be set as an unchangeable value by the program according to different error types, but it is recommended to set it as a changeable configuration item according to common error types; the order of magnitude is recommended to be set to seconds. That is, according to different error types, a constant is directly used in the program code as the survival time of the fault object in the fault list. For example, the code stipulates that the survival time of the disk and data group in the fault list is 1 second when receiving the error "disk I / O busy". The survival time set by this implementation method cannot be changed online once compiled unless the code is modified. It is recommended to make "the survival time of the disk and data group in the blacklist under different error types" as a configuration item so that it can be changed online without modifying the code. For example, the survival time corresponding to errors such as "disk I / O submission timeout", "disk I / O busy", and "service port refused to connect" are respectively used as separate configuration items, and the configuration value is determined by reading the configuration file.

[0096] Optionally, add a fault mark to the faulty object in the cache or delete it directly.

[0097] Optionally, for write requests, the access layer deletes the faulty data groups affected by the fault from the data group set cache used by the proxy service for write allocation, or marks the faulty data groups affected by the fault as faults in the data group set cache used by the proxy service for write allocation; if some faulty data groups or faulty disks are already in the fault list, these faulty data groups or faulty disks are skipped.

[0098] For read requests, the access layer also marks the faulty data blocks affected by the fault in the data group set cache used by the proxy service for write allocation; if some faulty data blocks or faulty disks are already in the fault list, these data blocks or disks are skipped. Since the failure of a single data block in a read request will not affect the read operation of the entire data group, the granularity for fault shielding is not the data group but the data block. In addition, instead of directly deleting the data block mapping information of the data group in the cache, a fault mark can be added to the faulty data block.

[0099] Optionally, set alarm information according to the fault masking content to provide reference for operation and maintenance personnel.

[0100] In one implementation, an alarm is set according to the fault level. For example, a first-level alarm is issued when a process-level fault occurs, a second-level medium or low-level alarm is issued when a disk-level fault occurs, and no alarm is issued when a data block-level fault occurs, and only an error log is printed in the log. The first-level alarm is a high-level alarm, and the second-level alarm is a medium or low-level alarm.

[0101] In another embodiment: alarm information is set according to the number of fault objects in the fault list. For example, when the number of fault objects in the fault list of a single proxy service exceeds a set threshold, an alarm information is sent to remind the operation and maintenance personnel to check the health status of the cluster, or to set some automatic processing procedures accordingly.

[0102] The above fault handling method is applied to the access layer of a distributed storage system; error information is obtained, and the error type and error source of the error information are determined; the fault level of the error information is determined according to the error type; according to the fault level and the error source, a fault object related to the error information is determined, the fault object includes at least one of a faulty disk and a fault data set, the fault data set is a data set related to the error information in the cache of a proxy service, and the proxy service is used to complete the function of the access layer; then, the fault object is added to a fault list, and the survival time of the fault object in the fault list is determined, so that the proxy service performs fault shielding on the fault object when processing data within the survival time, that is, shielding the fault by the access layer of the distributed storage system, the fault impact can be intercepted at the location (access layer) that directly communicates with the outside, reducing the external perception of the distributed storage system (cluster) fault, and controlling the fault impact range within the distributed storage system (cluster) so that it does not spread outward, so that the distributed storage system is not affected by local faults, thereby improving the overall availability of the distributed storage system.

[0103] Applied to the access layer of the distributed storage system, the access layer can sense the occurrence of faults earlier and intercept the impact of faults at the location that directly communicates with the outside as much as possible, reducing the external perception of cluster faults and the impact on front-end business, and controlling the scope of fault impact within the cluster without spreading outward, ensuring the overall availability of the system is not affected by local faults, thus playing a gatekeeper role.

[0104] The access module at the access layer can sense the occurrence of faults earlier, especially when requests are dense. Due to network delays and computing delays, there will always be delays and time differences in fault handling between different modules across processes or nodes, and if the cluster itself is under high load when a fault occurs, retries and other fault handling mechanisms may in turn cause other negative effects. Therefore, the above fault handling method can timely, accurately and effectively determine the scope of the fault and shield the fault at the access layer of the distributed storage system (cluster) and at the source of the request, which can effectively cool down the cluster and make it more stable.

[0105] By shielding faults in a hierarchical manner at the access layer, on the one hand, the impact of faults can be intercepted as much as possible at the location that directly communicates with the outside world, reducing the external perception of cluster faults, and controlling the scope of fault impact within the cluster so that it does not spread outward, ensuring that the overall availability of the system is not affected by local faults. On the other hand, it can avoid repeated access and retries to nodes with known serious faults, reduce unnecessary network and computing overhead, improve the overall performance and efficiency of the system, and provide better conditions for cluster fault recovery. For example, when the disk I / O load is too high, it is necessary to reduce the I / O requests on the disk. At this time, hierarchical processing can be used to temporarily isolate the disk during the write operation, which can effectively reduce the load on the disk. In addition, different early warning and alarm mechanisms can be set for faults of different levels in combination with the monitoring system to facilitate timely discovery, location, and handling of problems.

[0106] In addition, hierarchical shielding is performed for different types of faults in the distributed storage system (cluster), and unnecessary retry operations are avoided through the fault isolation mechanism to effectively control the impact range of the fault and reduce the request and processing pressure at the source of the fault. In order to ensure the low coupling between the modules of the entire cluster, it is built on the basis of the existing basic fault handling logic of the distributed storage system, but does not directly participate in the execution or replace the original fault handling function of the control layer, and is only provided as a strengthening means. The location of the main logic of the above fault handling method is limited to the inside of a single access layer proxy service process, and the operations of different access layer proxy service processes do not affect each other and the data is not shared.

[0107] In one embodiment, Figure 4 As shown, for the fault shielding triggering stage of a read request or a write request, the fault shielding triggering stage includes the following steps:

[0108] Step S402: The proxy service collects error information from the error report or response, and determines the error type and error source.

[0109] Step S404: The proxy service determines the fault level of the error according to predefined rules.

[0110] Step S406: The proxy service finds out a list of affected disks in the cluster and a list of affected data groups in the proxy service cache according to the error level and the error source.

[0111] Step S408, after the proxy service finds the affected disk list and data group list, it adds these disks and data groups to the write request blacklist in the fault shielding module, and specifies a survival time in the blacklist, which is used to eliminate faulty data groups or data blocks in the cache by adding fault marks or deleting them from the cache information summary.

[0112] In one embodiment, the distributed storage system also includes a control layer; adding the faulty object to a fault list and determining the survival time of the faulty object in the fault list, so that the proxy service performs fault shielding on the faulty object when processing data within the survival time, and further includes: responding to an external write request, checking whether the number of normal data groups in the data group set cached by the proxy service is sufficient; if so, determining the target data group that is not in the fault list from the data group set; if not, excluding the faulty disks in the fault list, and requesting the control layer to allocate a target data group that is not affected by the error information; and initiating a proxy write request to the target data group.

[0113] The fault shielding effectiveness phase for write requests is mainly processed in the basic module of the proxy service, which occurs after the fault shielding is triggered and before recovery, and when processing new write requests during this period. When the access layer receives a new external write request, it verifies the content of the external write request; if the verification passes, it executes the step of checking whether the number of normal data groups in the data group set cached by the proxy service is sufficient.

[0114] Among them, verifying the content of the external write request refers to verifying the legitimacy of the request body parameters of the external write request, such as checking the request parameters, confirming whether the storage pool requested to be written exists, whether the data to be written is empty, whether the size of the data to be written exceeds the maximum limit, etc.

[0115] If the verification fails, the request is considered failed and an error is returned, such as "parameter error".

[0116] If the number of normal data groups in the data group set cached by the proxy service is sufficient, data groups not affected by the failure are selected from the cached data group set for writing, ensuring that during this period, the data groups in the blacklist will not receive new external write requests.

[0117] If the number of normal data groups in the data group set cached by the proxy service is insufficient, the current faulty disk is obtained from the fault shielding module for exclusion, and a certain number of target data groups not affected by the fault are requested to be allocated to the control layer for writing. This ensures that during this period, the disks in the fault list will not receive new external write requests.

[0118] In this embodiment, in response to an external write request, it is checked whether the number of normal data groups in the data group set cached by the proxy service is sufficient. If so, the target data group that is not in the fault list is determined from the data group set. If not, the faulty disk in the fault list is excluded, and the control layer is requested to allocate a target data group that is not affected by the error information. Finally, a proxy write request is initiated to the target data group, which can achieve the effect of avoiding initiating a write request to the area with the fault. Different fault shielding processing flows are set according to the type of error to control the impact range of the fault so that various requests can be as successful as possible when a fault occurs.

[0119] In one embodiment, Figure 5 As shown, for the fault shielding validation phase of a write request, the fault shielding validation phase includes the following steps:

[0120] Step S502: The proxy service verifies the request content of the newly received external write request.

[0121] In step S504, the proxy service checks whether the number of data groups allocated for writing is sufficient, and if so, executes step S506; otherwise, executes step S508.

[0122] Step S506 , directly selecting a data group not affected by the failure from the cached data group set for writing.

[0123] Step S508, obtaining a current faulty disk list from the fault masking module for exclusion, and requesting the control layer to allocate a certain number of data groups not affected by the fault for writing.

[0124] Step S510: The proxy service initiates a proxy write request to the selected data group.

[0125] In one embodiment, the distributed storage system also includes a control layer and a storage layer; adding the faulty object to a fault list and determining the survival time of the faulty object in the fault list, so that the proxy service performs fault shielding on the faulty object when processing data within the survival time, and further includes: responding to an external read request, checking whether the data group where the to-be-read data corresponding to the external read request is located is hit in the cache of the proxy service; if so, selecting a target data block that is not in the fault list from the cache; if not, requesting the control layer for specified data group information, adding the specified data group information to the cache of the proxy service, and selecting a target data block that is not in the fault list from the cache; initiating a read request for the target data block to the storage layer.

[0126] During the fault shielding phase for read requests, when the access layer receives a new external read request, it verifies the content of the external read request; if the verification passes, it executes the step of checking whether the data group containing the to-be-read data corresponding to the external read request hits the cache of the proxy service.

[0127] Verifying the content of the external read request refers to verifying the validity of the request body parameters of the external read request, such as checking the request parameters, confirming whether the storage pool requested to be read exists, whether the data to be read is empty, etc.

[0128] If the verification fails, the request is considered failed and an error is returned, such as "parameter error".

[0129] Optionally, if the data group containing the data to be read corresponding to the external read request does not hit in the cache of the proxy service, the access layer requests the control layer to specify the data group information, adds the specified data group information to the cache of the proxy service, and compares the specified data group information with the fault disk in the fault list of the fault shielding module. If the specified data group information contains data blocks in the fault list, a fault mark is added to the faulty data block in the specified data group information.

[0130] For data groups with multiple replica redundancy rules, you only need to directly initiate a read request to a normal data block.

[0131] For data groups that use erasure code redundancy rules, a read request needs to be initiated to a certain data block, and the data block first reads a certain number of other data blocks in the storage layer and then completes the erasure code calculation to restore the data. In this case, the recommended normal data block list can be carried in the read request to avoid requesting faulty data blocks when the storage layer performs erasure code reading. However, the carried normal data block list is only used as a reference for the actual reading of the storage layer. If the number of required read copies is insufficient, other data blocks can be requested regardless of the shielding to ensure that the original reading effect of the system is not reduced.

[0132] If there are too many data blocks marked as faulty and no suitable data block can be found to initiate a read request, one optional implementation is to randomly select a data block to initiate a read request, and another optional implementation is to select the data block that was removed from the blacklist the earliest to initiate a read request. The access layer's judgment on the scope and time of the fault impact is an estimate, which may differ from the actual situation. Therefore, in this extreme case, various possibilities should be tried, and the relevant logic is designed based on the principle of not reducing the effect of the original read logic.

[0133] Among them, erasure coding: Erasure coding is a data protection technology used to provide data redundancy and fault tolerance in storage systems. It splits data into multiple data blocks and generates additional check blocks, so that even if some data blocks are lost or damaged, the original data can still be restored through the remaining data blocks and check blocks. Erasure coding is widely used in distributed storage systems, cloud storage, and big data storage to improve data reliability and storage efficiency. Compared with the redundant mode of multiple copies, the obvious advantage of erasure coding is that it has higher storage utilization.

[0134] In this embodiment, in response to an external read request, check whether the data group to be read corresponding to the external read request is hit in the cache of the proxy service; if so, select the target data block that is not in the fault list from the cache; if not, request the control layer for the specified data group information, add the specified data group information to the cache of the proxy service, and select the target data block that is not in the fault list from the cache; initiate a read request for the target data block to the storage layer, thereby achieving the effect of avoiding initiating a read request to the fault area as much as possible. Set different fault shielding processing flows according to the type of error, control the impact range of the fault, and make various requests as successful as possible when a fault occurs.

[0135] In one embodiment, Figure 6 As shown, for the fault shielding validation stage of the read request, the fault shielding validation stage includes the following steps:

[0136] Step S602: The proxy service verifies the request content of the newly received external read request.

[0137] Step S604, checking whether the data group information of the data to be read is hit in the cache, if yes, executing step S608, if no, executing step S606.

[0138] Step S606, the proxy service requests the control layer for information about a specified data group, adds the information to the cache, and compares it with the faulty disks in the blacklist of the fault shielding module, and if necessary, adds fault marks to the data blocks on these disks.

[0139] Step S608: Select a normal data block that is not marked as a fault.

[0140] Step S610: The proxy service initiates a read request for the selected data block to the storage layer.

[0141] In one embodiment, the distributed storage system also includes a control layer and a storage layer; the above method also includes: if the first fault object in the fault list triggers the fault handling function of the control layer, then obtaining a notification of updating or deleting the fault object sent by the control layer; according to the notification, updating or deleting the first fault object in the fault list to cooperate with the control layer in fault handling; if the second fault object in the fault list does not trigger the fault handling function of the control layer, and the length of time the second fault object has been in the fault list reaches the survival time, then deleting the second fault object from the fault list.

[0142] The fault masking recovery phase for write requests is mainly handled by the fault masking module of the access layer (proxy service), which occurs after the control layer triggers the fault handling function or after the lifetime of the faulty object in the target list expires.

[0143] If the fault triggers the fault handling function of the control layer, the control layer will send a notification to the access layer to update or delete the fault data group. After receiving the notification, the access layer (proxy service) updates or deletes the first fault object in the cache and fault shielding module according to the notification, and cooperates with the control layer to handle the fault. Optionally, the access layer removes the fault data set (data group or data block) that has been updated in the target list of the fault shielding module.

[0144] Optionally, the fault shielding module checks in real time whether the second fault object in the fault list has reached the corresponding survival time. If the second fault object does not receive an update or deletion notification from the control layer until expiration, the fault shielding module deletes the second fault object from the fault list.

[0145] Optionally, if the target data set (data group or data block) information in the cache is deleted at the same time when the fault is added to the fault list, the target data set is restored to its original position.

[0146] Optionally, if a fault mark exists for the first fault object or the second fault object, the fault mark is deleted.

[0147] For write requests, in actual practice, if a disk I / O submission timeout error occurs, the common reason is that the current disk's I / O load is too high. The survival time for this error can be configured to a shorter time, such as 1 second, so that the current proxy service avoids writing to it during this period and writes to other disks instead, which can reduce the I / O load of the problem disk to a certain extent. If no abnormalities continue to occur after writing after the survival time expires, it can be considered that the disk has returned to normal and no additional processing is required; if there are still abnormalities, add it to the fault list again to continue to avoid writing to it to reduce the load, and repeat this operation until the disk returns to normal or triggers the fault processing of the control layer.

[0148] Many service discovery modules use leases to determine whether a service exists. In this case, the error of refusing a connection to a service port can be configured as the lease expiration time, so that the access layer can avoid sending unsuccessful requests to the data storage service during the time difference between the service disconnection and the lease expiration. After the lease actually expires, the control layer will normally notify the access layer that the relevant data group is no longer writable; and if it is just a short-term network failure, then after a short period of time after being removed from the blacklist, the proxy service will resume writing to this data storage service.

[0149] Adding this fault shielding processing at the access layer can effectively alleviate the jitter of the distributed storage system (cluster). When a distributed storage system (cluster) fails but can automatically recover to a normal state in a short period of time through a certain method, it can reduce the external perception of this recovery phase and effectively improve the stability of the cluster.

[0150] In one embodiment, Figure 7 As shown, for the fault shielding recovery phase of a read request or a write request, the fault shielding recovery phase includes the following steps:

[0151] Step S702: The fault triggers the fault handling function of the control layer, and the control layer sends a notification to the access layer to update the fault data group information or delete the fault data group.

[0152] Step S704: After receiving the above notification, the proxy module updates the fault data group information in the cache and fault shielding module or deletes the fault data group according to the notification, and cooperates with the control layer to handle the fault. The data group or data block that has been updated in this step in the blacklist of the fault shielding module is removed.

[0153] Step S706, after the lifetime of some disks, data blocks or data groups with faults in the fault masking module in the blacklist expires, these disks, data blocks or data groups are removed from the blacklist, and normal information of these data blocks or data groups is restored in the cache.

[0154] In another embodiment, fault shielding of a delete request is described. The characteristics of a delete request are that it needs to operate on a specific data group and eventually succeed on all data blocks. This means that the delete request needs to succeed as much as possible on a specific data block, and the access layer's judgment on the fault range is a kind of prediction. Therefore, the method of bypassing specific data blocks through a shielding mechanism is no longer applicable.

[0155] Optionally, the deletion request is not blocked at the access layer, and the control layer is relied upon to handle the fault.

[0156] In another embodiment, another fault handling method is provided, which is applied to the access layer of a distributed storage system, wherein the distributed storage system further includes a control layer and a storage layer. The fault handling method includes three stages: fault shielding triggering, fault shielding taking effect, and fault shielding recovery. Different request types need to be processed separately; the fault shielding triggering stage includes steps A1 to A4, the fault shielding taking effect stage includes steps A5 to A6, and the fault shielding recovery stage includes step A7:

[0157] Step A1, obtaining error information, and determining the error type and error source of the error information; the error information includes error information of a read request or error information of a write request;

[0158] Step A2: if the error type indicates that a data block error occurs, then the fault level of the error information is determined to be a data block level fault; if the error type indicates that a disk error occurs, then the fault level of the error information is determined to be a disk level fault; if the error type indicates that a service process error occurs, then the fault level of the error information is determined to be a process level fault;

[0159] Step A3, if the fault level is a data block level fault, then determine a first object related to the error information, the first object includes a fault data set related to the error information and having an error in the cache of the proxy service; if the fault level is a disk level fault, then determine a second object related to the error information, the second object includes a first disk where the error source is located, and a fault data set related to the error information and associated with the first disk in the cache of the proxy service; if the fault level is a process level fault, then determine a third object related to the error information, the third object includes a second disk under the same data storage service process as the error source, and a fault data set related to the error information and associated with the second disk in the cache of the proxy service; the first object, the second object and the third object all belong to fault objects; the fault object includes at least one of a faulty disk and a fault data set, the fault data set is a data set related to the error information in the cache of the proxy service, and the proxy service is used to complete the function of the access layer;

[0160] Step A4, adding the fault object to the fault list, and determining the survival time of the fault object in the fault list, so that the proxy service performs fault shielding on the fault object when processing data within the survival time;

[0161] In response to an external write request, executing step A5; in response to an external read request, executing step A6;

[0162] Step A5, check whether the number of normal data groups in the data group set cached by the proxy service is sufficient; if so, determine the target data group that is not in the fault list from the data group set; if not, exclude the faulty disk in the fault list, and request the control layer to allocate the target data group that is not affected by the error information; initiate a proxy write request to the target data group;

[0163] Step A6, check whether the data group to be read corresponding to the external read request is hit in the cache of the proxy service; if so, select the target data block that is not in the fault list from the cache; if not, request the control layer for the specified data group information, add the specified data group information to the cache of the proxy service, and select the target data block that is not in the fault list from the cache; initiate a read request for the target data block to the storage layer;

[0164] Step A7, if the first fault object in the fault list triggers the fault handling function of the control layer, obtain the notification of updating or deleting the fault object sent by the control layer; according to the notification, update or delete the first fault object in the fault list to cooperate with the control layer in fault handling; if the second fault object in the fault list does not trigger the fault handling function of the control layer, and the length of time the second fault object has been in the fault list reaches the survival time, delete the second fault object from the fault list.

[0165] Optionally, due to the higher fault tolerance of read requests, a limited number of retries can be tolerated in order to simplify the read request process, and fault shielding processing may not be performed on read requests during implementation. No additional fault shielding operations are added to the access layer for read requests, and the current request is replaced with other data blocks for reading. In this case, simple read logic is preferred to tolerate the overhead caused by retries. Since the data group distribution is designed to be as dispersed as possible, as long as the cluster is still in a normal and available state and there are enough replicas, the read will be successful within the number of retry steps that is not higher than the number of replicas.

[0166] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0167] Based on the same inventive concept, the embodiment of the present application also provides a fault handling device for implementing the fault handling method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more fault handling device embodiments provided below can refer to the limitations on the fault handling method above, and will not be repeated here.

[0168] In an exemplary embodiment, Figure 8 As shown, a fault handling device is provided, including: an error type and error source determination module 802, a fault level determination module 804, a fault object determination module 806 and a fault shielding module 808, wherein:

[0169] The error type and error source determination module 802 is used to obtain error information and determine the error type and error source of the error information;

[0170] A fault level determination module 804 is used to determine the fault level of the error information according to the error type;

[0171] A fault object determination module 806 is used to determine a fault object related to the error information according to the fault level and the error source; the fault object includes at least one of a faulty disk and a fault data set, the fault data set is a data set related to the error information in the cache of a proxy service, and the proxy service is used to complete the function of the access layer;

[0172] The fault shielding module 808 is used to add the fault object to the fault list and determine the survival time of the fault object in the fault list, so that the proxy service shields the fault object when processing data within the survival time.

[0173] The above-mentioned fault handling device is applied to the access layer of a distributed storage system; it obtains error information and determines the error type and error source of the error information; it determines the fault level of the error information according to the error type; it determines the fault object related to the error information according to the fault level and the error source, and the fault object includes at least one of a faulty disk and a fault data set, and the fault data set is a data set related to the error information in the cache of a proxy service, and the proxy service is used to complete the function of the access layer; then, the fault object is added to the fault list, and the survival time of the fault object in the fault list is determined, so that the proxy service performs fault shielding on the fault object when processing data within the survival time, that is, shielding the fault by the access layer of the distributed storage system, the fault impact can be intercepted at the location (access layer) that directly communicates with the outside, reducing the external perception of the distributed storage system (cluster) fault, and controlling the fault impact range within the distributed storage system (cluster) without spreading outward, so that the distributed storage system is not affected by local faults, thereby improving the overall availability of the distributed storage system.

[0174] In one embodiment, the above-mentioned fault level determination module 804 is also used to determine that the fault level of the error information is a data block level fault if the error type indicates that an error occurs in the data block; if the error type indicates that an error occurs in the disk, then the fault level of the error information is determined to be a disk level fault; if the error type indicates that an error occurs in the service process, then the fault level of the error information is determined to be a process level fault.

[0175] In one embodiment, the above-mentioned fault object determination module 806 is also used to determine a first object related to the error information if the fault level is a data block level fault, the first object including a set of fault data related to the error information and in which an error occurs in the cache of the proxy service; if the fault level is a disk level fault, determine a second object related to the error information, the second object including a first disk where the error source is located, and a set of fault data related to the error information and associated with the first disk in the cache of the proxy service; if the fault level is a process level fault, determine a third object related to the error information, the third object including a second disk under the same data storage service process as the error source, and a set of fault data related to the error information and associated with the second disk in the cache of the proxy service; the first object, the second object and the third object all belong to fault objects.

[0176] In one embodiment, the distributed storage system also includes a control layer; the above-mentioned device also includes a fault masking validation module, which is used to respond to an external write request and check whether the number of normal data groups in the data group set cached by the proxy service is sufficient; if so, determine the target data group that is not in the fault list from the data group set; if not, exclude the faulty disks in the fault list, and request the control layer to allocate a target data group that is not affected by the error information; and initiate a proxy write request to the target data group.

[0177] In one embodiment, the distributed storage system also includes a control layer and a storage layer; the above-mentioned fault masking validation module is also used to respond to an external read request, check whether the data group where the data to be read corresponding to the external read request is located is hit in the cache of the proxy service; if so, select a target data block that is not in the fault list from the cache; if not, request the control layer for specified data group information, add the specified data group information to the cache of the proxy service, and select a target data block that is not in the fault list from the cache; and initiate a read request for the target data block to the storage layer.

[0178] In one embodiment, the distributed storage system also includes a control layer and a storage layer; the above-mentioned device also includes a fault masking recovery module, which is used to obtain a notification of updating or deleting a fault object sent by the control layer if the first fault object in the fault list triggers the fault handling function of the control layer; according to the notification, the first fault object in the fault list is updated or deleted to cooperate with the control layer to perform fault handling; if the second fault object in the fault list does not trigger the fault handling function of the control layer, and the duration of the second fault object in the fault list reaches the survival time, the second fault object is deleted from the fault list.

[0179] Each module in the above fault handling device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0180] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a fault handling method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0181] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0182] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0183] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0184] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0185] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0186] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0187] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0188] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A fault handling method, characterized in that: Applied to the access layer of a distributed storage system; the method comprises: Obtain error information, and determine the error type and error source of the error information; Determining a fault level of the error information according to the error type; Determine a fault object related to the error information according to the fault level and the error source; the fault object includes at least one of a faulty disk and a fault data set, the fault data set is a data set related to the error information in a cache of a proxy service, the proxy service is used to complete the function of the access layer; The fault object is added to a fault list, and the survival time of the fault object in the fault list is determined, so that the proxy service performs fault shielding on the fault object when processing data within the survival time.

2. The method according to claim 1, characterized in that The determining the fault level of the error information according to the error type includes: If the error type indicates that an error occurs in the data block, determining that the fault level of the error information is a data block level fault; If the error type indicates that a disk error occurs, determining that the fault level of the error information is a disk level fault; If the error type indicates that an error occurs in the service process, it is determined that the fault level of the error information is a process level fault.

3. The method according to claim 2, characterized in that The determining, according to the fault level and the error source, a fault object related to the error information includes: If the fault level is a data block level fault, determining a first object related to the error information, the first object comprising a set of fault data related to the error information and having an error in the cache of the proxy service; If the fault level is a disk-level fault, determining a second object related to the error information, the second object including a first disk where the error source is located, and a set of fault data related to the error information and associated with the first disk in the cache of the proxy service; If the fault level is a process-level fault, a third object related to the error information is determined, and the third object includes a second disk under the same data storage service process as the error source, and a set of fault data related to the error information and associated with the second disk in the cache of the proxy service; the first object, the second object and the third object all belong to fault objects.

4. The method according to claim 1, characterized in that: The distributed storage system further includes a control layer; after adding the fault object to a fault list and determining the survival time of the fault object in the fault list so that the proxy service performs fault shielding on the fault object when processing data within the survival time, the system further includes: In response to an external write request, checking whether the number of normal data groups in the data group set cached by the proxy service is sufficient; If yes, determining a target data group that is not in the fault list from the data group set; If not, exclude the faulty disks in the fault list, and request the control layer to allocate a target data group that is not affected by the error information; A proxy write request is initiated to the target data group.

5. The method according to claim 1, characterized in that The distributed storage system further includes a control layer and a storage layer; after adding the fault object to a fault list and determining the survival time of the fault object in the fault list so that the proxy service performs fault shielding on the fault object when processing data within the survival time, the system further includes: In response to an external read request, checking whether a data group containing the to-be-read data corresponding to the external read request is hit in the cache of the proxy service; If yes, selecting a target data block that is not in the fault list from the cache; If not, request the control layer for designated data group information, add the designated data group information to the cache of the proxy service, and select a target data block that is not in the fault list from the cache; A read request for the target data block is initiated to the storage layer.

6. The method according to any one of claims 1 to 5, characterized in that: The distributed storage system further includes a control layer and a storage layer; the method further includes: If the first fault object in the fault list triggers the fault handling function of the control layer, obtaining a notification of updating or deleting the fault object sent by the control layer; updating or deleting the first fault object in the fault list according to the notification to cooperate with the control layer in fault handling; If the second fault object in the fault list does not trigger the fault handling function of the control layer, and the duration of the second fault object in the fault list reaches the survival duration, the second fault object is deleted from the fault list.

7. A fault handling device, characterized in that: Applied to the access layer of a distributed storage system; the device comprises: An error type and error source determination module, used to obtain error information and determine the error type and error source of the error information; A fault level determination module, used to determine the fault level of the error information according to the error type; a fault object determination module, configured to determine a fault object related to the error information according to the fault level and the error source; the fault object includes at least one of a faulty disk and a fault data set, the fault data set being a data set related to the error information in a cache of a proxy service, the proxy service being configured to complete a function of the access layer; The fault shielding module is used to add the fault object to the fault list and determine the survival time of the fault object in the fault list, so that the proxy service shields the fault object when processing data within the survival time.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images