Fault processing method and device, computer device, readable storage medium and program product

Abstract

The application relates to a fault processing method and device, computer equipment, a readable storage medium and a program product, and belongs to the field of information technology. The method is applied to an access layer of a distributed storage system; the method comprises the following steps: acquiring error information, and determining an error type and an error source of the error information; determining a fault level of the error information according to the error type; determining a fault object related to the error information according to the fault level and the error source; the fault object comprises at least one of a fault disk and a fault data set, and the fault data set is a data set related to the error information in a cache of a proxy service, and the proxy service is used for completing the functions of the access layer; the fault object is added into a fault list, and a survival time length of the fault object in the fault list is determined, so that the proxy service performs fault shielding on the fault object when performing data processing within the survival time length. The method can improve the availability of the distributed storage system.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular to a fault handling method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Ensuring high availability of distributed storage systems has always been a complex and critical task, enabling the system to maintain stable operation and data accessibility even in the face of various unforeseen circumstances such as hardware failures, software errors, and network problems. To this end, traditional distributed storage systems employ various technical means, such as data redundancy, multi-replica storage, cross-region replication, and automatic fault recovery.

[0003] However, traditional fault handling methods still result in low availability of the distributed storage system after fault handling. Summary of the Invention

[0004] Therefore, it is necessary to provide a fault handling method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the availability of distributed storage systems in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a fault handling method applied to the access layer of a distributed storage system; the method includes:

[0006] Obtain error information and determine the error type and source of the error information;

[0007] Determine the fault level of the error message based on the error type;

[0008] Based on the fault level and the error source, a fault object related to the error information is determined; the fault object includes at least one of a faulty disk and a faulty data set, the faulty data set being a data set related to the error information in the cache of the proxy service, the proxy service being used to perform the functions of the access layer;

[0009] The faulty object is added to the fault list, and the lifespan of the faulty object in the fault list is determined, so that the proxy service can shield the faulty object when processing data within the lifespan.

[0010] In one embodiment, determining the fault level of the error information based on the error type includes:

[0011] If the error type indicates that an error has occurred in a data block, then the fault level of the error message is determined to be a data block-level fault.

[0012] If the error type indicates that a disk error has occurred, then the fault level of the error message is determined to be a disk-level fault.

[0013] If the error type indicates that an error has occurred in the service process, then the fault level of the error message is determined to be a process-level fault.

[0014] In one embodiment, determining the fault object related to the error information based on the fault level and the error source includes:

[0015] If the fault level is a data block level fault, then a first object related to the error information is determined, the first object including a set of fault data in the cache of the proxy service that are related to the error information and have caused the error;

[0016] If the fault level is a disk-level fault, then a second object related to the error information is determined. The second object includes the first disk where the error source is located, and a set of fault data in the cache of the proxy service that is related to the error information and associated with the first disk.

[0017] If the fault level is a process-level fault, then a third object related to the error information is determined. The third object includes a second disk under the same data storage service process as the source of the error, and a set of fault data in the cache of the proxy service that is related to the error information and associated with the second disk. The first object, the second object, and the third object are all fault objects.

[0018] In one embodiment, the distributed storage system further includes a control layer; after adding the faulty object to the fault list and determining the lifetime of the faulty object in the fault list, so that the proxy service can shield the faulty object from failure when processing data within the lifetime, the system further includes:

[0019] In response to an external write request, check whether the number of normal data groups in the data group set cached by the proxy service is sufficient;

[0020] If so, then determine the target data group that is not in the fault list from the data group set;

[0021] If not, then exclude the faulty disks from the fault list and request the control layer to allocate a target data group that is not affected by the error message;

[0022] Initiate a proxy write request to the target data group.

[0023] In one embodiment, the distributed storage system further includes a control layer and a storage layer; after adding the faulty object to the fault list and determining the lifetime of the faulty object in the fault list, so that the proxy service can shield the faulty object from failure when processing data within the lifetime, the system further includes:

[0024] In response to an external read request, check whether the data group containing the data to be read corresponding to the external read request is found in the cache of the proxy service;

[0025] If so, select the target data block that is not in the fault list from the cache;

[0026] If not, request the specified data group information from the control layer, add the specified data group information to the cache of the proxy service, and select the target data block that is not in the fault list from the cache;

[0027] Initiate a read request for the target data block to the storage layer.

[0028] In one embodiment, the distributed storage system further includes a control layer and a storage layer; the method further includes:

[0029] If the first fault object in the fault list triggers the fault handling function of the control layer, then a notification to update or delete the fault object sent by the control layer is obtained; according to the notification, the first fault object in the fault list is updated or deleted to cooperate with the control layer in fault handling;

[0030] If the second fault object in the fault list does not trigger the fault handling function of the control layer, and the duration of the second fault object in the fault list reaches the survival time, then the second fault object will be deleted from the fault list.

[0031] Secondly, this application also provides a fault handling device applied to the access layer of a distributed storage system; the device includes:

[0032] An error type and error source determination module is used to acquire error information and determine the error type and error source of the error information;

[0033] A fault level determination module is used to determine the fault level of the error information based on the error type.

[0034] The fault object determination module is used to determine the fault object related to the error information based on the fault level and the error source; the fault object includes at least one of a faulty disk and a faulty data set, the faulty data set is a data set related to the error information in the cache of the proxy service, and the proxy service is used to perform the functions of the access layer;

[0035] The fault masking module is used to add the faulty object to the fault list and determine the lifespan of the faulty object in the fault list, so that the proxy service can mask the faulty object when processing data within the lifespan.

[0036] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0037] Obtain error information and determine the error type and source of the error information;

[0038] Determine the fault level of the error message based on the error type;

[0039] Based on the fault level and the error source, a fault object related to the error information is determined; the fault object includes at least one of a faulty disk and a faulty data set, the faulty data set being a data set related to the error information in the cache of the proxy service, the proxy service being used to perform the functions of the access layer;

[0040] The faulty object is added to the fault list, and the lifespan of the faulty object in the fault list is determined, so that the proxy service can shield the faulty object when processing data within the lifespan.

[0041] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0042] Obtain error information and determine the error type and source of the error information;

[0043] Determine the fault level of the error message based on the error type;

[0044] Based on the fault level and the error source, a fault object related to the error information is determined; the fault object includes at least one of a faulty disk and a faulty data set, the faulty data set being a data set related to the error information in the cache of the proxy service, the proxy service being used to perform the functions of the access layer;

[0045] The faulty object is added to the fault list, and the lifespan of the faulty object in the fault list is determined, so that the proxy service can shield the faulty object when processing data within the lifespan.

[0046] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0047] Obtain error information and determine the error type and source of the error information;

[0048] Determine the fault level of the error message based on the error type;

[0049] Based on the fault level and the error source, a fault object related to the error information is determined; the fault object includes at least one of a faulty disk and a faulty data set, the faulty data set being a data set related to the error information in the cache of the proxy service, the proxy service being used to perform the functions of the access layer;

[0050] The faulty object is added to the fault list, and the lifespan of the faulty object in the fault list is determined, so that the proxy service can shield the faulty object when processing data within the lifespan.

[0051] The aforementioned fault handling method, apparatus, computer equipment, computer-readable storage medium, and computer program product are applied to the access layer of a distributed storage system. The method involves: acquiring error information and determining the error type and source; determining the fault level of the error information based on the error type; identifying fault objects related to the error information based on the fault level and source, whereby the fault objects include at least one of a faulty disk and a faulty data set. The faulty data set is a cached set of data related to the error information stored in a proxy service, which performs the functions of the access layer. The fault objects are then added to a fault list, and their lifetime in the list is determined. This allows the proxy service to shield the fault objects during data processing within their lifetime, effectively shielding them from the fault at the access layer of the distributed storage system. This interception of fault impacts at the point of direct communication with the outside (access layer) reduces external perception of the distributed storage system (cluster) faults, confining the impact of faults within the distributed storage system (cluster) and preventing their spread. This ensures the distributed storage system is unaffected by localized faults and improves its overall availability. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating a fault handling method in one embodiment;

[0054] Figure 2 This is an architecture diagram of a distributed storage system in one embodiment;

[0055] Figure 3 This is a schematic diagram of the hierarchical structure of storage layer management in one embodiment;

[0056] Figure 4 This is a flowchart illustrating the fault masking triggering phase in one embodiment;

[0057] Figure 5 This is a flowchart illustrating the fault masking activation phase of a write request in one embodiment.

[0058] Figure 6 This is a flowchart illustrating the fault masking activation phase of a read request in one embodiment.

[0059] Figure 7 This is a flowchart illustrating the fault masking and recovery phase in one embodiment;

[0060] Figure 8 This is a structural block diagram of a fault handling device in one embodiment;

[0061] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] In one embodiment, such as Figure 1As shown, a fault handling method is provided. This embodiment illustrates the application of this method to a computer device, which can be a terminal or a server. It is understood that this method can also be applied to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0064] In this embodiment, the fault handling method is applied to the access layer of a distributed storage system and includes the following steps:

[0065] Step S102: Obtain the error information and determine the error type and source of the error information.

[0066] It is understandable that computer equipment includes distributed storage systems, such as... Figure 2 As shown, the distributed storage system for this data includes an access layer, a control layer, and a storage layer. The access layer is responsible for receiving external requests, performing load balancing and security control, and forwarding requests to appropriate storage nodes. The control layer manages system metadata, monitors node health, performs fault detection and recovery, and ensures data consistency. The storage layer actually stores and processes the data, performing data replication and read / write operations to ensure data persistence and availability. In terms of fault handling, the control layer plays a leading role, possessing a global perspective to monitor the health of the entire system, receiving anomaly reports from other layers, and quickly detecting and responding to faults.

[0067] The access layer is the entry point for the entire storage cluster (distributed storage system), and the services that perform the functions of the access layer are called proxy services. Within the access layer, multiple proxy services can be horizontally scaled based on the number of concurrent external requests and the performance and scale of the entire cluster; different proxy services operate independently. The access layer receives external requests, analyzes and processes them, and invokes storage layer services to complete various data operations. In object storage, this external request originates from the object storage protocol gateway. To distinguish between these two types of requests, requests received by the access layer from the outside are called external requests, and requests sent by the access layer to the storage layer are called proxy requests. Based on the operation type, requests can also be categorized as write requests, read requests, and delete requests. The access layer can contain multiple proxy service processes.

[0068] At the storage layer, management structures are implemented differently, and controlling the scope of failure impact is also an issue. For example, in some storage system implementations, a data storage service process manages only one physical disk, while in others it manages multiple physical disks. Too many processes increase the complexity of storage system management and maintenance and waste computer system resources; however, managing too many disks by a single process increases the complexity of process processing logic and the risk of failure impact spreading. The size of data blocks on each physical disk also varies, and data blocks in different states may exist on the same disk. All of these factors necessitate strengthening fault tolerance and redundancy design.

[0069] As the entry point to the entire storage cluster, the access layer directly receives various data operation requests from the object storage protocol gateway in object storage. The impact of some faults may first manifest in the processing of individual requests, especially during periods of high request density. The access layer can detect faults early, and even short-term fluctuations can affect a large number of requests. Due to network and computational latency, fault handling across different modules in different processes or nodes always involves delays and time lags, potentially leading to delayed handling of certain faults. Furthermore, if the cluster is already under high load when a fault occurs, retries and other fault handling mechanisms may have negative consequences. Therefore, strengthening fault identification and mitigation measures at the access layer contributes to the overall stability of the cluster.

[0070] Optionally, during the fault masking trigger phase, the access layer (proxy service) collects error information from the error report or response to determine the error type and source. The error information can be from read requests, write requests, or delete requests.

[0071] If the proxy service fails to send a request to the data storage service, it determines the error type based on the error message and marks the source of the error as the main data block of the requested data group and its corresponding disk. If the proxy service successfully sends a request to the data storage service, but an exception occurs during the data storage service's processing, the data storage service will include the error type, error message, and error source in the returned response.

[0072] Error types can be defined by the operating system or by the distributed storage system itself. Error types include at least one of the following: service not found, service connection refused, request timeout, disk I / O error, disk I / O commit timeout, data block compression in progress, or other internal data block errors.

[0073] The error originated from the execution of the data block that caused the error and the disk where that data block was located.

[0074] Step S104: Determine the fault level of the error message based on the error type.

[0075] The fault level can be set to different levels based on the scope of the fault's impact. The scope of the fault's impact and the fault level are positively correlated; that is, the larger the scope of the fault's impact, the higher the fault level.

[0076] Optionally, the fault level of the error message can be determined based on the error type, including: if the error type indicates that a data block has been corrupted, the fault level of the error message is determined to be a data block level fault; if the error type indicates that a disk has been corrupted, the fault level of the error message is determined to be a disk level fault; if the error type indicates that a service process has been corrupted, the fault level of the error message is determined to be a process level fault.

[0077] like Figure 3 As shown, the applicable scenarios include a common hierarchical structure for storage layer management: a physical node has at least one data storage service process, a data storage service process manages at least one physical disk, and a physical disk can contain multiple data blocks. A data block is the basic unit for replica management in the storage layer, and a data block can contain multiple data entries. A combination of data blocks distributed across different nodes or disks and meeting certain redundancy rules is called a data group.

[0078] according to Figure 3 Based on the hierarchical structure of the storage layer and the potential impact of errors, the fault types of error messages are divided into the following three levels:

[0079] Level 1: Data block level failures. These include failures during data block compression or other internal data block failures. For write operations, these failures typically only affect the writing of a specific data group. Data blocks are distinguished by their data block IDs, which are assigned by the control layer of the distributed storage system (cluster) and must be unique within the cluster. A data group refers to a collection of replicas that meet certain redundancy rules; it is a logical grouping of storage objects.

[0080] Level 2: Disk-level failures. Examples include disk I / O errors and disk I / O commit timeouts. These errors typically affect the normal read and write operations of the entire disk. Disks are distinguished by disk IDs, which can be the disk's serial number or assigned by the cluster. The disk ID must be unique within the distributed storage system (cluster) to identify a disk within the cluster. Each disk registers an entry in the service discovery module, and this registration should include the disk ID.

[0081] Level 3: Process-level failures. Examples include service not found, service port connection refused, and request timeouts. These errors often mean the entire data storage service process cannot provide service normally. Processes can be distinguished through socket addresses in service discovery. This requires all services within the same process to share a single port, and the address information registered in the service discovery module must be the same socket address. A process on the network can be identified by its socket address.

[0082] Optionally, if the fault level of the error message cannot be determined based on the error type, the fault level of the error message shall be determined according to the scope of its impact.

[0083] Step S106: Determine the fault object related to the error information based on the fault level and error source; the fault object includes at least one of a faulty disk and a faulty data set, the faulty data set being a data set related to the error information in the cache of the proxy service, the proxy service being used to complete the functions of the access layer.

[0084] Here, a faulty object is an object affected by a fault. A faulty object includes at least one of a faulty disk and a faulty data set. A faulty data set is a collection of data in the proxy service's cache that has been affected by the error message. A faulty data set can be a faulty data group or a faulty data block.

[0085] Optionally, based on the fault level and the source of the error, fault objects related to the error information are determined, including: if the fault level is a data block level fault, a first object related to the error information is determined, which includes a set of fault data in the proxy service's cache that is related to the error information and has resulted in an error; if the fault level is a disk level fault, a second object related to the error information is determined, which includes a first disk where the source of the error is located, and a set of fault data in the proxy service's cache that is related to the error information and associated with the first disk; if the fault level is a process level fault, a third object related to the error information is determined, which includes a second disk under the same data storage service process as the source of the error, and a set of fault data in the proxy service's cache that is related to the error information and associated with the second disk; the first object, the second object, and the third object are all fault objects.

[0086] For write requests, if the fault level is a block-level fault, the impact is on the current data group. Therefore, the first object includes the faulty data group in the proxy service's cache that is related to the error information and has experienced the error. If the fault level is a disk-level fault, the impact is on the disk where the error originated and the data group associated with that disk. Therefore, the second object includes the first disk where the error originated, and the faulty data group in the proxy service's cache that is related to the error information and associated with the first disk. If the fault level is a process-level fault, the impact is on the disks under the same data storage service process as the error origin and the data groups associated with those disks. Therefore, the third object includes the second disk under the same data storage service process as the error origin, and the faulty data group in the proxy service's cache that is related to the error information and associated with the second disk. In the above, "data group associated with a disk" refers to a specific data block within that data group located on that disk.

[0087] For read requests, if the fault level is a block-level fault, the impact is limited to the current block. Therefore, the first object includes the faulty data block in the proxy service's cache that is related to the error information and has encountered an error. If the fault level is a disk-level fault, the impact is limited to the disk where the error originated and the data blocks distributed on that disk. Therefore, the second object includes the first disk where the error originated and the faulty data block in the proxy service's cache that is related to the error information and associated with the first disk. If the fault level is a process-level fault, the impact is limited to the disk and the data blocks distributed on that disk within the same data storage service process as the error origin. Therefore, the third object includes the second disk within the same data storage service process as the error origin and the faulty data block in the proxy service's cache that is related to the error information and associated with the second disk.

[0088] Understandably, write requests are characterized by the following: writing is a modification operation that requires successful writing to all data blocks within a data group, but it doesn't have to be a specific data group. If a write operation fails on one data group, it can be changed to another. The proxy service caches the set of data group information used for write requests in memory. Read requests, on the other hand, are characterized by the following: the data group containing the data to be read is clearly defined, but since reading is a non-modification operation and the data group is redundant, reading a certain number of data blocks that meet the redundancy requirements is sufficient. The proxy service caches information about the data groups recently accessed.

[0089] Therefore, it is important to emphasize that write requests search for data groups, while read requests search for data blocks. Due to the redundancy design, if only one data block in a data group fails, it will affect the writing of the entire data group, but will not affect the reading of the entire data group.

[0090] Understandably, classifying errors into different fault levels based on their source and type facilitates the estimation of the error's importance and scope of impact, timely identification of which nodes, disks, or data may be affected, and blocking access to these potentially affected contents for a period of time can avoid repeated access and retries to known severely faulty nodes, reduce unnecessary network and computational overhead, improve the overall performance and efficiency of the system, and provide better conditions for cluster fault recovery.

[0091] Step S108: Add the faulty object to the fault list and determine the lifespan of the faulty object in the fault list, so that the proxy service can shield the faulty object when processing data within the lifespan.

[0092] The fault list is a list used for fault masking. The fault list indicates that the faulty objects added to the fault list are in a masked state. The fault list is the same as the blacklist.

[0093] The lifetime is the duration for which a faulty object remains on the fault list. During this lifetime, the faulty object is in a fault-masked state and cannot be processed. The lifetime can be set as needed. If the error message is a read request, the fault list will be the read request fault list; if the error message is a write request, the fault list will be the write request fault list; if the error message is a delete request, the fault list will be the delete request fault list.

[0094] Optionally, the access layer (proxy service) adds the faulty object to the fault list in the fault shielding module and configures the lifetime of the faulty object in the fault list.

[0095] This lifetime can be programmatically defined as an unchangeable value based on different error types, but it is recommended to set it as a changeable configuration item according to common error types; the recommended order of magnitude is seconds. That is, depending on the error type, a constant can be directly used in the program code as the lifetime of the faulty object in the fault list. For example, specifying in the code that the lifetime of the disk and data group in the fault list is 1 second when a "Disk I / O busy" error is received. This implementation method sets the lifetime in a way that cannot be changed online once compiled unless the code is modified. It is recommended to make "the lifetime of the disk and data group in the blacklist under different error types" a configuration item, allowing it to be changed online without modifying the code. For example, the lifetime corresponding to errors such as "Disk I / O commit timeout," "Disk I / O busy," and "Service port connection refused" can be set as separate configuration items, with the configuration value determined by reading the configuration file.

[0096] Optionally, faulty objects that are faulty in the cache can be marked as faulty or deleted directly.

[0097] Optionally, for write requests, the access layer removes the faulty data groups affected by the fault from the cache of the data group set used by the proxy service for write allocation, or marks the faulty data groups affected by the fault in the cache of the data group set used by the proxy service for write allocation; if some faulty data groups or faulty disks are already in the fault list, these faulty data groups or faulty disks are skipped.

[0098] For read requests, the access layer simultaneously marks the affected faulty data blocks in the data group cache used by the proxy service for write allocation; if some faulty data blocks or faulty disks are already on the fault list, these data blocks or disks are skipped. Since a single data block failure in a read request does not affect the read operation of the entire data group, the granularity of fault masking is not the data group but the data block. Furthermore, instead of directly deleting the data block mapping information of the data group in the cache, fault markers can be added to the faulty data blocks.

[0099] Optionally, alarm information can be set based on the content of the fault masking to provide reference for operation and maintenance personnel.

[0100] In one implementation: alarms are set according to the fault level. For example, a first-level alarm is triggered when a process-level fault occurs, a second-level (medium or low-level) alarm is triggered when a disk-level fault occurs, and no alarm is triggered when a data block-level fault occurs; only an error log is printed in the log. The first-level alarm is a high-level alarm, and the second-level alarm is a medium or low-level alarm.

[0101] In another implementation: alarm messages are set based on the number of faulty objects in the fault list. For example, if the number of faulty objects in the fault list of a single agent service exceeds a set threshold, an alarm message is sent to remind operations and maintenance personnel to check the health status of the cluster, or to set up some automated processing procedures accordingly.

[0102] The above fault handling method is applied to the access layer of a distributed storage system. It involves acquiring error information and determining the error type and source; determining the fault level based on the error type; and identifying the fault object related to the error information based on the fault level and source. This fault object includes at least one of a faulty disk and a faulty data set, where the faulty data set is a cached set of data related to the error information from a proxy service used to perform the functions of the access layer. The fault object is then added to a fault list, and its lifetime in the list is determined. This allows the proxy service to shield the fault object during data processing within its lifetime. In other words, by shielding the fault at the access layer of the distributed storage system, the impact of the fault can be intercepted at the point of direct communication with the outside (access layer), reducing external perception of the distributed storage system (cluster) faults and controlling the impact of the fault within the distributed storage system (cluster) to prevent it from spreading outwards. This ensures that the distributed storage system is not affected by localized faults and improves the overall availability of the distributed storage system.

[0103] Applied to the access layer of distributed storage systems, the access layer can detect the occurrence of faults earlier and intercept the impact of faults at the location that directly communicates with the outside world as much as possible. This reduces the external perception of cluster faults and their impact on front-end services, keeping the impact of faults within the cluster and preventing them from spreading outward. It ensures that the overall availability of the system is not affected by local faults and plays a gatekeeping role.

[0104] The access module in the access layer can detect faults early, especially under intensive request conditions. Due to network and computational latency, there will always be delays and time differences in fault handling between different modules across processes or nodes. Furthermore, if the cluster is already under high load when a fault occurs, retries and other fault handling mechanisms may cause other negative effects. Therefore, the above fault handling methods can effectively and accurately determine the scope of the fault and shield it at the source of the request in the access layer of the distributed storage system (cluster), thereby effectively cooling down the cluster and making it more stable.

[0105] By hierarchically shielding faults at the access layer, the impact of faults is intercepted at points directly communicating with the outside world, reducing external awareness of cluster faults and confining their impact within the cluster to prevent outward spread. This ensures overall system availability is unaffected by localized failures. Furthermore, it avoids repeated accesses and retries to known severely faulty nodes, reducing unnecessary network and computational overhead, improving overall system performance and efficiency, and providing better conditions for cluster fault recovery. For example, when disk I / O load is too high, I / O requests on that disk need to be reduced. Hierarchical processing, by temporarily isolating the disk during write operations, can effectively reduce its load. Additionally, by integrating with a monitoring system, different early warning and alarm mechanisms can be set for different levels of faults, facilitating timely detection, location, and handling of problems.

[0106] Furthermore, different types of faults in the distributed storage system (cluster) are tiered and shielded, and a fault isolation mechanism is used to avoid unnecessary retries, effectively controlling the scope of the fault's impact and reducing the request and processing pressure on the fault source. To ensure low coupling between modules throughout the cluster, the fault handling logic is built upon the existing basic fault handling logic of the distributed storage system, but it does not directly participate in the execution or replace the original fault handling function of the control layer; it is only provided as an enhancement. The main logic of the above fault handling method is confined to the internal environment of a single access layer proxy service process; the operations of different access layer proxy service processes do not affect each other, and data is not shared between them.

[0107] In one embodiment, such as Figure 4 As shown, the fault masking triggering phase for read or write requests includes the following steps:

[0108] In step S402, the agent service collects error information from the error report or response to determine the error type and source.

[0109] In step S404, the agent service determines the fault level of the error according to predefined rules.

[0110] In step S406, the agent service identifies the list of affected disks in the cluster and the list of affected data groups in the agent service cache based on the error level and error source.

[0111] In step S408, after the agent service finds the list of affected disks and data groups, it adds these disks and data groups to the write request blacklist in the fault masking module and specifies a lifespan in the blacklist. It then uses methods such as adding fault markers or deleting from the cache information summary to exclude faulty data groups or data blocks in the cache.

[0112] In one embodiment, the distributed storage system further includes a control layer; after adding the faulty object to the fault list and determining the lifetime of the faulty object in the fault list, so that the proxy service can shield the faulty object from failure when processing data within the lifetime, the system further includes: in response to an external write request, checking whether the number of normal data groups in the data group set cached by the proxy service is sufficient; if so, determining the target data group that is not in the fault list from the data group set; if not, excluding the faulty disk in the fault list and requesting the control layer to allocate the target data group that has not been affected by the error information; and initiating a proxy write request to the target data group.

[0113] The fault masking phase for write requests is primarily handled within the basic modules of the proxy service. This occurs after fault masking is triggered and before recovery, during the processing of new write requests arriving in this period. When the access layer receives a new external write request, it validates the content of the request. If the validation passes, it checks whether the number of normal data groups in the proxy service's cached data group set is sufficient.

[0114] Among them, validating the content of external write requests refers to validating the request body parameters of the external write request. For example, checking the request parameters to confirm whether the storage pool to be written exists, whether the data to be written is empty, and whether the size of the data to be written exceeds the maximum limit, etc.

[0115] If the validation fails, the request is considered to have failed, and an error such as "parameter error" is returned.

[0116] If the number of normal data groups in the cached data group set of the proxy service is sufficient, then the data group that has not been affected by the failure will be selected from the cached data group set for writing, ensuring that the data group in the blacklist will not receive new external write requests during this period.

[0117] If the number of normal data groups in the data group set cached by the proxy service is insufficient, the current faulty disk is obtained from the fault masking module for exclusion, and a certain number of target data groups that are not affected by the fault are requested from the control layer for writing. This ensures that the disks in the fault list will not receive new external write requests during this period.

[0118] In this embodiment, in response to an external write request, the system checks whether the number of normal data groups in the data group set cached by the proxy service is sufficient. If so, it determines the target data group that is not on the fault list from the data group set; otherwise, it excludes the faulty disk from the fault list and requests the control layer to allocate the target data group that has not been affected by the error information. Finally, a proxy write request is initiated to the target data group, which can achieve the effect of avoiding write requests to areas with faults. Different fault shielding processes are set according to the type of error to control the scope of the fault's impact and ensure that all requests succeed as much as possible when a fault occurs.

[0119] In one embodiment, such as Figure 5 As shown, the fault masking activation phase for write requests includes the following steps:

[0120] Step S502: The proxy service verifies the content of the newly received external write request.

[0121] Step S504: The agent service checks whether the number of data groups used for writing allocation is sufficient. If so, proceed to step S506; otherwise, proceed to step S508.

[0122] Step S506: Select the data group that was not affected by the fault directly from the cached data group set for writing.

[0123] Step S508: Obtain the current list of faulty disks from the fault masking module for exclusion, and request the control layer to allocate a certain number of data groups that are not affected by the fault for writing.

[0124] In step S510, the proxy service initiates a proxy write request to the selected data group.

[0125] In one embodiment, the distributed storage system further includes a control layer and a storage layer; after adding the faulty object to the fault list and determining the lifetime of the faulty object in the fault list, so that the proxy service can shield the faulty object when processing data within the lifetime, the system further includes: in response to an external read request, checking whether the data group containing the data to be read corresponding to the external read request is hit in the cache of the proxy service; if so, selecting the target data block that is not in the fault list from the cache; if not, requesting the specified data group information from the control layer, adding the specified data group information to the cache of the proxy service, and selecting the target data block that is not in the fault list from the cache; and initiating a read request for the target data block from the storage layer.

[0126] During the fault masking phase for read requests, when the access layer receives a new external read request, it verifies the content of the external read request. If the verification passes, it performs a step to check whether the data group containing the data to be read corresponding to the external read request is cached in the proxy service.

[0127] Among them, validating the content of an external read request refers to validating the request body parameters of the external read request, such as checking the request parameters to confirm whether the storage pool to be read exists, whether the data to be read is empty, etc.

[0128] If the validation fails, the request is considered to have failed, and an error such as "parameter error" is returned.

[0129] Optionally, if the data group containing the data to be read corresponding to the external read request is not found in the cache of the proxy service, the access layer requests the specified data group information from the control layer, adds the specified data group information to the cache of the proxy service, and compares the specified data group information with the fault disks in the fault list of the fault masking module. If there is a data block in the fault list in the specified data group information, a fault mark is added to the fault data block in the specified data group information.

[0130] For data groups with multiple replica redundancy rules, you only need to send a read request to a normal data block.

[0131] For datasets using erasure coding redundancy rules, a read request needs to be initiated for a specific data block. This data block then needs to read a certain number of other data blocks from the storage layer before erasure coding calculations are performed to restore the data. In this case, a recommended list of normal data blocks can be included in the read request to minimize the need for the storage layer to request faulty data blocks during erasure coding. However, this list of normal data blocks is only for reference during actual storage layer reads. If the required number of copies is insufficient, the blocking can be ignored to request other data blocks, ensuring that the original read performance of the system is not compromised.

[0132] If too many data blocks are currently marked as faulty, making it impossible to find a suitable data block to initiate a read request, one possible implementation is to randomly select a data block to initiate a read request. Another possible implementation is to select the data block that was removed from the blacklist earliest to initiate a read request. The access layer's judgment on the scope and duration of the fault's impact is an estimate and may differ from the actual situation. Therefore, in such extreme cases, various possibilities should be explored, and all related logic is designed with the principle of not reducing the effectiveness of the original read logic.

[0133] Erasure coding, a data protection technology, provides data redundancy and fault tolerance in storage systems. It divides data into multiple blocks and generates additional check blocks, allowing the original data to be recovered from the remaining blocks even if some blocks are lost or corrupted. Erasure coding is widely used in distributed storage systems, cloud storage, and big data storage to improve data reliability and storage efficiency. Compared to multi-replica redundancy, erasure coding offers a significant advantage in higher storage utilization.

[0134] In this embodiment, in response to an external read request, it checks whether the data group containing the data to be read corresponding to the external read request is found in the proxy service's cache. If so, a target data block not in the fault list is selected from the cache. If not, the specified data group information is requested from the control layer, added to the proxy service's cache, and a target data block not in the fault list is selected from the cache. A read request for the target data block is then initiated from the storage layer, thereby minimizing the need to initiate read requests to faulty areas. Different fault shielding processes are set according to the type of error to control the scope of the fault's impact and ensure that all requests succeed as much as possible when a fault occurs.

[0135] In one embodiment, such as Figure 6 As shown, the fault masking activation phase for read requests includes the following steps:

[0136] Step S602: The proxy service verifies the request content of the newly received external read request.

[0137] Step S604: Check if the data group information where the data to be read is located is hit in the cache. If yes, proceed to step S608; otherwise, proceed to step S606.

[0138] In step S606, the agent service requests the specified data group information from the control layer, adds the information to the cache, compares it with the faulty disks in the fault masking module's blacklist, and adds fault markers to the data blocks on these disks if a fault is found.

[0139] Step S608: Select normal data blocks that have not been marked as faulty.

[0140] In step S610, the proxy service sends a read request to the storage layer for the selected data block.

[0141] In one embodiment, the distributed storage system further includes a control layer and a storage layer; the method further includes: if a first faulty object in the fault list triggers the fault handling function of the control layer, then obtaining a notification from the control layer to update or delete the faulty object; updating or deleting the first faulty object in the fault list according to the notification to cooperate with the control layer in fault handling; if a second faulty object in the fault list does not trigger the fault handling function of the control layer, and the duration of the second faulty object in the fault list reaches the lifespan, then deleting the second faulty object from the fault list.

[0142] The fault shielding and recovery phase for write requests is mainly handled by the fault shielding module of the access layer (proxy service), which occurs after the control layer triggers the fault handling function or after the lifespan of the faulty object in the target list expires.

[0143] If a fault triggers the fault handling function of the control layer, the control layer will send a notification to the access layer to update or delete the faulty data group. Upon receiving the notification, the access layer (proxy service) updates or deletes the first faulty object in its cache and fault masking module according to the notification, cooperating with the control layer to handle the fault. Optionally, the access layer removes faulty data sets (data groups or data blocks) that have been updated in the target list of the fault masking module.

[0144] Optionally, the fault masking module checks in real time whether the second fault object in the fault list has reached the corresponding lifespan. If the second fault object does not receive an update or deletion notification from the control layer until the expiration date, the fault masking module will delete the second fault object from the fault list.

[0145] Optionally, if the target data set (data group or data block) information in the cache is deleted when the fault is added to the fault list, the target data set is restored to its original position.

[0146] Optionally, if a fault mark exists for either the first or second fault object, then the fault mark is deleted.

[0147] For write requests, in practice, if a disk I / O commit timeout error occurs, a common cause is excessive I / O load on the current disk. The lifetime for this error can be configured to a short time, such as 1 second, so that the current proxy service avoids writing to it during this period and instead writes to other disks, which can reduce the I / O load on the problematic disk to some extent. If no further errors occur after writing after the lifetime expires, the disk can be considered to have recovered and no further action is needed. If errors persist, the disk should be added back to the fault list to continue avoiding writes and reducing the load. This process should be repeated until the disk recovers or the control layer's fault handling is triggered.

[0148] Many service discovery modules use leases to determine service availability. Therefore, errors indicating a service port connection refusal can be configured to reflect the lease expiration time. This allows the access layer to avoid sending unsuccessful requests to the data storage service between the service outage and the lease expiration time. Once the lease has actually expired, the control layer will normally notify the access layer that the relevant data group is no longer writable. If it's just a brief network outage, the proxy service will resume writing to the data storage service after the data group is removed from the blacklist within a short period.

[0149] Adding this fault shielding processing to the access layer can effectively alleviate the jitter of the distributed storage system (cluster); when the distributed storage system (cluster) fails but can automatically recover to a normal state in a short time through certain methods, it can reduce the external perception of this recovery phase and effectively improve the stability of the cluster.

[0150] In one embodiment, such as Figure 7 As shown, the fault masking and recovery phase for read or write requests includes the following steps:

[0151] In step S702, the fault triggers the fault handling function of the control layer, and the control layer sends a notification to the access layer to update the fault data group information or delete the fault data group.

[0152] In step S704, after receiving the above notification, the agent module updates the fault data group information in the cache and fault masking module or deletes the fault data group according to the notification, cooperating with the control layer's fault handling. The data groups or data blocks whose information has been updated in this step are removed from the fault masking module's blacklist.

[0153] Step S706: After the timeout period of certain disks, data blocks or data groups that have failed in the fault masking module expires in the blacklist, these disks, data blocks or data groups are removed from the blacklist, and the normal information of these data blocks or data groups is restored in the cache.

[0154] In another embodiment, fault masking for delete requests is described. A delete request is characterized by needing to operate on a specific data group and ultimately succeed on all data blocks. This necessitates that delete requests succeed as much as possible on specific data blocks, and since the access layer's assessment of the fault scope is a prediction, bypassing specific data blocks through a masking mechanism is no longer applicable.

[0155] Optionally, instead of blocking deletion requests at the access layer, fault handling can be handled by the control layer.

[0156] In another embodiment, a different fault handling method is provided, applied to the access layer of a distributed storage system. This distributed storage system also includes a control layer and a storage layer. The fault handling method comprises three stages: fault masking triggering, fault masking activation, and fault masking recovery. Different processing is required for different request types. The fault masking triggering stage includes steps A1 to A4, the fault masking activation stage includes steps A5 to A6, and the fault masking recovery stage includes step A7.

[0157] Step A1: Obtain error information and determine the error type and source; error information includes error information for read requests or write requests.

[0158] Step A2: If the error type indicates that an error has occurred in a data block, then the fault level of the error message is determined to be a data block level fault; if the error type indicates that an error has occurred in a disk, then the fault level of the error message is determined to be a disk level fault; if the error type indicates that an error has occurred in a service process, then the fault level of the error message is determined to be a process level fault.

[0159] Step A3: If the fault level is a data block level fault, then determine the first object related to the error information. The first object includes a set of faulty data related to the error information and which has caused the error in the proxy service's cache. If the fault level is a disk level fault, then determine the second object related to the error information. The second object includes the first disk where the error source is located, and a set of faulty data related to the error information and associated with the first disk in the proxy service's cache. If the fault level is a process level fault, then determine the third object related to the error information. The third object includes the second disk under the same data storage service process as the error source, and a set of faulty data related to the error information and associated with the second disk in the proxy service's cache. The first object, the second object, and the third object are all fault objects. Fault objects include at least one of a faulty disk and a faulty data set. The faulty data set is a set of data related to the error information in the proxy service's cache. The proxy service is used to complete the functions of the access layer.

[0160] Step A4: Add the faulty object to the fault list and determine the lifespan of the faulty object in the fault list, so that the proxy service can shield the faulty object when processing data within the lifespan.

[0161] In response to an external write request, proceed to step A5; in response to an external read request, proceed to step A6.

[0162] Step A5: Check if the number of normal data groups in the data group set cached by the proxy service is sufficient; if so, determine the target data group that is not on the fault list from the data group set; if not, exclude the faulty disks on the fault list and request the control layer to allocate the target data group that has not been affected by the error message; initiate a proxy write request to the target data group.

[0163] Step A6: Check if the data group containing the data to be read corresponding to the external read request is hit in the cache of the proxy service; if yes, select the target data block that is not in the fault list from the cache; if no, request the specified data group information from the control layer, add the specified data group information to the cache of the proxy service, and select the target data block that is not in the fault list from the cache; initiate a read request for the target data block from the storage layer.

[0164] Step A7: If the first fault object in the fault list triggers the fault handling function of the control layer, then obtain the notification sent by the control layer to update or delete the fault object; according to the notification, update or delete the first fault object in the fault list to cooperate with the control layer in fault handling; if the second fault object in the fault list does not trigger the fault handling function of the control layer, and the duration of the second fault object in the fault list reaches the lifespan, then delete the second fault object from the fault list.

[0165] Optionally, due to the higher fault tolerance of read requests, a limited number of retries can be tolerated to simplify the read request process. In implementation, fault masking for read requests can be omitted. No additional fault masking operations are added to the access layer for read requests; the current request reads a different data block. In this case, simple read logic is preferred, tolerating the overhead of retries. Since the data groups are distributed as widely as possible by design, as long as the cluster remains in a normal and available state and there are sufficient replicas, the read will definitely succeed within a retry count not exceeding the number of replicas.

[0166] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0167] Based on the same inventive concept, this application also provides a fault handling apparatus for implementing the fault handling method described above. The solution provided by this apparatus is similar to the solution described in the above method; therefore, the specific limitations in one or more fault handling apparatus embodiments provided below can be found in the limitations of the fault handling method described above, and will not be repeated here.

[0168] In one exemplary embodiment, such as Figure 8 As shown, a fault handling device is provided, including: an error type and error source determination module 802, a fault level determination module 804, a fault object determination module 806, and a fault shielding module 808, wherein:

[0169] The error type and error source determination module 802 is used to acquire error information and determine the error type and error source of the error information;

[0170] The fault level determination module 804 is used to determine the fault level of the error information based on the error type.

[0171] The fault object determination module 806 is used to determine the fault object related to the error information based on the fault level and the error source; the fault object includes at least one of a faulty disk and a faulty data set, the faulty data set is a data set related to the error information in the cache of the proxy service, and the proxy service is used to complete the function of the access layer;

[0172] The fault masking module 808 is used to add the fault object to the fault list and determine the lifespan of the fault object in the fault list, so that the proxy service can mask the fault object when processing data within the lifespan.

[0173] The aforementioned fault handling device is applied to the access layer of a distributed storage system. It acquires error information and determines the error type and source. Based on the error type, it determines the fault level of the error information. Based on the fault level and source, it identifies the fault object related to the error information. This fault object includes at least one of a faulty disk and a faulty data set. The faulty data set is a cached set of data related to the error information in a proxy service, which performs the functions of the access layer. The fault object is then added to a fault list, and its lifetime in the list is determined. This allows the proxy service to shield the fault object during data processing within its lifetime. In other words, by shielding the fault at the access layer of the distributed storage system, the impact of the fault can be intercepted at the location directly communicating with the outside (access layer), reducing external perception of the distributed storage system (cluster) faults. The impact of the fault is controlled within the distributed storage system (cluster) and does not spread outwards, ensuring that the distributed storage system is not affected by localized faults and improving the overall availability of the distributed storage system.

[0174] In one embodiment, the fault level determination module 804 is further configured to determine the fault level of the error information as a data block level fault if the error type indicates an error in a data block; determine the fault level of the error information as a disk level fault if the error type indicates an error in a disk; and determine the fault level of the error information as a process level fault if the error type indicates an error in a service process.

[0175] In one embodiment, the fault object determination module 806 is further configured to: if the fault level is a data block level fault, determine a first object related to the error information, the first object including a set of fault data in the proxy service's cache that is related to the error information and has caused the error; if the fault level is a disk level fault, determine a second object related to the error information, the second object including a first disk where the error source is located, and a set of fault data in the proxy service's cache that is related to the error information and associated with the first disk; if the fault level is a process level fault, determine a third object related to the error information, the third object including a second disk in the same data storage service process as the error source, and a set of fault data in the proxy service's cache that is related to the error information and associated with the second disk; the first object, the second object, and the third object are all fault objects.

[0176] In one embodiment, the distributed storage system further includes a control layer; the above apparatus further includes a fault masking module, used to, in response to an external write request, check whether the number of normal data groups in the data group set cached by the proxy service is sufficient; if so, determine a target data group not in the fault list from the data group set; if not, exclude the faulty disks in the fault list, and request the control layer to allocate a target data group not affected by the error information; and initiate a proxy write request to the target data group.

[0177] In one embodiment, the distributed storage system further includes a control layer and a storage layer; the aforementioned fault masking module is further configured to, in response to an external read request, check whether the data group containing the data to be read corresponding to the external read request is found in the cache of the proxy service; if so, select a target data block not in the fault list from the cache; if not, request specified data group information from the control layer, add the specified data group information to the cache of the proxy service, and select a target data block not in the fault list from the cache; and initiate a read request for the target data block from the storage layer.

[0178] In one embodiment, the distributed storage system further includes a control layer and a storage layer; the device further includes a fault masking and recovery module, configured to: if a first fault object in the fault list triggers the fault handling function of the control layer, obtain a notification from the control layer to update or delete the fault object; update or delete the first fault object in the fault list according to the notification to cooperate with the control layer in fault handling; if a second fault object in the fault list does not trigger the fault handling function of the control layer, and the second fault object's duration in the fault list reaches its lifespan, then delete the second fault object from the fault list.

[0179] Each module in the aforementioned fault handling device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0180] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a fault handling method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0181] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0182] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0183] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0184] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0185] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0186] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0187] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0188] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A failure handling method characterized by, An access layer applied to a distributed storage system, the distributed storage system further including a control layer and a storage layer; the method includes: Obtain error information and determine the error type and source of the error information; Determine the fault level of the error message based on the error type; Based on the fault level and the error source, a fault object related to the error information is determined; the fault object includes at least one of a faulty disk and a faulty data set, the faulty data set being a data set related to the error information in the cache of the proxy service, the proxy service being used to complete the functions of the access layer; different access layer proxy services operate independently of each other and do not share data. The faulty object is added to the fault list, and the lifespan of the faulty object in the fault list is determined, so that the proxy service can shield the faulty object when processing data within the lifespan. In response to an external write request, the access layer verifies the content of the external write request. If the verification passes, it checks whether the number of normal data groups in the data group set cached by the proxy service is sufficient. If so, it determines the target data group that is not in the fault list from the data group set. If not, it excludes the faulty disks in the fault list and requests the control layer to allocate the target data group that is not affected by the error information. Then, it initiates a proxy write request to the target data group. In response to an external read request, the access layer verifies the content of the external read request. If the verification passes, it checks whether the data group containing the data to be read corresponding to the external read request is found in the cache of the proxy service. If so, it selects a target data block from the cache that is not in the fault list. If not, it requests specified data group information from the control layer, adds the specified data group information to the cache of the proxy service, and selects a target data block from the cache that is not in the fault list. It then initiates a read request for the target data block from the storage layer.

2. The method of claim 1, wherein, Determining the fault level of the error information based on the error type includes: If the error type indicates that an error has occurred in a data block, then the fault level of the error message is determined to be a data block-level fault. If the error type indicates that a disk error has occurred, then the fault level of the error message is determined to be a disk-level fault. If the error type indicates that an error has occurred in the service process, then the fault level of the error message is determined to be a process-level fault.

3. The method of claim 2, wherein, The step of determining the fault object related to the error information based on the fault level and the error source includes: If the fault level is a data block level fault, then a first object related to the error information is determined, the first object including a set of fault data in the cache of the proxy service that are related to the error information and have caused the error; If the fault level is a disk-level fault, then a second object related to the error information is determined. The second object includes the first disk where the error source is located, and a set of fault data in the cache of the proxy service that is related to the error information and associated with the first disk. If the fault level is a process-level fault, then a third object related to the error information is determined. The third object includes a second disk under the same data storage service process as the source of the error, and a set of fault data in the cache of the proxy service that is related to the error information and associated with the second disk. The first object, the second object, and the third object are all fault objects.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: If the first fault object in the fault list triggers the fault handling function of the control layer, then a notification to update or delete the fault object sent by the control layer is obtained; according to the notification, the first fault object in the fault list is updated or deleted to cooperate with the control layer in fault handling; If the second fault object in the fault list does not trigger the fault handling function of the control layer, and the duration of the second fault object in the fault list reaches the survival time, then the second fault object will be deleted from the fault list.

5. A failure handling apparatus characterized by comprising: An access layer for a distributed storage system, the distributed storage system further including a control layer and a storage layer; the device includes: An error type and error source determination module is used to acquire error information and determine the error type and error source of the error information; A fault level determination module is used to determine the fault level of the error information based on the error type. The fault object determination module is used to determine the fault object related to the error information based on the fault level and the error source; the fault object includes at least one of a faulty disk and a faulty data set, the faulty data set is a data set related to the error information in the cache of the proxy service, and the proxy service is used to complete the functions of the access layer; the operations of different access layer proxy services do not affect each other and their data is not shared; The fault masking module is used to add the faulty object to the fault list and determine the lifespan of the faulty object in the fault list, so that the proxy service can mask the faulty object when processing data within the lifespan. The fault masking module is used to respond to external write requests. The access layer verifies the content of the external write request. If the verification passes, it checks whether the number of normal data groups in the data group set cached by the proxy service is sufficient. If so, it determines the target data group that is not in the fault list from the data group set. If not, it excludes the faulty disks in the fault list and requests the control layer to allocate the target data group that is not affected by the error information. It then initiates a proxy write request to the target data group. The fault masking module is also used to respond to external read requests. The access layer verifies the content of the external read request. If the verification passes, it checks whether the data group containing the data to be read corresponding to the external read request is found in the cache of the proxy service. If so, it selects a target data block that is not in the fault list from the cache. If not, it requests the specified data group information from the control layer, adds the specified data group information to the cache of the proxy service, and selects a target data block that is not in the fault list from the cache. It then initiates a read request for the target data block from the storage layer.

6. The apparatus of claim 5, wherein, The fault level determination module is further configured to determine the fault level of the error information as a data block level fault if the error type indicates an error in a data block; determine the fault level of the error information as a disk level fault if the error type indicates an error in a disk; and determine the fault level of the error information as a process level fault if the error type indicates an error in a service process.

7. The apparatus of claim 6, wherein, The fault object determination module is further configured to determine a first object related to the error information if the fault level is a data block level fault, wherein the first object includes a set of fault data in the cache of the proxy service that are related to the error information and have caused the error; If the fault level is a disk-level fault, then a second object related to the error information is determined. The second object includes the first disk where the error source is located, and a set of fault data in the cache of the proxy service that is related to the error information and associated with the first disk. If the fault level is a process-level fault, then a third object related to the error information is determined. The third object includes a second disk under the same data storage service process as the source of the error, and a set of fault data in the cache of the proxy service that is related to the error information and associated with the second disk. The first object, the second object, and the third object are all fault objects.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images