Fault recovery method and device, electronic equipment and storage medium

Abstract

The invention discloses a fault recovery method and device, electronic equipment and a storage medium, and relates to the technical field of storage, and the method comprises the steps: for a request which is not processed completely, extracting a completion timestamp from a corresponding main request management object, determining that the completion timestamp is smaller than a second timestamp corresponding to the current moment, and if yes, executing the step 1; if yes, the request is overtime, and the service level may have a fault. At the moment, the fault service level can be determined according to the preset fault detection rule and the reference object corresponding to the preset fault detection rule, and the fault service level is accurately recovered. Therefore, hot restart does not need to be carried out on the storage management equipment, the problem of data loss caused by hot restart can be avoided, and the stability of the storage system is improved.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a fault recovery method, device, electronic device, and storage medium. Background Art

[0002] In the field of storage technology, a storage management device performs read and write operations corresponding to requests received from a host device. Typically, various failures may occur in the storage management device, causing request processing timeouts and preventing the read or write tasks from completing within the specified timeframe. Therefore, the storage management device performs a hot restart to promptly clear timed-out requests.

[0003] However, hot restart can cause data loss and greatly reduce the stability of the storage system. Summary of the Invention

[0004] The present application provides a fault recovery method, device, electronic device, storage medium, and program product to solve the problem of data loss caused by fault recovery and reduced stability of the storage system.

[0005] This application provides a fault recovery method, including:

[0006] Extracting a first timestamp and object identification information from a pre-acquired target primary request management object, wherein the target primary request management object is a primary request management object corresponding to a target request, and the target request is any one of the at least one request that has not been completely processed;

[0007] When it is determined that the first timestamp is less than the second timestamp corresponding to the current moment, determining the faulty service level according to a preset fault detection rule and a reference object corresponding to the preset fault detection rule, wherein the faulty service level is any service level among multiple service levels passed through to complete any request, and the reference object is the target primary request management object or object identification information;

[0008] Recover the faulty business layer.

[0009] This application also provides a fault recovery device, including:

[0010] an extraction module, configured to extract a first timestamp and object identification information from a pre-acquired target primary request management object, wherein the target primary request management object is a primary request management object corresponding to a target request, and the target request is any one of the at least one request that has not been completely processed;

[0011] a determination module configured to, when determining that the first timestamp is less than a second timestamp corresponding to the current moment, determine a faulty service level based on a preset fault detection rule and a reference object corresponding to the preset fault detection rule, wherein the faulty service level is any one of a plurality of service levels passed through to complete any request, and the reference object is a target primary request management object or object identification information;

[0012] The recovery module is used to recover the faulty service layer.

[0013] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned fault recovery methods when executing the computer program.

[0014] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned fault recovery methods are implemented.

[0015] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned fault recovery methods when executed by a processor.

[0016] Through this application, for unfinished requests, the completion timestamp can be extracted from the corresponding main request management object, and the completion timestamp can be determined to be less than the second timestamp corresponding to the current moment. If so, it means that the request has timed out, and there may be a failure at the service level. At this time, based on the preset fault detection rules and the reference objects corresponding to the preset fault detection rules, the faulty service level can be determined and accurately restored to the faulty service level. This eliminates the need to hot-restart the storage management device, avoids data loss caused by hot-restart, and improves the stability of the storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 A schematic diagram of the architecture of a storage system provided in an embodiment of the present application;

[0019] Figure 2 A flowchart of a fault recovery method provided in an embodiment of the present application;

[0020] Figure 3A schematic diagram of a business processing flow of a request provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of a process for identifying a timed-out request provided in an embodiment of the present application;

[0022] Figure 5 A schematic diagram of a fault recovery process provided in an embodiment of the present application;

[0023] Figure 6 A schematic diagram of a process for performing fault recovery processing on a RAID service layer provided in an embodiment of the present application;

[0024] Figure 7 A schematic diagram of the structure of a fault recovery device provided in an embodiment of the present application;

[0025] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0027] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0028] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0029] The fault recovery method provided in this application can be implemented by the storage system, such as Figure 1 As shown, a storage system may include a processing device and a storage management device. The processing device can generate requests based on user operations or existing business processes and send them to the storage management device. The storage management device processes the requests and performs the corresponding read and write operations. The requests can be read or write requests, also commonly referred to as IO.

[0030] The embodiment of the present application provides a fault recovery method, which can be executed by the above-mentioned storage management device, such as Figure 2 As shown, the specific processing steps of the fault recovery method may include:

[0031] Step S201: extracting a first timestamp and object identification information from a pre-acquired target main request management object.

[0032] The target primary request management object may be a primary request management object corresponding to the target request, and the target request may be any one of the at least one uncompleted request. The object identification information may be a number, for example, a 4-byte digital number "0xE12D6C38".

[0033] Specifically, each time the storage management device receives a request from the processing device, it can generate a corresponding main request management object for the request to monitor whether the service processing of the request has timed out. Accordingly, taking the target request as an example, obtaining the target main request management object may include the following steps:

[0034] Step 1: When a target request sent by a processing device is received, a third timestamp corresponding to the moment when the target request is received is obtained, and object identification information is allocated to the target request.

[0035] The target request may include an operation type, such as a read operation or a write operation.

[0036] Step 2: According to the operation type, obtain the business processing time corresponding to the operation type.

[0037] Step three: determine the first timestamp based on the third timestamp and the service processing duration corresponding to the operation type.

[0038] Step 4: Construct a target main request management object corresponding to the target request according to the object identification information, the first timestamp, and the operation type.

[0039] Specifically, after receiving the target request sent by the processing device, the storage management device can first obtain a third timestamp corresponding to the current moment (the current moment here refers to the moment when the target request is received) and assign a unique object identification information to the target request, such as the object identification information mentioned above. In addition, the storage management device can parse the target request to obtain the operation type, address information, and when the operation type is a write operation, it can also parse out the write data. The address information may include the identification information of the storage array managed by the storage management device, the starting position on the storage array corresponding to the identification information of the storage array, the operation length, etc.

[0040] Since the business processing durations corresponding to different operation types are different, the storage management device can obtain the business processing duration corresponding to the operation type according to the operation type. Further, the storage management device can use the third timestamp as the starting point and the business processing duration as the added duration to obtain the first timestamp, that is, obtain the completion timestamp (Time To Live, TTL) corresponding to the target request. Before the moment corresponding to the completion timestamp (including the moment corresponding to the completion timestamp), the storage management device completes processing the target request, which can be considered to be the timely completion of the target request. Before the moment corresponding to the completion timestamp, the storage management device has not completed processing the target request, which can be considered to be a processing timeout.

[0041] Finally, the storage management device can construct a target primary request management object corresponding to the target request based on the object identification information, the first timestamp, and the operation type. Alternatively, the target primary request management object can also include the parsed address information and write data. This allows only the primary request management object to be distributed to each level.

[0042] In this way, by constructing the main request management object, the business processing process of the request can be monitored and whether the request has timed out can be determined in a timely manner, so that the timed out request and the business level where the timed out request is located can be processed accordingly in a timely manner.

[0043] For example, the data structure of the target main request management object can be as follows:

[0044] {

[0045] Object identification information;

[0046] First timestamp;

[0047] Operation type;

[0048] Address information;

[0049] Operation field;

[0050] Timeout indication information;

[0051] …

[0052] }

[0053] The operation field is used to fill in write data when the operation type is a write operation type, and to fill in data read from the disk of the storage array when the operation type is a read operation.

[0054] In some optional implementations, the service processing durations of different operation types may also be consistent, for example, they may all be 300 seconds.

[0055] In some optional implementations, the storage management device may first obtain its own load situation and determine the service processing duration based on its own load situation and operation type, which may specifically include:

[0056] Obtain the utilization rate of the central processing unit (CPU), the utilization rate of the memory, and the average processing time of requests within a preset time period before the current moment, included in the storage management device. According to a preset normalization rule, convert the utilization rate of the CPU, the utilization rate of the memory, and the average processing time of requests into a preset interval. The preset interval can be [0, 1]. Further, calculate the average value of the three indicator values (CPU utilization, memory utilization, and average processing time of requests) converted into the preset interval to obtain a load value. According to the operation type, obtain a correction factor corresponding to the operation type.

[0057] For example, you can use the following expression:

[0058] T expected =T base ×(1+α·L)×β op

[0059] Among them, T expected is the business processing time, T base is the preset basic processing time, α is the preset load impact coefficient, which can be determined according to the storage type of the storage management device, L is the load value, and β op is the correction factor corresponding to the operation type.

[0060] In this way, the estimated service processing time can be adjusted more flexibly and accurately based on real-time load conditions. This allows for more accurate determination of request processing timeouts in subsequent processes, avoiding resource waste caused by unnecessary fault recovery operations and reducing service downtime.

[0061] In some optional implementations, after completing the operation of constructing the target primary request management object, the storage management device may perform a business processing process for the target request. To facilitate subsequent determination of which part of the target request has timed out, in addition to performing corresponding business processing operations, each business-level processing may also perform an operation of generating a backup request management object. Accordingly, the specific processing of the target request by the storage device may further include the following steps:

[0062] In the forward business processing of the current business level, when the current business level is the highest business level, the storage management device can utilize the business program in the current business level to perform corresponding business processing operations based on the address information and operation type, or based on the address information, operation type and write data, and obtain the forward processing result of the current business level. Alternatively, when the current business level is not the highest business level, the storage management device can utilize the business program in the current business level to perform corresponding business processing operations based on the address information and operation type, as well as the forward processing result of the third business level (a business level one level higher than the current business level), or based on the address information, operation type, write data, and the forward processing result of the third business level, and obtain the forward processing result of the current business level. After completing the business processing operation of the current business level, the storage management device can send the forward processing result of the current business level and the target main request management object to the fourth business level that is one level lower than the current business level, and then perform business processing of the fourth business level.

[0063] In addition, the storage management device can obtain a callback function corresponding to the current service level, add the callback function to the first field of the target primary request management object, and generate a backup request management object corresponding to the current service level.

[0064] For example, the data structure of a backup request management object can be as follows:

[0065] {

[0066] Object identification information;

[0067] First timestamp;

[0068] Operation type;

[0069] Address information;

[0070] Operation field;

[0071] callback function;

[0072] Timeout indication information;

[0073] …

[0074] }

[0075] The storage management device can obtain the reverse processing results of the current business level and feed them back to the third business level, that is, enter the reverse business processing process. In the process of reverse business processing, the storage management device can use the business program of the current business level to perform business processing on the reverse processing results fed back from the lower level, obtain the reverse processing results of the current business level, and use the callback function in the backup request management object of the current business level to feed back the reverse processing results of the current business level to the higher business level. After completing the feedback operation, the storage management device can delete the backup request management object of the target request at the current business level. Similarly, after completing the business processing of the highest business level, the storage management device can feed back the reverse processing results generated by the highest business level to the processing device. At this point, the business processing process of the target request is completed.

[0076] In some optional implementations, the storage management device may also update the second field in the target main request management object to the identification information of the current business level when performing business processing at the current business level, so that the faulty business level can be accurately determined subsequently.

[0077] Since a failure may occur at any business level during the above process, causing the target request to time out, and the at least one request that has not been processed in the storage management device may include both timed-out requests and non-timed-out requests, the storage management device can periodically traverse the primary request management objects corresponding to the at least one request that has not been processed, find the timed-out requests, and process them accordingly. Taking the target primary request management object as an example, the storage management device can extract the first timestamp and object identification information from a specified field in the target primary request management object.

[0078] In some optional implementations, when the current business level is the Redundant Array of Independent Disks (RAID) business level, the storage management device can use the business program of the current business level to split the target request, obtain multiple sub-requests corresponding to the target request, determine the address information corresponding to each sub-request, and create a corresponding sub-request management object for each sub-request.

[0079] The address information corresponding to each sub-request may include disk identification information on the storage array corresponding to the storage array identification information, a start address on the disk corresponding to the disk identification information, and an operation length.

[0080] The process of splitting a target request may include:

[0081] According to the starting address on the storage array and the preset stripe length, the stripe identifier and the starting address on the stripe corresponding to the stripe identifier are determined; according to the offset on the stripe corresponding to the stripe identifier and the preset block size, the block identification information (corresponding to the disk identification information) and the starting address on the block corresponding to the block identification information are determined; according to the block identification information (corresponding to the disk identification information) and the starting address on the block corresponding to the block identification information, the target request is split to obtain multiple sub-requests.

[0082] The process of creating a subrequest management object may include:

[0083] Step 1: Create corresponding sub-object identification information for each sub-request based on the object identification information.

[0084] For example, if the object identification information is "1", the identification information of multiple sub-objects may be "1-1", "1-2", "1-3", and "1-4" respectively.

[0085] Step 2: Generate a sub-request management object corresponding to each sub-request based on the sub-object identification information and address information corresponding to each sub-request and the target main request management object.

[0086] For example, a subrequest management object could look like this:

[0087] {

[0088] Sub-object identification information;

[0089] First timestamp;

[0090] Operation type;

[0091] Address information;

[0092] Operation field;

[0093] callback function;

[0094] Timeout indication information;

[0095] …

[0096] }

[0097] Step S202 : When it is determined that the first timestamp is less than the second timestamp corresponding to the current moment, the fault service level is determined according to a preset fault detection rule and a reference object corresponding to the preset fault detection rule.

[0098] The faulty service level may be any service level among multiple service levels through which any request is completed, and the reference object may be the target main request management object or object identification information.

[0099] Specifically, the storage management device can obtain a second timestamp corresponding to the current moment (the current moment here can be the moment when the first timestamp is extracted), and compare the second timestamp corresponding to the current moment with the first timestamp. If the first timestamp is greater than or equal to the second timestamp, it means that the target request has not timed out and there is no need to perform timeout processing on the target request. If the first timestamp is less than the second timestamp, it means that the target request has timed out and timeout processing can be performed on the target request.

[0100] The storage management device may pre-set different fault detection rules. Accordingly, under different preset fault detection rules, the preset fault detection rules and their corresponding reference objects may be used to determine the fault service level, specifically including the following two methods.

[0101] Method 1: The reference object is the target main request management object.

[0102] Step 1: Extract business-level identification information from the target primary request management object.

[0103] Step 2: Determine the first service level corresponding to the identification information of the service level as the faulty service level.

[0104] Specifically, corresponding to the solution in which the storage management device updates the second field to the identification information of the current business level where the target request is located in step S201, accordingly, when the target request is detected to have timed out, the storage management device can directly extract the identification information of the business level from the second field of the target main request management object, and determine the first business level corresponding to the extracted identification information of the business level as the faulty business level.

[0105] In this way, since the target main request management object directly contains the identification information of the current business level (through the second field), when a request timeout is detected, the business level where the fault occurred can be directly extracted from it without the need for additional traversal or search operations, which is more efficient.

[0106] Method 2: The reference object is object identification information.

[0107] Step 1: According to the object identification information, determine, from multiple business levels, a second business level where the backup request management object corresponding to the object identification information is located.

[0108] Step 2: determine the second service level as the fault service level.

[0109] Specifically, the storage management device can traverse each business level separately according to the level of the business level. Each time a business level is traversed, it can determine whether the traversed business level includes the backup request management object corresponding to the object identification information based on the object identification information. If so, it means that the storage management device is performing the business processing corresponding to the traversed business level, and the traversed business level can be determined as a faulty business level. If not, it means that the storage management device is not performing the business processing corresponding to the traversed business level, and the next business level can be traversed. And so on, until the faulty business level is determined. The second business level is any of the above-mentioned traversed business levels.

[0110] In this way, by checking whether there is a backup request management object corresponding to the object identification information at each business level, it is ensured that no potential failure points are missed, which is especially suitable for application scenarios with multi-level dependencies.

[0111] In the method below, the fault business level determined is one.

[0112] In the second approach, the determined faulty service level may include one or more. Referring to the aforementioned service processing of the target request, the backup request management object in the service level is deleted only after both the forward and reverse service processing of the service level are completed. Therefore, there may be multiple service levels including the backup request management object corresponding to the first identification information.

[0113] Step S203: recover the faulty service layer.

[0114] Specifically, the business layer may include one or more backup request management objects and business programs corresponding to the business layer. If a timed-out request occurs at the business layer, the business processing of the request needs to be terminated, and the business layer needs to be checked and restored so as to continue subsequent processing. Accordingly, step S203 may specifically include the following steps:

[0115] Step 1: Perform fault marking processing on the backup request management object corresponding to the object identification information in the fault business layer.

[0116] Step 1: Fill the first field of the backup request management object corresponding to the object identification information with the first preset indication information.

[0117] Among them, the first field is the field corresponding to the callback function. The callback function is used to call back a third business level that is one level higher than the current business level after successfully completing the processing of the faulty business level to perform business processing corresponding to the third business level. The first preset indication information is used to indicate that calling the callback function is prohibited.

[0118] Step 2: Setting the timeout indication information included in the first backup request management object as second preset indication information.

[0119] The second preset indication information is used to indicate that the service processing of the target request has timed out. For example, the second preset indication information may be "Ture".

[0120] Specifically, the storage management device may fill the first field of the backup request management object corresponding to the object identification information at the faulty business layer with a blank. In this way, even if the storage management device receives a reverse processing result from the next business layer, it may not provide feedback to the higher business layer, thereby blocking the path for returning the processing result of the target request. In addition, the storage management device may set the timeout indication information included in the first backup request management object to the second preset indication information.

[0121] First, by filling the first field (i.e., the field where the callback function is located) with the first preset indication information (for example, setting it to empty or a specific value), it is possible to effectively prevent the callback of higher-level business logic after the successful completion of the faulty business layer processing. This can avoid the chain reaction caused by the underlying failure, prevent the error from propagating upward, and protect other business layers from being affected. In addition, even if the reverse processing result is received from the next business layer, since the callback function is disabled, the storage management device will not continue to feedback the processing result to the higher business layer, which can reduce the redundant operations of the storage management device, simplify the recovery process, and improve the overall recovery efficiency. Secondly, setting the timeout indication information to the second preset indication information (such as "True") can clearly identify that a timeout failure has occurred in the current request. This clear status identification helps with subsequent monitoring, logging, and troubleshooting, making it easier for operation and maintenance personnel to quickly locate the problem.

[0122] Step 2: Recover the business program in the faulty business layer.

[0123] Step 1: Identify the running status of the business program at the faulty business level.

[0124] Step 2: When the operating state is the first preset state, directly perform self-checking and recovery operations on the faulty service layer.

[0125] The first preset state is used to indicate that the fault service layer is running normally.

[0126] Step 3: Alternatively, when the operating state is the second preset state, trigger the fault service layer to enter the third preset state and perform self-checking and recovery operations.

[0127] The second preset state is a stuck state caused by a timeout in the service processing of the target request at the faulty service layer, and the third preset state is a state after the second preset state ends. For example, the second preset state may be a stuck state in a silent state, and the third preset state may be a silent state.

[0128] Step 3: Reclaim the resources occupied by the target request.

[0129] For example, the resource may be a memory resource.

[0130] Specifically, the storage management device can first identify the operating status of the service program of the faulty service layer. If the operating status is the first preset state, the faulty service layer is operating normally and can directly perform self-test and recovery operations. If the operating status is the second preset state, the faulty service layer can be triggered to enter the second preset state. Because the target request has been blocked, the faulty service layer will not be stuck after re-entering the second preset state. Instead, it can exit the second preset state and enter the third preset state to perform self-test and recovery operations.

[0131] When the business layer is in the first preset state (i.e., operating normally), self-checking and recovery operations can be performed directly. For the second preset state that is stuck due to a timeout, by triggering the entry into the third preset state (e.g., the state after the stuck state ends), the business layer can resume the normal processing flow. This method ensures that normal operation can be restored as soon as possible even in the event of an abnormal situation. After completing the self-checking and recovery operations, the resources occupied by the target request are recycled, which can prevent memory leaks and other resource waste problems and extend the stable operation time of the storage management device.

[0132] In some optional embodiments, when the faulty service level is a RAID service level, before reclaiming the resources occupied by the target request, the storage management device may further determine whether the faulty service level also includes a sub-request management object corresponding to the object identification information. If it is determined that a sub-request management object corresponding to the object identification information exists, the callback function in each sub-request management object may be set to the first preset indication information, and the resources occupied by each sub-request management object may be reclaimed.

[0133] First, by checking and processing the sub-request management object, it can be ensured that not only the main request management object, but also all its related sub-request management objects are correctly marked and cleaned up. This can completely isolate the scope of the fault impact and prevent subsequent problems caused by unprocessed sub-requests. Setting the callback function in each sub-request management object to the first preset indication information (for example, setting it to empty or a specific value) can effectively prevent any operation that attempts to call back higher-level business logic. This avoids the chain reaction caused by the underlying fault and protects other business levels from being affected. Secondly, not only the resources occupied by the main request are recycled, but also the resources occupied by each sub-request management object are recycled, which can ensure that resources are effectively released, reduce the risk of memory leaks, and improve the overall performance and stability of the system. Finally, through detailed inspection and processing of sub-request management objects, the storage management device's control ability over complex business levels (such as RAID) is enhanced. This fine-grained fault handling mechanism improves the reliability and robustness of the storage management device, allowing the storage device to maintain efficient operation even when a local fault occurs.

[0134] In some optional implementations, fault information is generated based on the object identification information and the identification information of the faulty service level. The fault information is sent to the processing device to instruct the processing device to reissue the target request. In this way, even timed-out requests can be processed promptly and the corresponding operation completed.

[0135] The fault recovery method of the embodiment of the present application can extract the completion timestamp from the corresponding main request management object for the request that has not been completed, and determine whether the completion timestamp is less than the second timestamp corresponding to the current moment. If so, it means that the request has timed out, and there may be a failure in the business layer. At this time, the faulty business layer can be determined based on the preset fault detection rules and the reference object corresponding to the preset fault detection rules, and the faulty business layer can be accurately restored. In this way, there is no need to hot restart the storage management device, which can avoid the data loss problem caused by hot restart and improve the stability of the storage system. In addition, the efficiency of recovering the faulty business layer is higher than that of hot restart, especially for storage management devices with frequent business, which can greatly reduce the duration of business stagnation.

[0136] The following describes in detail the execution process of the above fault recovery method using a specific example.

[0137] When the storage management device processes a request through multiple service layers including the protocol service layer (Host Layer, HL) interacting with the host (i.e., the above-mentioned processing device), the data fast read cache service layer (Cache, CA), the virtualization service layer (Virtualization Group, VG), and the RAID service layer, the processing flow of the request sent by the host can be as follows: Figure 3 As shown, the specific steps are as follows:

[0138] After receiving a request, the HL business layer in the storage management device assigns identification information and a completion timestamp to the request. Based on the address information, operation information, completion timestamp, and identification information in the request, it encapsulates a main request management object corresponding to the request. This allows the storage management device to initiate request timeout detection. After processing the request, the HL business layer sends the request to the CA business layer in the storage management device. The CA business layer then sends the request to the VG business layer in the storage management device. The VG business layer then sends the request to the RAID business layer in the storage management device. The RAID business layer splits the request into multiple sub-requests based on the disk and sends each sub-request to the corresponding disk for read and write operations. Furthermore, the RAID business layer can assign sub-object identification information to each sub-request and generate a request management object corresponding to each sub-request. This processing corresponds to the forward business processing described above.

[0139] After the disk completes the read and write operations corresponding to each sub-request, it feeds the results back to the RAID service layer. Based on the results of all the sub-requests, the RAID service layer constructs a total processing result corresponding to the request and feeds it back to the VG service layer. The VA service layer then processes the sub-requests and feeds the total processing result back to the CA service layer. The CA service layer then processes the sub-requests and feeds the total processing result back to the HL service layer. The HL service layer then processes the sub-requests and feeds the total processing result back to the processing device. This process corresponds to the reverse process described above.

[0140] Furthermore, during the forward business processing of each business level, the storage management device can generate a corresponding backup request management object for each business level. During the reverse business processing of each business level, after feeding back the overall processing result to the next higher business level, the storage management device can delete the backup request management object generated during the forward business processing.

[0141] like Figure 4As shown, the HL business layer of the storage management device can periodically traverse at least one request that has not been completed. Every time a request is traversed, it can determine whether the traversed request has timed out based on the timestamp corresponding to the current moment and the completion timestamp included in the main request management object corresponding to the traversed request. If it times out, the HL business layer can extract the object identification information from the main request management object, broadcast the object identification information to each business layer, and the business program of each business layer will perform timeout processing.

[0142] The process of timeout handling at each business level can be as follows: Figure 5 As shown, the specific steps are as follows:

[0143] The business program of the business layer determines whether the current business layer includes a backup request management object corresponding to the object identification information based on the object identification information. If so, it determines that a timeout request is detected in the current business layer. If not, it determines that no timeout request is detected in the current business layer. When it is determined that a timeout request is detected in the current business layer, the path for the timeout request to return the processing result to the previous business layer can be blocked, and the processing of the timeout request can be directly terminated to reclaim the resources occupied by the timeout request. Then, the running status of the business program of the current business layer is checked again to determine whether the running status is in a silent state. If so, the silent state is re-triggered. If not, the self-check and recovery operation is directly performed. Alternatively, when it is determined that no timeout request is detected in the current business layer, the self-check and recovery operation can also be directly performed.

[0144] like Figure 6 As shown, in the RAID business level, when it is determined that a timeout request is detected, a cyclic detection can be performed based on the object identification information to determine whether there is a sub-request management object corresponding to the object identification information. When it is determined that there is a sub-request management object corresponding to the object identification information, the path for the sub-request to return the processing result to the upper business level can be blocked, the processing of the sub-request can be terminated, and the resources occupied by the sub-request can be reclaimed.

[0145] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0146] The embodiment of the present application also provides a fault recovery device, such as Figure 7 Shown, including:

[0147] An extraction module 710 is configured to extract a first timestamp and object identification information from a pre-acquired target primary request management object, wherein the target primary request management object is a primary request management object corresponding to a target request, and the target request is any one of the at least one unprocessed request;

[0148] Determination module 720 is configured to, when it is determined that the first timestamp is less than a second timestamp corresponding to the current moment, determine a faulty service level based on a preset fault detection rule and a reference object corresponding to the preset fault detection rule, wherein the faulty service level is any service level among multiple service levels passed through to complete any request, and the reference object is a target primary request management object or object identification information;

[0149] The recovery module 730 is used to recover the faulty service layer.

[0150] In some optional implementations, when the reference object is a target primary request management object, the determination module 720 is specifically configured to:

[0151] Extracting business-level identification information from the target primary request management object;

[0152] The first service level corresponding to the identification information of the service level is determined as the faulty service level.

[0153] In some optional implementations, when the reference object is object identification information, the determination module 720 is specifically configured to:

[0154] determining, according to the object identification information, a second business level at which the backup request management object corresponding to the object identification information is located among the multiple business levels;

[0155] The second service level is determined as the fault service level.

[0156] In some optional implementations, the recovery module 730 is specifically configured to:

[0157] Performing fault marking processing on the backup request management object corresponding to the object identification information in the fault business layer;

[0158] Recover the business programs in the faulty business layer.

[0159] In some optional implementations, the recovery module 730 is specifically configured to:

[0160] Filling a first field of the backup request management object corresponding to the object identification information with first preset indication information, wherein the first field is a field corresponding to a callback function, and the callback function is used to, after successfully completing processing of the current business level, call back a third business level that is one level higher than the current business level to perform business processing corresponding to the third business level, and the current business level is any one of the multiple business levels, and the first preset indication information is used to indicate that calling the callback function is prohibited;

[0161] And, the timeout indication information included corresponding to the object identification information is set as second preset indication information, wherein the second preset indication information is used to indicate that the service processing of the target request has timed out.

[0162] In some optional implementations, the recovery module 730 is specifically configured to:

[0163] Identify the operational status of business programs in the faulty business layer;

[0164] When it is determined that the operating state is a first preset state, directly performing a self-check and recovery operation on the faulty service layer, wherein the first preset state is used to indicate that the faulty service layer is operating normally;

[0165] Alternatively, when the running state is the second preset state, the faulty service layer is triggered to enter the third preset state, and then self-check and recovery operations are performed, wherein the second preset state is a stuck state of the faulty service layer due to a timeout in the service processing of the target request, and the third preset state is a state after the second preset state ends;

[0166] Reclaim the resources occupied by the target request.

[0167] In some optional implementations, the apparatus further includes an acquisition module 740, configured to:

[0168] When receiving a target request sent by the processing device, obtaining a third timestamp corresponding to the moment when the target request is received, and assigning object identification information to the target request, wherein the target request includes an operation type;

[0169] According to the operation type, obtain the business processing time corresponding to the operation type;

[0170] Determine the first timestamp based on the third timestamp and the service processing duration corresponding to the operation type;

[0171] A target main request management object corresponding to the target request is constructed according to the object identification information, the first timestamp, and the operation type.

[0172] For the description of the features in the embodiment corresponding to the fault recovery device, reference can be made to the relevant description of the embodiment corresponding to the fault recovery method, which will not be repeated here.

[0173] The embodiment of the present application also provides an electronic device, such as Figure 8 As shown, it includes a memory 10 and a processor 20, the memory 10 stores a computer program, and the processor 20 is configured to run the computer program to execute the steps in any of the above-mentioned fault recovery method embodiments.

[0174] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned fault recovery method embodiments when running.

[0175] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0176] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned fault recovery method embodiments are implemented.

[0177] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned fault recovery method embodiments are implemented.

[0178] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0179] The above is a detailed introduction to a fault recovery method, device, electronic device, storage medium, and program product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A fault recovery method, characterized in that: The method comprises: Extracting a first timestamp and object identification information from a pre-acquired target primary request management object, wherein the target primary request management object is a primary request management object corresponding to a target request, and the target request is any one of the at least one request that has not been completely processed; When it is determined that the first timestamp is less than a second timestamp corresponding to the current moment, determining a faulty service level according to a preset fault detection rule and a reference object corresponding to the preset fault detection rule, wherein the faulty service level is any one of multiple service levels passed through to complete any request, and the reference object is the target primary request management object or the object identification information; Perform recovery processing on the faulty service layer.

2. The fault recovery method according to claim 1, characterized in that: When the reference object is the target primary request management object, when it is determined that the first timestamp is less than a second timestamp corresponding to the current moment, determining the fault service level according to a preset fault detection rule and the reference object corresponding to the preset fault detection rule includes: Extracting identification information of the business level from the target main request management object; A first service level corresponding to the identification information of the service level is determined as the faulty service level.

3. The fault recovery method according to claim 1, characterized in that: When the reference object is the object identification information, when it is determined that the first timestamp is less than a second timestamp corresponding to the current moment, determining the fault service level according to a preset fault detection rule and a reference object corresponding to the preset fault detection rule includes: determining, according to the object identification information, a second business level at which the backup request management object corresponding to the object identification information is located among the multiple business levels; The second service level is determined as the fault service level.

4. The fault recovery method according to any one of claims 1 to 3, characterized in that: The recovering process of the faulty service layer includes: Performing fault marking processing on the backup request management object corresponding to the object identification information in the fault service layer; Recovering the service program in the faulty service layer; Reclaim the resources occupied by the target request.

5. The fault recovery method according to claim 4, characterized in that: The performing fault marking processing on the backup request management object corresponding to the object identification information in the fault service layer includes: Filling a first field of a backup request management object corresponding to the object identification information with first preset indication information, wherein the first field is a field corresponding to a callback function, the callback function being configured to, after successfully completing processing of a current service level, call back a third service level that is one level higher than the current service level to perform service processing corresponding to the third service level, the current service level being any one of the plurality of service levels, and the first preset indication information being configured to indicate prohibition of calling the callback function; And, setting the timeout indication information corresponding to the object identification information as second preset indication information, wherein the second preset indication information is used to indicate that the service processing of the target request has timed out.

6. The fault recovery method according to claim 4, characterized in that: The recovering process of the service program in the faulty service layer includes: Identifying the running status of the service program in the fault service layer; When it is determined that the operating state is a first preset state, directly performing a self-check and recovery operation on the faulty service layer, wherein the first preset state is used to indicate that the faulty service layer is operating normally; Alternatively, when the operating state is the second preset state, the faulty business layer is triggered to enter the third preset state, and then self-check and recovery operations are performed, wherein the second preset state is the stuck state of the faulty business layer due to the timeout of the business processing of the target request, and the third preset state is the state after the second preset state ends.

7. The fault recovery method according to any one of claims 1 to 3, characterized in that: Before extracting the first timestamp and object identification information from the pre-acquired target main request management object, the method further includes: When receiving the target request sent by the processing device, obtaining a third timestamp corresponding to the moment when the target request is received, and assigning the object identification information to the target request, wherein the target request includes an operation type; According to the operation type, obtaining the service processing duration corresponding to the operation type; Determine the first timestamp according to the third timestamp and the service processing duration corresponding to the operation type; A target main request management object corresponding to the target request is constructed according to the object identification information, the first timestamp, and the operation type.

8. A fault recovery device, characterized in that: include: an extraction module, configured to extract a first timestamp and object identification information from a pre-acquired target primary request management object, wherein the target primary request management object is a primary request management object corresponding to a target request, and the target request is any one of at least one request that has not been completely processed; a determination module configured to, when determining that the first timestamp is less than a second timestamp corresponding to the current moment, determine a faulty service level according to a preset fault detection rule and a reference object corresponding to the preset fault detection rule, wherein the faulty service level is any one of a plurality of service levels passed through to complete any request, and the reference object is the target primary request management object or the object identification information; A recovery module is used to recover the faulty service layer.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the fault recovery method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the fault recovery method according to any one of claims 1 to 7.

Images