Method and device for repairing storage system failure, electronic equipment, medium and product

Abstract

The application discloses a storage system fault repairing method and device, electronic equipment, medium and product, and relates to the technical field of storage systems. When an external request initiated by an external initiator of a storage system is initiated to the storage system and needs to be processed by the storage system, the external request is processed through a complex call chain. The storage system is monitored, and an alarm event of the storage system is captured. A request identifier in the alarm event needs to be obtained. From a pre-constructed request track database, track data corresponding to the request identifier is obtained. When the alarm event is captured, the track data corresponding to the request identifier is aggregated, and a fault feature sequence is constructed according to the track data. From a pre-constructed fault repairing database, a repairing strategy corresponding to the fault feature sequence is obtained, and the repairing of a known fault can be realized. This process does not need manual analysis of a call link, speculation of a fault reason, and manual intervention for completing the repairing, and the efficiency and accuracy of fault repairing are improved.

Description

Technical Field

[0001] This application relates to the field of storage system technology, and in particular to methods, apparatus, electronic devices, media and products for repairing storage system faults. Background Technology

[0002] In modern enterprise-level storage systems, as business scales up and data volumes surge, storage systems typically consist of multiple distributed services, databases, and middleware components. When external requests originating from outside the storage system require its processing, these components collaborate through complex call chains to complete data storage and processing tasks. When storage system failures occur, such as service timeouts and database query failures, operations and maintenance personnel need to sift through massive amounts of logs and alarm data to pinpoint the cause of the failure.

[0003] However, storage systems generate a large number of log records during operation. When an alarm occurs, operations and maintenance personnel need to manually analyze information such as call chains, error codes, and timestamps to deduce the possible causes of the failure and take manual intervention measures to repair it. This process is not only time-consuming and labor-intensive, but also prone to errors in analysis due to human oversight, which reduces the efficiency and accuracy of fault repair. Summary of the Invention

[0004] This application provides methods, apparatus, electronic devices, media, and products for repairing storage system faults, in order to at least address the problems of low efficiency and accuracy in fault repair in related technologies.

[0005] This application provides a method for repairing storage system faults, including: monitoring the storage system and capturing alarm events of the storage system; obtaining request identifiers from the alarm events; wherein the request identifiers are generated for external requests to the storage system; obtaining trajectory data corresponding to the request identifiers from a pre-built request trajectory database; wherein the pre-built request trajectory database stores trajectory data of different external requests; constructing a fault feature sequence of the alarm events based on the trajectory data corresponding to the request identifiers; obtaining repair strategies corresponding to the fault feature sequences from a pre-built fault repair database; and repairing the alarm events according to the repair strategies.

[0006] This application also provides a storage system fault repair apparatus, comprising: a capture module for monitoring the storage system and capturing alarm events of the storage system; a first acquisition module for acquiring request identifiers from the alarm events, wherein the request identifiers are generated for external requests to the storage system; a second acquisition module for acquiring trajectory data corresponding to the request identifiers from a pre-built request trajectory database, wherein the pre-built request trajectory database stores trajectory data of different external requests; a construction module for constructing a fault feature sequence of the alarm events based on the trajectory data corresponding to the request identifiers; a third acquisition module for acquiring repair strategies corresponding to the fault feature sequences from a pre-built fault repair database; and a repair module for repairing the alarm events according to the repair strategies.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described methods for repairing memory system failures when executing the computer program.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described methods for repairing storage system faults.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described methods for repairing storage system faults.

[0010] This application enables these components to collaborate through a complex call chain to complete data storage and processing tasks when an external request originates from outside the storage system and requires processing by the storage system. The storage system is monitored, and alarm events are captured. Since these alarm events are caused by the storage system processing external requests, it is necessary to obtain the request identifiers involved in the alarm events. Trajectory data corresponding to the request identifiers is obtained from a pre-built request trajectory database. Upon capturing an alarm event, the trajectory data related to the request identifiers is aggregated, and a fault feature sequence is constructed based on the trajectory data. Subsequently, the repair strategy corresponding to the fault feature sequence is obtained from a pre-built fault repair database, enabling automatic repair of known faults. This process eliminates the need for manual analysis of the call chain, inference of the fault cause, and manual intervention to complete the repair, improving the efficiency and accuracy of fault location and repair. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A schematic diagram illustrating a scenario for a storage system fault repair method provided in an embodiment of this application;

[0013] Figure 2 A flowchart illustrating the storage system fault repair method provided in this application embodiment. Figure 1 ;

[0014] Figure 3 A flowchart illustrating the storage system fault repair method provided in this application embodiment. Figure 2 ;

[0015] Figure 4 A flowchart illustrating the storage system fault repair method provided in this application embodiment. Figure 3 ;

[0016] Figure 5 A flowchart illustrating the storage system fault repair method provided in this application embodiment. Figure 4 ;

[0017] Figure 6 A schematic diagram of the structure of a storage system fault repair device provided in an embodiment of this application;

[0018] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0020] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0021] To address the aforementioned technical issues, this application proposes the following technical concept: Pre-constructing a request trajectory database and a fault repair database. The pre-constructed request trajectory database stores trajectory data for different external requests; the pre-constructed fault repair database stores multiple known fault characteristic sequences and corresponding repair strategies for each known fault characteristic sequence. When an external request originating from outside the storage system initiates a request that requires processing by the storage system, a unique request identifier is generated for each external request, and this request identifier is transmitted throughout the entire request chain. When an alarm event of the storage system is captured, only the request identifier in the alarm event needs to be obtained, and the trajectory data corresponding to the request identifier can be retrieved from the pre-constructed request trajectory database. Based on the trajectory data corresponding to each request identifier, a fault characteristic sequence for the alarm event is constructed; the repair strategy corresponding to the fault characteristic sequence is retrieved from the pre-constructed fault repair database, and repair can then be performed. This process eliminates the need for manual analysis of the call chain, inference of the fault cause, and manual intervention to complete the repair, thus improving the efficiency and accuracy of fault location and repair.

[0022] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] The specific application environment architecture or specific hardware architecture on which the execution of the storage system failure repair method depends is described here.

[0024] refer to Figure 1 , Figure 1 This is a schematic diagram illustrating a scenario for a storage system fault repair method provided in an embodiment of this application. The electronic device provided in this embodiment includes: a receiving device 101, a processing device 102, and a display device 103.

[0025] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the method for repairing storage system failures. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0026] In the specific implementation process, the receiving device 101 can be an input / output interface or a communication interface, and can capture alarm events of the storage system.

[0027] The processing device 102 can obtain a request identifier from the alarm event, perform a series of processes on the request identifier to obtain a repair strategy, and repair the alarm event according to the repair strategy.

[0028] The display device 103 can be used to display the repair results.

[0029] It should be understood that the aforementioned processor can be implemented by reading instructions from memory and executing those instructions, or it can be implemented through chip circuitry.

[0030] Furthermore, the network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0031] Figure 2 A flowchart illustrating the storage system fault repair method provided in this application embodiment. Figure 1 ,like Figure 2 As shown, embodiments of this application provide a method for repairing storage system faults. The method is described in detail below:

[0032] S201: Monitor the storage system and capture alarm events from the storage system.

[0033] In this embodiment, the core metrics that the storage system needs to monitor include system status metrics, such as disk utilization, memory usage, and service liveness; request processing metrics, such as request response timeouts, error codes, and database interaction failures; and log metrics, such as the frequency of error log keywords.

[0034] Specifically, the aforementioned core indicators are collected through real-time monitoring tools. Each indicator collected in real-time is compared with a preset threshold. If any indicator exceeds the threshold for two consecutive collection periods, it is considered an alarm event. The threshold is set according to the actual situation.

[0035] S202: Obtain the request identifier in the alarm event; where the request identifier is generated for external requests to the storage system.

[0036] In this embodiment, when an external request is initiated by an initiator from outside the storage system and needs to be processed by the storage system, a unique request identifier is generated for each external request to ensure that the request identifiers of different external requests are not duplicated. This request identifier will be transmitted transparently throughout the entire request chain.

[0037] In this embodiment, automatic pass-through in the external request call chain is achieved through hook functions in the system framework, message queue client, and database driver, ensuring that the request identifier follows the external request through any service, thread, and asynchronous task without being lost or tampered with, thus avoiding tracing chain breaks.

[0038] In this embodiment, for different external requests, a request identifier is generated at the storage system entry point or the first service node to ensure that any external request can be bound to a request identifier as soon as it enters the storage system, thus avoiding the loss of tracking due to the lack of a request identifier after the external request enters the internal storage system entry point or the first service node.

[0039] For example: if the request entry point of the storage system is the storage API gateway, then a request identifier is generated at the gateway layer; if there is no gateway, the request identifier is generated at the first service that receives the external request.

[0040] S203: Obtain the trajectory data corresponding to the request identifier from the pre-built request trajectory database; the pre-built request trajectory database stores trajectory data of different external requests.

[0041] In this embodiment, the pre-built request trajectory database is indexed by the request identifier as the primary key.

[0042] Specifically, using the request identifier as the primary key, the query interface of the pre-built request trajectory database is called to retrieve all trajectory data corresponding to the request identifier from the database, including the stage identifier, parent stage identifier, service name, business name, request status, and timestamp of any processing stage the external request has passed through.

[0043] The process identifier is a 64-bit identifier generated using a random number generator, such as a1b2c3d4, used to identify an operation within the service, including a function call and a database query. The parent process identifier refers to the process identifier that called the current processing process. It can be used to reconstruct the call tree; if the current process identifier is the first process, then there is no parent process identifier. The service name is the service name of the current processing request. The business name is the business name of the current operation.

[0044] The timestamp is in the microsecond range.

[0045] In this embodiment, a request identifier is generated using a random generator; a stage identifier is also generated using a random generator.

[0046] For example, for any external request, a request identifier f5a1b2c3d4e5f678 is generated. The first processing stage of the current external request is entering service A, generating a stage identifier a1b2c3d4. Upon entering service A, the service name auth-svc is generated, and the operation performed in service A generates the business name GetUserInfo with a timestamp of 1646640123456789. After processing in service A is completed, the request enters the next processing stage, service B, at which point a stage identifier e5f6g7h8 is generated. Because service A calls service B, the parent stage identifier is a1b2c3d4. Upon entering service B, the service name user-svc is generated, and the operation performed in service B generates the business name QueryMySQL with a timestamp of 1646640123456800.

[0047] S204: Construct a fault characteristic sequence of alarm events based on the trajectory data corresponding to the request identifier.

[0048] Specifically, multiple stage identifiers are obtained from the trajectory data corresponding to the request identifier; the trajectory data consists of data from different external requests at multiple processing stages; based on the trajectory data corresponding to the request identifier, the feature vectors corresponding to each stage identifier are obtained; based on the feature vectors corresponding to each stage identifier, the fault feature sequence of the alarm event is obtained.

[0049] In this embodiment, the service name, business name, request status, request result, involved database, and timestamp are obtained from the trajectory data corresponding to the request identifier. Based on the service name, business name, request status, request result, involved database, and timestamp, the feature vector corresponding to each stage identifier is obtained.

[0050] For example, the feature vector corresponding to each stage identifier is: [

[0052] service: user-svc,

[0053] operation: GET / api / v1 / user / {var},

[0054] http_status: 500,

[0055] error: true,

[0056] database: orders-db,

[0057] timestamp: 1646640123456789 ]

[0059] In this dataset, `service` represents the service name, which is `user-svc`; `operation` represents the business logic name, which is `GET / api / v1 / user / {var}`; `http_status` represents the request status, which is 500; `error` represents the request result, which is `true`; `database` represents the database involved, which is `orders-db`; and `timestamp` represents the timestamp, which is 1646640123456789. The feature vectors provide a clear visual representation of the data from different stages of an external request's processing.

[0060] In this embodiment, the purpose of step S204 is to transform the complete trajectory data of a single request into a sequence with a time-series relationship. For example, the feature vectors corresponding to the identifiers of each stage are sorted according to the order of timestamps to obtain the fault feature sequence of the alarm event: [

[0062] A: auth-svc:AuthUser,

[0063] B: user-svc:GET / user / {var} (success),

[0064] C: order-svc:GET / orders (timeout_error) ]

[0066] The fault characteristic sequence of the alarm event consists of four stages, executed sequentially on services A, B, and C. On service A, the service name is auth-svc, and the business name is AuthUser; on service B, the service name is user-svc, the business name is GET / user / {var}, and the request status is success; on service C, the service name is order-svc, the business name is GET / orders, and the request status is timeout_error.

[0067] S205: Obtain the repair strategy corresponding to the fault feature sequence from the pre-built fault repair database.

[0068] Specifically, multiple known fault feature sequences are obtained from a pre-built fault repair database; the pre-built fault repair database includes multiple known fault feature sequences and repair strategies corresponding to each known fault feature sequence; the multiple known fault feature sequences are matched sequentially with the fault feature sequence to obtain the matching degree of each known fault feature sequence; the repair strategies corresponding to the known fault feature sequences whose matching degree exceeds a preset matching degree threshold are determined as the repair strategies corresponding to the fault feature sequences.

[0069] In this embodiment, the fault feature sequence includes request status, service name, and service name.

[0070] For example, a known sequence of fault characteristics in a pre-built fault repair database is as follows: [

[0072] A: auth-svc:AuthUser,

[0073] B: user-svc: GET / user / {id} (success),

[0074] C: order-svc:GET / orders (slow) ]

[0076] In this embodiment, the step of obtaining the matching degree of each known fault feature sequence includes Sa~Se:

[0077] Sa: Obtain the same service name from multiple known fault feature sequences and fault feature sequences.

[0078] Sb: Determine the temporal order matching degree of each known fault feature sequence based on whether the temporal order of the same service name in each known fault feature sequence is the same as the temporal order of the same service name in the fault feature sequence; where the same temporal order and different temporal order correspond to different temporal order matching degrees.

[0079] In this embodiment, the timing order matching degree is 100% for those with the same timing order and 0% for those with different timing orders.

[0080] For example, in the known fault feature sequence and the fault feature sequence, the same service names are auth-svc, user-svc, and order-svc. If the temporal order of these three service names in the known fault feature sequence is the same as the temporal order in the fault feature sequence, then the temporal order matching degree is 100%.

[0081] Sc: Based on the same service name, determine the request state matching degree of each known fault feature sequence by comparing the request state corresponding to the same service name in each known fault feature sequence with the request state corresponding to the same service name in the fault feature sequence; where the same request state and different request states correspond to different request state matching degrees.

[0082] In this embodiment, the request status matching degree is 100% for requests with the same status. For requests with different statuses, the degree is determined based on the number of requests with the same service name and the number of requests with the same status. In this case, the request status matching degree = .in, This indicates the number of requests with the same status. This indicates the number of times the service name is the same.

[0083] For example, regarding the service name auth-svc, the corresponding request status in the known fault feature sequence is empty, and the corresponding request status in the fault feature sequence is empty; regarding the service name user-svc, the corresponding request status in the known fault feature sequence is success, and the corresponding request status in the fault feature sequence is success; regarding the service name order-svc, the corresponding request status in the known fault feature sequence is slow, indicating a slow query exception, and the corresponding request status in the fault feature sequence is timeout_error, indicating a timeout error. In this case, the request status matching degree = 2 / 3 × 100%.

[0084] Sd: Based on the same service name, determine the service name matching degree of each known fault feature sequence by comparing the service name corresponding to the same service name in each known fault feature sequence with the service name corresponding to the same service name in the fault feature sequence; where the same service name and different service names correspond to different service name matching degrees.

[0085] In this embodiment, the matching degree of service names with the same name is 100%. If the service names are different, the matching degree is determined based on the number of identical service names and the number of identical service names. In this case, the matching degree is = .in, This indicates the number of times the business name is the same. This indicates the number of times the service name is the same.

[0086] For example, regarding the service name auth-svc, the corresponding business name in the known fault feature sequence is AuthUser, and the corresponding business name in the fault feature sequence is also AuthUser; regarding the service name user-svc, the corresponding business name in the known fault feature sequence is GET / user / {id}, and the corresponding business name in the fault feature sequence is GET / user / {var}. The parameters are different—one is {id}, and the other is {var}—but both essentially call the same business name, only the input parameters differ; regarding the service name order-svc, the corresponding business name in the known fault feature sequence is GET / orders, and the corresponding business name in the fault feature sequence is also GET / orders. Therefore, the business name matching degree is 100%.

[0087] Se: Based on the temporal sequence matching degree, request status matching degree, business name matching degree, preset temporal sequence matching degree weight, preset request status matching degree weight, and preset business name matching degree weight, the matching degree of each known fault feature sequence is obtained through weighted fusion.

[0088] Optionally, the preset time sequence matching weight is 0.5, the preset request status matching weight is 0.3, and the preset business name matching weight is 0.2.

[0089] For example, the matching degree of this known fault feature sequence is 100%×0.5+2 / 3×100%×0.3+100%×0.2=90%.

[0090] Optionally, the matching threshold is 80%. If the current known fault feature sequence exceeds the matching threshold, then the repair strategy corresponding to the current known fault feature sequence is obtained. The matching threshold is set according to the actual situation.

[0091] In this embodiment, there is also a situation where the sequence length of a known fault feature sequence is inconsistent with the sequence length of the fault feature sequence. In this case, a sequence length correction coefficient can be introduced. For example, if the sequence lengths are not the same, the matching degree is multiplied by the sequence length correction coefficient to obtain the final matching degree. Optionally, the sequence length correction coefficient can be 0.9. The sequence length correction coefficient is determined according to the actual situation.

[0092] Optionally, if there is no repair strategy corresponding to the fault feature sequence in the pre-built fault repair database, the fault feature sequence is sent to the operation and maintenance terminal so that the operation and maintenance terminal can perform manual repair based on the fault feature sequence; the manual repair strategy sent by the operation and maintenance terminal is received; and the fault feature sequence and the manual repair strategy are saved to the pre-built fault repair database to update the pre-built fault repair database.

[0093] In this embodiment, if the matching degree of each known fault feature sequence in the pre-built fault repair database is less than the matching degree threshold, then there is no repair strategy corresponding to the fault feature sequence in the pre-built fault repair database.

[0094] In this embodiment, when there is no repair strategy corresponding to the fault feature sequence in the pre-built fault repair database, the pre-built fault repair database will be updated, and the fault scenarios that can be dealt with will be gradually expanded, thereby improving the system availability.

[0095] Optionally, if the pre-built fault repair database does not contain a repair strategy corresponding to the fault feature sequence, a preset emergency repair strategy, such as service restart, can be adopted. This ensures that basic repair capabilities are still provided in unknown fault scenarios, preventing the storage system from being unavailable for extended periods.

[0096] S206: Repair the alarm event according to the repair strategy.

[0097] In summary, when an external request originates from outside the storage system and requires its processing, these components collaborate through a complex call chain to complete data storage and processing tasks. Monitoring the storage system and capturing its alarm events is crucial. Since these alarm events are caused by the storage system processing external requests, we need to obtain the request identifiers involved in the alarm events. We retrieve the trajectory data corresponding to the request identifiers from a pre-built request trajectory database. Upon capturing an alarm event, we aggregate the trajectory data related to the request identifiers and construct a fault feature sequence based on the trajectory data. Subsequently, we obtain the repair strategies corresponding to the fault feature sequences from a pre-built fault repair database, enabling automatic repair of known faults. This process eliminates the need for manual analysis of the call chain, inference of fault causes, and manual intervention to complete the repair, improving the efficiency and accuracy of fault location and repair.

[0098] refer to Figure 3 , Figure 3 A flowchart illustrating the storage system fault repair method provided in this application embodiment. Figure 2 Based on the above embodiments, this embodiment describes the process of pre-building a fault repair database, as detailed below:

[0099] S301: Obtain multiple historical alarm events of the storage system, as well as the historical repair strategies for each historical alarm event.

[0100] In this embodiment, historical alarm events that have been closed and effectively repaired, along with repair strategies, are selected. The time range covers the most recent 3 to 6 months, and the historical sample size is sufficient to include high-frequency faults.

[0101] S302: Obtain the request identifier from each historical alarm event.

[0102] In this embodiment, firstly, the corresponding request identifier is extracted from the original logs of each historical alarm event; secondly, the format of the obtained request identifier is checked to ensure that the corresponding trajectory data can be queried in step S303; if the extraction fails, the historical alarm event is marked as an invalid sample and will not be included in the construction of the fault repair database for the time being.

[0103] S303: Based on the request identifier, retrieve the trajectory data corresponding to the request identifier from the pre-built request trajectory database.

[0104] In this embodiment, the query interface of the request trajectory database is called to obtain the trajectory data of the request identifier.

[0105] S304: Construct a historical fault feature sequence for each historical alarm event based on the trajectory data corresponding to the request identifier.

[0106] In this embodiment, fault features, including service name, request status, and business name, are filtered out from the trajectory data to construct a historical fault feature sequence.

[0107] S305: Save the historical fault feature sequences of each historical alarm event and the historical repair strategies of each historical alarm event in the form of mapping pairs to obtain a pre-built fault repair database.

[0108] In this embodiment, by simulating the antigen-antibody mechanism of the biological immune system, fault feature sequences are used as antigens and repair strategies are used as antibodies to form a mapping relationship and save them, thus obtaining a pre-constructed fault repair database.

[0109] In this embodiment, the pre-built fault repair database stores multiple known fault feature sequences and corresponding repair strategies for each known fault feature sequence.

[0110] In this embodiment, the characteristic gene sequence is a unique description of a fault mode, which can be understood as the cause of the fault; the repair strategy is an effective solution for dealing with the fault mode, which may be an operation instruction or workflow such as executing an automated script, expanding service capacity, or performing fault switching.

[0111] In this embodiment, when an alarm event is first processed, the fault feature sequence and the corresponding repair strategy are saved as a mapping pair to a pre-built fault repair database.

[0112] In summary, because the pre-built fault repair database stores multiple known fault feature sequences and corresponding repair strategies for each known fault feature sequence, when an alarm event is captured, the corresponding repair strategy can be quickly retrieved from the pre-built fault repair database based on the fault feature sequence of the alarm event. This reduces the repair time for the same fault from hours of manual processing to minutes, lowering maintenance manpower costs and time, and further improving the efficiency and accuracy of fault repair.

[0113] refer to Figure 4 , Figure 4 A flowchart illustrating the storage system fault repair method provided in this application embodiment. Figure 3 Based on the above embodiments, this embodiment describes the process of pre-building the request trajectory database, as detailed below:

[0114] S401: When the storage system receives any external request, it generates a request identifier corresponding to the current external request.

[0115] In this embodiment, when an external request originating from outside the storage system is sent to the storage system and requires processing by the storage system, a unique request identifier is generated for each external request to ensure that the request identifiers of different external requests are not duplicated. This request identifier is transmitted throughout the entire request chain. Through hook functions, it is ensured that the request identifier follows the request through any service, thread, and asynchronous task, without being lost or tampered with, thus avoiding tracing chain breaks.

[0116] S402: When the current external request enters any processing stage, collect multiple attribute values ​​of the current external request at the current processing stage.

[0117] In this embodiment, data needs to be collected for every step an external request goes through, including service calls, asynchronous tasks, database operations, and cache interactions. This data can be collected from the storage system's logs, including service name, operation name, request status, request result, involved databases, and time.

[0118] Optionally, when the storage system prints logs, it must also output the request identifier of the current context as core metadata. The print structure is: request identifier + system log structure, including log print time, log type, and description.

[0119] S403: Using the request identifier as the primary key, the system stores multiple attribute values ​​of the current external request at each processing stage in a structured manner to obtain a pre-built request trajectory database.

[0120] In this embodiment, the request identifier is used as the primary key, and each request identifier corresponds to a request trajectory data, which contains multiple attribute values.

[0121] In summary, the pre-built request trajectory database provides data support for the entire fault matching and repair process. It allows for the retrieval of trajectory data for request identifiers by calling the database's query interface. Upon capturing an alarm event, the trajectory data corresponding to the request identifier can also be retrieved. The pre-built request trajectory database enables the rapid aggregation of trajectory data related to request identifiers, further improving the efficiency of fault repair.

[0122] refer to Figure 5 , Figure 5 A flowchart illustrating the storage system fault repair method provided in this application embodiment. Figure 4 Based on the above embodiments, this embodiment describes the process of converting a pre-built request trajectory database into a graph-structured request trajectory database, as detailed below:

[0123] In this embodiment, the pre-built request trajectory database is converted into a graph-structured request trajectory database; during the conversion process, the following steps are performed; the steps include S501~S507:

[0124] S501: Obtain the trajectory data corresponding to each request identifier from the pre-built request trajectory database.

[0125] In this embodiment, the trajectory data is divided into multiple independent trajectory data units according to the request identifier, which facilitates the subsequent construction of the graph structure of each request identifier.

[0126] S502: Obtain multiple stage identifiers from the trajectory data corresponding to each request identifier; the trajectory data consists of data from different external requests at multiple processing stages.

[0127] In this embodiment, the process identifiers of each processing stage are extracted one by one from the trajectory data corresponding to each request identifier to form a set of process identifiers for that request identifier.

[0128] S503: Identify multiple links and define them as multiple nodes.

[0129] In this embodiment, to avoid node conflicts caused by duplicate process names under different request identifiers, the node identifier adopts a combination format of request identifier and process name to ensure global uniqueness.

[0130] S504: Determine the multiple attribute values ​​corresponding to the identifiers of each link as the node attributes of each node.

[0131] Among them, several attribute values ​​include the parent stage identifier.

[0132] In this embodiment, multiple attributes corresponding to each stage identifier are extracted from the trajectory data corresponding to each request identifier, and used as key-value pair attributes of the node, maintaining the attribute format consistent with the structured database. Multiple attributes are then associated with the node to form attributed nodes.

[0133] S505: Determine the edge attributes between nodes based on the parent node identifier.

[0134] In this embodiment, the parent stage identifier refers to the stage identifier that calls the current processing stage. If the current stage identifier is the first stage, then there is no parent stage identifier.

[0135] In this embodiment, the edge attribute is used to describe the relationship between parent and child nodes. Nodes without a parent node identifier are designated as starting nodes. Following the direction from the starting point to the ending point, and combining the calling relationship reflected by the parent node identifier, an edge is generated for each node to ensure the uniqueness of the edge.

[0136] S506: Construct a graph structure for each request identifier based on multiple nodes, the node attributes of each node, and the edge attributes between each node.

[0137] In this embodiment, the set of nodes and the set of edges under the same request identifier are combined to form a complete graph structure.

[0138] In this embodiment, after the construction is completed, the integrity of the graph structure of each request identifier is checked, such as whether any non-root node has a corresponding incoming edge and whether the node attributes are complete. If the verification passes, it is marked as valid; otherwise, it is marked as invalid and can be manually completed later.

[0139] S507: Based on the graph structure of each request identifier, obtain the request trajectory database of the graph structure.

[0140] In this embodiment, a suitable graph database is selected according to business requirements, and a batch import tool for the graph database is used to write the graph structure of each request identifier into the database in batches, avoiding the performance bottleneck caused by single insertion, and obtaining a request trajectory database of graph structure.

[0141] In summary, transforming linear trajectory data into a visualized graph structure, with nodes and edges, intuitively presents the flow path of each request identifier. Graph traversal algorithms can then be used to quickly obtain the associated data of request identifiers and rapidly locate the upstream and downstream dependencies of abnormal nodes.

[0142] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0143] Figure 6 A schematic diagram of the structure of a storage system fault repair device provided in an embodiment of this application. (See attached diagram.) Figure 6 As shown, embodiments of this application also provide a storage system fault repair device, including: a capture module 601, a first acquisition module 602, a second acquisition module 603, a construction module 604, a third acquisition module 605, and a repair module 606.

[0144] The capture module 601 is used to monitor the storage system and capture alarm events of the storage system.

[0145] The first acquisition module 602 is used to acquire the request identifier in the alarm event; wherein the request identifier is generated for external requests to the storage system.

[0146] The second acquisition module 603 is used to acquire trajectory data corresponding to the request identifier from a pre-built request trajectory database; wherein the pre-built request trajectory database stores trajectory data of different external requests.

[0147] Module 604 is used to construct a fault feature sequence of alarm events based on the trajectory data corresponding to the request identifier.

[0148] The third acquisition module 605 is used to acquire the repair strategy corresponding to the fault feature sequence from the pre-built fault repair database.

[0149] Repair module 606 is used to repair alarm events according to the repair strategy.

[0150] In one possible implementation, the construction module 604 is specifically used to: obtain multiple stage identifiers from the trajectory data corresponding to the request identifier; the trajectory data consists of data from different external requests at multiple processing stages; obtain the feature vectors corresponding to each stage identifier based on the trajectory data corresponding to the request identifier; and obtain the fault feature sequence of the alarm event based on the feature vectors corresponding to each stage identifier.

[0151] In one possible implementation, the third acquisition module 605 is specifically used to: acquire multiple known fault feature sequences from a pre-built fault repair database; wherein the pre-built fault repair database includes multiple known fault feature sequences and repair strategies corresponding to each known fault feature sequence; sequentially match the multiple known fault feature sequences with the fault feature sequence to obtain the matching degree of each known fault feature sequence; and determine the repair strategies corresponding to the known fault feature sequences whose matching degree exceeds a preset matching degree threshold as the repair strategies corresponding to the fault feature sequences.

[0152] In one possible implementation, the fault feature sequence includes a request state, a service name, and a service name; correspondingly, the third acquisition module 605 is further configured to: acquire the same service name from multiple known fault feature sequences and fault feature sequences; determine the temporal order matching degree of each known fault feature sequence based on whether the temporal order of the same service name in each known fault feature sequence is the same as the temporal order of the same service name in the fault feature sequence; wherein, the same temporal order and different temporal order correspond to different temporal order matching degrees; determine the temporal order matching degree of each known fault feature sequence based on whether the request state corresponding to the same service name in each known fault feature sequence is the same as the request state corresponding to the same service name in the fault feature sequence. The matching degree of request status for each known fault feature sequence is determined; where the same request status and different request status correspond to different matching degrees. Based on the same service name, the matching degree of the business name corresponding to each known fault feature sequence is determined by comparing it with the corresponding business name in the fault feature sequence. Where the same business name and different business names correspond to different matching degrees. The matching degree of each known fault feature sequence is obtained through weighted fusion based on the temporal order matching degree, request status matching degree, business name matching degree, preset temporal order matching degree weight, preset request status matching degree weight, and preset business name matching degree weight.

[0153] In one possible implementation, the storage system fault repair device further includes an update module. Specifically, the update module is used to: if there is no repair strategy corresponding to the fault feature sequence in the pre-built fault repair database, send the fault feature sequence to the operation and maintenance end so that the operation and maintenance end can perform manual repair based on the fault feature sequence; receive the manual repair strategy sent by the operation and maintenance end; and save the fault feature sequence and the manual repair strategy to the pre-built fault repair database to update the pre-built fault repair database.

[0154] In one possible implementation, the storage system fault repair device further includes a first pre-construction module. The first pre-construction module is specifically used for: acquiring multiple historical alarm events of the storage system, and historical repair strategies for each historical alarm event; acquiring request identifiers from each historical alarm event; acquiring trajectory data corresponding to the request identifiers from a pre-constructed request trajectory database based on the request identifiers; constructing historical fault feature sequences for each historical alarm event based on the trajectory data corresponding to the request identifiers; and saving the historical fault feature sequences and historical repair strategies for each historical alarm event in the form of mapping pairs to obtain a pre-constructed fault repair database.

[0155] In one possible implementation, the storage system fault repair device further includes a second pre-construction module. Specifically, the second pre-construction module is used for: generating a request identifier corresponding to the current external request when the storage system receives any external request; collecting multiple attribute values ​​of the current external request at the current processing stage when the current external request enters any processing stage; and using the request identifier as the primary key, structurally storing the multiple attribute values ​​of the current external request at each processing stage to obtain a pre-constructed request trajectory database.

[0156] In one possible implementation, the multiple attribute values ​​include a stage identifier; the storage system fault repair apparatus further includes a generation module. Specifically, the generation module is used to: generate a request identifier using a random generator; and generate a stage identifier using a random generator.

[0157] In one possible implementation, the storage system fault repair device further includes a conversion module. The conversion module is specifically used to: convert a pre-built request trajectory database into a graph-structured request trajectory database; during the conversion process, the following steps are performed: obtaining trajectory data corresponding to each request identifier from the pre-built request trajectory database; obtaining multiple stage identifiers from the trajectory data corresponding to each request identifier; the trajectory data consists of data from different external requests at multiple processing stages; identifying the multiple stage identifiers as multiple nodes; identifying multiple attribute values ​​corresponding to each stage identifier as node attributes of each node; wherein the multiple attribute values ​​include a parent stage identifier; determining the edge attributes between nodes based on the parent stage identifier; constructing a graph structure for each request identifier based on the multiple nodes, the node attributes of each node, and the edge attributes between nodes; and obtaining a graph-structured request trajectory database based on the graph structure of each request identifier.

[0158] For a description of the features in the embodiment corresponding to the storage system failure repair device, please refer to the relevant description of the embodiment corresponding to the storage system failure repair method, which will not be repeated here.

[0159] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 7 As shown, the electronic device provided in this embodiment includes at least one processor 701 and a memory 702. Optionally, the electronic device further includes a communication component 703. The processor 701, memory 702, and communication component 703 are connected via a bus.

[0160] In a specific implementation, at least one processor 701 executes computer execution instructions stored in memory 702, causing at least one processor 701 to execute the above-described storage system fault repair method embodiment.

[0161] The specific implementation process of processor 701 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0162] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0163] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0164] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0165] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the storage system fault repair method.

[0166] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0167] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described methods for repairing storage system faults.

[0168] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described embodiments of the storage system fault repair method.

[0169] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0170] The foregoing has provided a detailed description of a method, apparatus, electronic device, medium, and product for repairing storage system faults. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to aid in understanding the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for repairing storage system faults, characterized in that, include: Monitor the storage system and capture alarm events of the storage system; Obtain the request identifier from the alarm event; The request identifier is generated for external requests to the storage system; the request identifier is transmitted transparently throughout the request chain. Aggregate trajectory data corresponding to the request identifier from a pre-built request trajectory database; The pre-built request trajectory database contains trajectory data of different external requests; Based on the trajectory data corresponding to the request identifier, determine the sequence with temporal order corresponding to each link identifier, and construct the fault feature sequence of the alarm event; Multiple known fault feature sequences are obtained from a pre-built fault repair database; these multiple known fault feature sequences are then matched sequentially with the fault feature sequence; wherein, the fault feature sequence includes request status, business name, and service name; When the sequence length of the known fault feature sequence is the same as that of the fault feature sequence, the matching degree of each known fault feature sequence is obtained based on the fault feature sequence and the time sequence order. When the sequence length of the known fault feature sequence is inconsistent with the sequence length of the fault feature sequence, the matching degree is multiplied by the sequence length correction coefficient to obtain the final matching degree. The repair strategy corresponding to the known fault feature sequence whose matching degree exceeds a preset matching degree threshold is determined as the repair strategy corresponding to the fault feature sequence. The alarm event is repaired according to the repair strategy.

2. The method according to claim 1, characterized in that, The step of constructing the fault feature sequence of the alarm event based on the trajectory data corresponding to the request identifier includes: Multiple stage identifiers are obtained from the trajectory data corresponding to the request identifier; the trajectory data consists of data from different external requests at multiple processing stages. Based on the trajectory data corresponding to the request identifier, obtain the feature vector corresponding to each stage identifier; Based on the feature vectors corresponding to the identifiers of each stage, the fault feature sequence of the alarm event is obtained.

3. The method according to claim 1, characterized in that, Accordingly, obtaining the matching degree of each known fault feature sequence based on the fault feature sequence and the time sequence includes: Obtain the same service name from the plurality of known fault feature sequences and the fault feature sequences; Based on the same service name, the temporal order matching degree of each known fault feature sequence is determined by comparing its temporal order with that of the same service name in the fault feature sequence; wherein, the same temporal order and the different temporal order correspond to different temporal order matching degrees. Based on the same service name, the request status corresponding to each known fault feature sequence is compared with the request status corresponding to the same service name in the fault feature sequence to determine the request status matching degree of each known fault feature sequence; wherein, the same request status and the different request status correspond to different request status matching degrees. Based on the same service name, determine the service name matching degree of each known fault feature sequence by comparing the service name corresponding to each known fault feature sequence with the service name corresponding to the same service name in the fault feature sequence; wherein, the same service name and different service names correspond to different service name matching degrees. Based on the time sequence matching degree, the request status matching degree, the service name matching degree, the preset time sequence matching degree weight, the preset request status matching degree weight, and the preset service name matching degree weight, the matching degree of each known fault feature sequence is obtained through weighted fusion.

4. The method according to claim 1, characterized in that, The method further includes: If the pre-built fault repair database does not contain a repair strategy corresponding to the fault feature sequence, the fault feature sequence will be sent to the operation and maintenance terminal so that the operation and maintenance terminal can perform manual repair based on the fault feature sequence. Receive manual repair strategies sent by the operation and maintenance terminal; The fault feature sequence and the manual repair strategy are saved to the pre-built fault repair database to update the pre-built fault repair database.

5. The method according to claim 1, characterized in that, The method further includes: Obtain multiple historical alarm events of the storage system, as well as the historical repair strategies for each historical alarm event; Retrieve the request identifier from each historical alarm event; Based on the request identifier, retrieve the trajectory data corresponding to the request identifier from the pre-built request trajectory database; Based on the trajectory data corresponding to the request identifier, construct a historical fault feature sequence for each historical alarm event; The historical fault feature sequences of each historical alarm event and the historical repair strategies of each historical alarm event are saved in the form of mapping pairs to obtain a pre-built fault repair database.

6. The method according to claim 1, characterized in that, The method further includes: When the storage system receives any external request, it generates the request identifier corresponding to the current external request. When the current external request enters any processing stage, collect multiple attribute values ​​of the current external request at the current processing stage; Using the request identifier as the primary key, the multiple attribute values ​​of the current external request at each processing stage are stored in a structured manner to obtain a pre-built request trajectory database.

7. The method according to claim 6, characterized in that, The multiple attribute values ​​include a stage identifier; Accordingly, the method further includes: The request identifier is generated using a random generator; The random generator is used to generate the stage identifier.

8. The method according to claim 6, characterized in that, The method further includes: Transform the pre-built request trajectory database into a graph-structured request trajectory database; During the conversion process, perform the following steps; The steps include: Obtain trajectory data corresponding to each request identifier from the pre-built request trajectory database; Multiple stage identifiers are obtained from the trajectory data corresponding to each request identifier; the trajectory data consists of data from different external requests at multiple processing stages. The multiple links are identified and defined as multiple nodes; The multiple attribute values ​​corresponding to the identifiers of each stage are determined as the node attributes of each node; wherein, the multiple attribute values ​​include the parent stage identifier; Based on the parent node identifier, determine the edge attributes between each node; Based on the plurality of nodes, the node attributes of each node, and the edge attributes between the nodes, construct a graph structure for each request identifier; Based on the graph structure of each request identifier, a request trajectory database with a graph structure is obtained.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the method for repairing a storage system failure as described in any one of claims 1-8 when executing the computer program.

Images