Group fault detection method and device, computer device and storage medium

Abstract

This application relates to a method, apparatus, computer device, and storage medium for detecting cluster failures. The method includes: acquiring read / write requests for business data stored in the storage layer; in response to the read / write requests, acquiring a monitoring status identifier of the storage layer; if the monitoring status identifier is off, determining whether a cluster failure has occurred in the storage layer during the read / write operation of the business data via the read / write requests; and if a cluster failure has occurred in the storage layer, generating a storage layer failure alert. This method can promptly alert maintenance personnel to repair cluster failures, reducing the potential risk of systemic collapse at the control layer, avoiding recovery time after a crash, and further reducing the unavailability time of the distributed database.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular to a method, apparatus, computer equipment, and storage medium for detecting mass failures. Background Technology

[0002] Distributed databases typically employ a two-layer architecture: the bottom layer is the storage layer responsible for storing business data, and the upper layer is the control layer responsible for handling business data operation requests. The storage layer usually ensures the reliability of business data and the availability of services by combining data redundancy and multi-active structures. However, when the storage layer inevitably fails and becomes unavailable, it may cause a collective "down" of the control nodes in the control layer.

[0003] In traditional technologies, distributed databases are typically repaired and fault detected only after a collective failure has occurred and the database has crashed. In addition, after the storage layer recovers, each control node in the control layer needs to be restarted and metadata needs to be loaded.

[0004] While this method can accomplish fault repair and detection of distributed databases, for distributed databases deployed through large-scale clusters, the business data recovery time is relatively long, further extending the unavailability time of the distributed database. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, computer equipment, and storage medium for detecting mass failures that can accurately detect mass failures to avoid the collapse of distributed databases, in order to address the aforementioned technical problems.

[0006] Firstly, this application provides a method for detecting mass failures, including:

[0007] Get read and write requests for business data stored in the storage layer;

[0008] In response to read / write requests, obtain the monitoring status identifier of the storage layer;

[0009] When the monitoring status is set to off, determine whether a mass failure occurs in the storage layer during the process of reading and writing business data through read and write requests;

[0010] In the event of a mass failure at the storage layer, a storage layer failure alert will be generated.

[0011] In one embodiment, determining whether a mass failure occurs in the storage layer during the process of reading and writing business data via read / write requests includes:

[0012] In response to read and write requests, perform read and write operations on business data;

[0013] In the event of a read / write failure, obtain the reason for the failure.

[0014] Based on the reasons for read / write failures, determine whether a cluster failure has occurred in the storage layer.

[0015] In one embodiment, determining whether a mass failure has occurred in the storage layer based on the reasons for read / write failures includes:

[0016] Obtain the characteristics of mass failures corresponding to different preset mass failures;

[0017] Match the causes of read / write failures with the characteristics of various group failures;

[0018] If the match fails, it is determined that no mass failure has occurred in the storage layer;

[0019] If a match is found, it is determined that a mass failure has occurred in the storage layer, and the mass failure corresponding to the successfully matched mass failure feature is taken as the target failure.

[0020] In one embodiment, the method further includes:

[0021] When the monitoring status is enabled, read and write requests are added to a pre-set waiting queue;

[0022] If a read / write request fails to be added to the waiting queue, an alert will be displayed indicating that the waiting queue is full.

[0023] If a read / write request is successfully added to the waiting queue, the timer is controlled to start timing the request. When the timer reaches a preset duration threshold, the read / write request is removed from the waiting queue, and a request timeout reminder is output.

[0024] In one embodiment, the method further includes:

[0025] In the event of a mass failure at the storage layer, the monitoring status flag will be set to enabled.

[0026] In one embodiment, the method further includes:

[0027] When the monitoring status is enabled, perform a health check on the storage layer; the health check includes at least one of the following: storage environment check, hardware status check, software configuration check, and performance check.

[0028] If the health check is passed, the monitoring status indicator will be reset to off.

[0029] Secondly, this application also provides a mass failure detection device, comprising:

[0030] The request retrieval module is used to retrieve read and write requests for business data stored in the storage layer.

[0031] The identifier acquisition module is used to obtain the monitoring status identifier of the storage layer in response to read and write requests;

[0032] The fault diagnosis module is used to determine whether a mass failure has occurred in the storage layer during the process of reading and writing business data through read and write requests, when the monitoring status is marked as off.

[0033] The alert generation module is used to generate storage layer failure alerts in the event of a mass failure in the storage layer.

[0034] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0035] Get read and write requests for business data stored in the storage layer;

[0036] In response to read / write requests, obtain the monitoring status identifier of the storage layer;

[0037] When the monitoring status is set to off, determine whether a mass failure occurs in the storage layer during the process of reading and writing business data through read and write requests;

[0038] In the event of a mass failure at the storage layer, a storage layer failure alert will be generated.

[0039] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0040] Get read and write requests for business data stored in the storage layer;

[0041] In response to read / write requests, obtain the monitoring status identifier of the storage layer;

[0042] When the monitoring status is set to off, determine whether a mass failure occurs in the storage layer during the process of reading and writing business data through read and write requests;

[0043] In the event of a mass failure at the storage layer, a storage layer failure alert will be generated.

[0044] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0045] Get read and write requests for business data stored in the storage layer;

[0046] In response to read / write requests, obtain the monitoring status identifier of the storage layer;

[0047] When the monitoring status is set to off, determine whether a mass failure occurs in the storage layer during the process of reading and writing business data through read and write requests;

[0048] In the event of a mass failure at the storage layer, a storage layer failure alert will be generated.

[0049] The aforementioned method, apparatus, computer equipment, and storage medium for detecting mass failures acquire read / write requests for business data stored in the storage layer; respond to the read / write requests by acquiring the monitoring status identifier of the storage layer; if the monitoring status identifier is off, determine whether a mass failure has occurred in the storage layer during the read / write process of business data; and generate a storage layer failure alert if a mass failure has occurred in the storage layer. This implementation introduces a monitoring status identifier for the storage layer. When the monitoring status identifier is off, it indicates that the storage layer is healthy and does not require monitoring. Further determination of whether a mass failure has occurred is needed during the read / write process of business data. When a mass failure occurs in the storage layer, a storage layer failure alert is output, promptly reminding maintenance personnel to repair the mass failure. This reduces the potential risk of systemic collapse in the control layer, avoids recovery time after a collapse, and further reduces the unavailability time of the distributed database. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is an application environment diagram of a group fault detection method provided in this embodiment;

[0052] Figure 2 This is a flowchart illustrating the first method for detecting mass faults provided in this embodiment;

[0053] Figure 3A This is a flowchart illustrating a group fault judgment step provided in this embodiment;

[0054] Figure 3B This embodiment provides a storage layer architecture diagram;

[0055] Figure 4A This is a schematic diagram of a process for adjusting an identifier, as provided in this embodiment.

[0056] Figure 4B This embodiment provides a flowchart for handling storage layer failures and recovery processes.

[0057] Figure 4C This embodiment provides a flowchart for storage layer fault identification and processing.

[0058] Figure 5 This is a flowchart illustrating the second method for detecting mass faults provided in this embodiment;

[0059] Figure 6 This is a structural block diagram of a group fault detection device provided in this embodiment;

[0060] Figure 7 This is an internal structural diagram of a computer device provided in this embodiment. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0062] The mass failure detection method provided in this application can be applied to, for example... Figure 1 In the application environment shown, control node 101 in the control layer communicates with storage node 102 in the storage layer. The storage layer typically consists of multiple nodes, responsible for storing and retrieving business data. Data is sliced, with each node storing a portion of the business data shards. Copies of these shards are stored on other nodes to ensure the availability and persistence of the business data. The control layer typically consists of one or more control nodes, responsible for receiving and parsing client requests, coordinating and managing transactions, maintaining cluster metadata and status information, and ensuring system availability and reliability. Control node 101 obtains read / write requests for the business data stored in the corresponding storage node 102 in the storage layer; in response to the read / write requests, it obtains the monitoring status identifier of the storage layer; if the monitoring status identifier is off, it determines whether a mass failure has occurred in the storage layer during the read / write process of accessing the business data in storage node 102; if a mass failure has occurred in the storage layer, it generates a storage layer failure alert.

[0063] In one exemplary embodiment, such as Figure 2 As shown, a method for detecting group faults is provided, which can be applied to... Figure 1 Taking control node 101 as an example, the explanation includes the following steps 201 to 204. Wherein:

[0064] Step 201: Obtain read / write requests for business data stored in the storage layer.

[0065] Read and write requests can be understood as operations that read or write business data stored in storage nodes within the storage layer.

[0066] Specifically, the control node in the control layer receives the initial request sent by the client; the controller in the control node selects read and write requests from each initial request for subsequent processing; for non-read and write requests, the controller responds to the non-read and write request and executes the corresponding non-read and write operation.

[0067] Step 202: In response to the read / write request, obtain the monitoring status identifier of the storage layer.

[0068] The monitoring status flag can be understood as an indicator of the health status of the storage layer. When the monitoring status flag is off, it means that the storage layer is in good health and there is no fault. When the monitoring status flag is on, it means that the storage layer has a fault and its health status is being monitored in real time.

[0069] Specifically, the control node responds to read / write requests by obtaining the current monitoring status identifier of the storage layer. If the monitoring status identifier is off, then proceed to step 203. If the monitoring status identifier is on, it proves that the storage layer has failed, and the reception of read / write requests can be prohibited, and a read / write failure reminder can be returned to the client.

[0070] Step 203: If the monitoring status is marked as off, determine whether a mass failure occurs in the storage layer during the process of reading and writing business data through read and write requests.

[0071] Among them, a group failure can be understood as a situation in which multiple storage nodes in the storage layer fail within the same or similar time period.

[0072] Specifically, when the control node detects that the monitoring status is off, it determines that the storage layer is in normal health and can perform normal read and write operations. Then, based on the read and write request, it identifies the storage node in the storage layer corresponding to the read and write request; performs the read and write operation corresponding to the read and write request on the business data stored in the corresponding storage node; and determines whether a mass failure has occurred in the storage layer based on the completion status of the read and write operation.

[0073] For example, if the control node completes the read and write operations on the business data, it proves that no mass failure has occurred in the storage layer; if the control node fails to complete the read and write operations on the business data and receives a read / write exception alert returned by the storage node corresponding to the read / write request, it is determined that a mass failure has occurred in the storage layer at this time.

[0074] Step 204: In the event of a mass failure in the storage layer, generate a storage layer failure alert.

[0075] Among them, storage layer fault alerts can be understood as alerts to maintenance personnel that a group of storage layer failures has occurred.

[0076] Specifically, when the control node determines that a mass failure has occurred in the storage layer, it identifies the storage node that experienced the mass failure and the name of the mass failure on the corresponding storage node; based on the storage node identifier and the name of the mass failure, it generates a storage layer failure alert.

[0077] The aforementioned method for detecting cluster failures involves: acquiring read / write requests for business data stored in the storage layer; responding to these requests by acquiring the storage layer's monitoring status identifier; determining whether a cluster failure occurred in the storage layer during the read / write process of business data; and generating a storage layer failure alert if a cluster failure occurred. This implementation introduces a storage layer monitoring status identifier. When the identifier is off, it indicates that the storage layer is healthy and requires no monitoring. Further investigation is needed during the read / write process of business data to determine if a cluster failure has occurred. When a cluster failure occurs, a storage layer failure alert is output, promptly reminding maintenance personnel to address the issue. This reduces the potential risk of systemic collapse at the control layer, minimizes recovery time after a crash, and further reduces the unavailability time of the distributed database.

[0078] Figure 3A This is a flowchart illustrating the steps for determining a mass failure in one embodiment. This embodiment further refines the steps in the above embodiment for determining whether a mass failure has occurred in the storage layer during the process of reading and writing business data via read / write requests, when the monitoring status is off. It provides an optional method for determining a mass failure, including the following steps:

[0079] Step 301: In response to the read / write request, perform read / write operations on the business data.

[0080] Specifically, the control node responds to read / write requests by determining the storage node corresponding to the read / write request and performing read / write operations on the corresponding business data in the storage node.

[0081] Step 302: In the event of a read / write failure, obtain the reason for the failure.

[0082] Specifically, after receiving read / write exception alerts from the storage nodes, the control node parses the read / write exception alerts to determine the reasons for the read / write failures.

[0083] Step 303: Determine whether a mass failure has occurred in the storage layer based on the reasons for read / write failures.

[0084] One possible implementation is that the control node inputs the reasons for read / write failures into a pre-trained fault detection model, and the fault detection model parses the fault conditions of the storage layer from the reasons for read / write failures; if the fault condition is a group failure, it proves that a group failure has occurred in the storage layer; otherwise, it proves that the corresponding storage node in the storage layer has only experienced a general failure.

[0085] Another optional implementation is to obtain the group failure characteristics corresponding to different preset group failures; match the read / write failure reasons with each group failure characteristic; if the match fails, it is determined that no group failure has occurred in the storage layer; if the match succeeds, it is determined that a group failure has occurred in the storage layer, and the group failure corresponding to the successfully matched group failure characteristic is taken as the target failure. Here, preset group failures can be understood as the types of group failures that have occurred in the storage layer within a historical period.

[0086] Specifically, the control node acquires the group failure characteristics corresponding to different preset group failures; it matches the read / write failure reasons with each group failure characteristic; if the match fails, it is determined that no group failure has occurred in the storage layer, and the storage node that failed the read / write operation can be directly determined to have only experienced a general failure based on the read / write reason; if the match succeeds, it proves that a group failure has occurred in the storage layer, and the successfully matched group failure characteristics are further determined, with the group failure corresponding to the group failure characteristic being taken as the target failure. It should be noted that in the process of generating storage layer failure alerts, this embodiment can also generate storage layer failure alerts based on the target failure.

[0087] Another possible implementation is, for example Figure 3B The storage layer architecture diagram shown in this embodiment allows for the inclusion of a fault detector within the storage layer. The control node invokes the fault detector, which uses at least one set of fault filters to filter and match read / write failure causes (i.e., matching the read / write failure causes against corresponding group failure characteristics in the fault filters). If a match is found, it indicates a group failure has occurred in the storage layer, and a group failure alert is returned to the client. The advantage of this setup is that the fault detector can accurately detect whether a group failure has occurred in the storage layer.

[0088] It should be noted that the fault identifier contains a set of fault filters. The filtering rules of each fault filter are consistent with one of the storage layer group fault characteristics. It can accurately filter out storage layer group faults that meet the characteristics. This set of fault filters constitutes a complete storage layer group fault identifier, which can not only fully identify all group faults that occur in the storage layer, but also avoid capturing general faults.

[0089] In the above embodiments, in response to read / write requests, read / write operations are performed on business data; in the event of read / write failure, the reason for the failure is obtained; based on the reason for the failure, it is determined whether a mass failure has occurred in the storage layer. This embodiment accurately determines whether a mass failure has occurred in the storage layer by identifying the reason for the read / write failure and marks such failures, thereby improving the accuracy of mass failure detection and reducing the potential risks of mass failures.

[0090] Figure 4A This is a flowchart illustrating the identifier adjustment step in one embodiment. This embodiment further refines the steps required in the event of a mass failure at the storage layer as described above, providing an optional approach to the identifier adjustment step, including the following steps:

[0091] Step 401: In the event of a mass failure in the storage layer, adjust the monitoring status flag to "on".

[0092] Specifically, when the control node determines that a mass failure has occurred in the storage layer, it will adjust the monitoring status flag to "on".

[0093] It should be noted that, as Figure 3B The storage layer architecture diagram shown illustrates that the monitoring status flag can be set on the status monitor in the control layer. This status monitor includes a monitoring status flag and a health check task. By default, the monitoring status flag is off (False), the health check task is in sleep mode, and it listens for changes in the monitoring status flag. When the fault identifier sets the monitoring status flag to on (True) (i.e., when the fault identifier detects a cluster fault, it transmits this signal to the status monitor, setting its monitoring status flag to True), the status monitor, upon detecting the change from False to True, wakes up the internal health check task to perform real-time monitoring of the storage layer's health status. When the storage layer is detected to have recovered, the monitoring status flag is reset to False, the health check task returns to sleep mode, and it resumes listening to the status flag.

[0094] For example, when the monitoring status flag is enabled, a health check is performed on the storage layer; the health check includes at least one of storage environment check, hardware status check, software configuration check, and performance check; if the health check passes, the monitoring status flag is reset to disabled.

[0095] Specifically, such as Figure 4BThe flowchart shown illustrates the storage layer fault diagnosis and recovery process. When the control node detects that the monitoring status flag is on or has changed from off to on, it wakes up a pre-configured health check task. This task performs a health check on the storage layer. Once the storage layer is detected as having recovered, the status flag is reset to off, and the health check task returns to sleep mode, resuming its monitoring of the status flag. The advantage of this setup is that by monitoring the storage's health status in real time, it can promptly determine whether the storage layer has recovered, allowing for timely updates to the monitoring status flag and expedited restoration of read / write functionality for business data.

[0096] It should be noted that, as Figure 3B The storage layer architecture diagram shown in this embodiment also allows for the configuration of a service denial mechanism within the storage layer. This service denial mechanism includes a finite-length waiting queue and a timer. When the status monitor is enabled, client read / write requests are placed in the waiting queue and a timer begins. When the waiting queue is full or the request times out (i.e., current time – request reception time > preset duration threshold), the request is removed from the waiting queue, and the removed read / write request is rejected, returning a read / write failure notification to the client.

[0097] For example, when the monitoring status is enabled, read and write requests are added to a pre-set waiting queue; if a read or write request fails to be added to the waiting queue, a reminder that the waiting queue is full is output; if a read or write request is successfully added to the waiting queue, a timer is controlled to count the requests, and when the request count reaches a preset duration threshold, the read or write request is removed from the waiting queue, and a request timeout reminder is output.

[0098] Specifically, such as Figure 4C The flowchart shown illustrates the storage layer fault identification and handling process. When the control node receives a read / write request and confirms that the monitoring status is enabled, it constructs a request structure `<Request for Read / Write, Request Time>` using the read / write request and request time. This request structure is then added to the service denial unit's waiting queue. The service denial unit's timer continuously monitors whether the request timeout has reached a preset threshold. If the request timeout reaches the preset threshold, the request structure is removed from the waiting queue, and a request timeout warning is output. Alternatively, if the service denial unit's waiting queue is full, the request structure is removed from the waiting queue, and a waiting queue full warning is output.

[0099] The advantage of this setup is that it effectively avoids the problem of the control node becoming stuck due to the unavailability of the storage layer and the resulting backlog of requests. It also avoids the problem of the client frequently retries due to the control node processing read and write request failures too quickly.

[0100] In one embodiment, this embodiment provides an optional method for detecting group failures, using the application of this method to a server as an example for illustration. For example... Figure 5 As shown, the method includes the following steps:

[0101] Step 501: Obtain read / write requests for business data stored in the storage layer.

[0102] Step 502: In response to the read / write request, obtain the monitoring status identifier of the storage layer.

[0103] Step 503: When the monitoring status is marked as off, respond to read / write requests and perform read / write operations on the business data.

[0104] Step 504: In the event of a read / write failure, obtain the reason for the failure.

[0105] Step 505: Obtain the group fault characteristics corresponding to different preset group faults.

[0106] Step 506: Match the reasons for read / write failures with the characteristics of each group of failures.

[0107] Step 507: If a match fails, output the reason for the read / write failure.

[0108] Step 508: If the match is successful, the monitoring status flag is set to enabled, the group fault corresponding to the successfully matched group fault feature is taken as the target fault, and a reminder that the storage layer has a target fault is output.

[0109] Step 509: If the monitoring status is enabled, add the read / write request to the pre-set waiting queue.

[0110] Step 510: If the read / write request fails to be added to the waiting queue, output a notification that the waiting queue is full.

[0111] Step 511: If the read / write request is successfully added to the waiting queue, control the timer to start timing the request. If the timer reaches the preset duration threshold, remove the read / write request from the waiting queue and output a request timeout reminder.

[0112] Step 512: Perform a health check on the storage layer; the health check includes at least one of the following: storage environment check, hardware status check, software configuration check, and performance check.

[0113] Step 513: If the health check passes, reset the monitoring status indicator to off.

[0114] It should be noted that in this embodiment, steps 509-511 are the operation steps of adding read and write requests to the waiting queue when the monitoring status is enabled; steps 512-513 are the operation steps of performing health checks on the storage layer when the monitoring status is enabled. These two operation steps need to be executed synchronously.

[0115] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0116] Based on the same inventive concept, this application also provides a group failure detection device for implementing the aforementioned group failure detection method. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more group failure detection device embodiments provided below can be found in the limitations of the group failure detection method described above, and will not be repeated here.

[0117] In one exemplary embodiment, such as Figure 6 As shown, a group fault detection device is provided, including: a request acquisition module 10, an identifier acquisition module 11, a fault judgment module 12, and an alert generation module 13, wherein:

[0118] Request retrieval module 10 is used to retrieve read and write requests for business data stored in the storage layer;

[0119] The identifier acquisition module 11 is used to acquire the monitoring status identifier of the storage layer in response to read / write requests;

[0120] The fault diagnosis module 12 is used to determine whether a mass failure occurs in the storage layer during the process of reading and writing business data through read and write requests when the monitoring status is marked as off.

[0121] The reminder generation module 13 is used to generate a storage layer fault reminder in the event of a mass failure in the storage layer.

[0122] In some embodiments, the fault determination module 12 includes:

[0123] The read / write operation unit is used to perform read / write operations on business data in response to read / write requests;

[0124] The reason acquisition unit is used to obtain the reason for the read / write failure in the event of a read / write failure.

[0125] The fault diagnosis unit is used to determine whether a cluster failure has occurred in the storage layer based on the reasons for read / write failures.

[0126] In some embodiments, the fault determination unit is further configured to acquire the group fault characteristics corresponding to different preset group faults; match the read / write failure reasons with each group fault characteristic; if the matching fails, determine that no group fault has occurred in the storage layer; if the matching succeeds, determine that a group fault has occurred in the storage layer, and take the group fault corresponding to the successfully matched group fault characteristic as the target fault.

[0127] In some embodiments, the mass failure detection device further includes:

[0128] The monitoring activation module is used to add read and write requests to a pre-set waiting queue when the monitoring status is enabled; if a read or write request fails to be added to the waiting queue, it outputs a reminder that the waiting queue is full; if a read or write request is successfully added to the waiting queue, it controls a timer to count the requests, and when the request count reaches a preset time threshold, it removes the read or write request from the waiting queue and outputs a request timeout reminder.

[0129] In some embodiments, the mass failure detection device further includes:

[0130] The monitoring adjustment module is used to adjust the monitoring status flag to "on" in the event of a mass failure in the storage layer.

[0131] In some embodiments, the monitoring activation module is further configured to perform a health check on the storage layer when the monitoring status indicator is enabled; the health check includes at least one of storage environment detection, hardware status detection, software configuration detection, and performance detection; if the health check passes, the monitoring status indicator is reset to disabled.

[0132] Each module in the aforementioned mass fault detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0133] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a method for detecting group faults.

[0134] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0135] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0136] Get read and write requests for business data stored in the storage layer;

[0137] In response to read / write requests, obtain the monitoring status identifier of the storage layer;

[0138] When the monitoring status is set to off, determine whether a mass failure occurs in the storage layer during the process of reading and writing business data through read and write requests;

[0139] In the event of a mass failure at the storage layer, a storage layer failure alert will be generated.

[0140] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0141] In response to read and write requests, perform read and write operations on business data;

[0142] In the event of a read / write failure, obtain the reason for the failure.

[0143] Based on the reasons for read / write failures, determine whether a cluster failure has occurred in the storage layer.

[0144] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0145] Obtain the characteristics of mass failures corresponding to different preset mass failures;

[0146] Match the causes of read / write failures with the characteristics of various group failures;

[0147] If the match fails, it is determined that no mass failure has occurred in the storage layer;

[0148] If a match is found, it is determined that a mass failure has occurred in the storage layer, and the mass failure corresponding to the successfully matched mass failure feature is taken as the target failure.

[0149] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0150] When the monitoring status is enabled, read and write requests are added to a pre-set waiting queue;

[0151] If a read / write request fails to be added to the waiting queue, an alert will be displayed indicating that the waiting queue is full.

[0152] If a read / write request is successfully added to the waiting queue, the timer is controlled to start timing the request. When the timer reaches a preset duration threshold, the read / write request is removed from the waiting queue, and a request timeout reminder is output.

[0153] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0154] In the event of a mass failure at the storage layer, the monitoring status flag will be set to enabled.

[0155] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0156] When the monitoring status is enabled, perform a health check on the storage layer; the health check includes at least one of the following: storage environment check, hardware status check, software configuration check, and performance check.

[0157] If the health check is passed, the monitoring status indicator will be reset to off.

[0158] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0159] Get read and write requests for business data stored in the storage layer;

[0160] In response to read / write requests, obtain the monitoring status identifier of the storage layer;

[0161] When the monitoring status is set to off, determine whether a mass failure occurs in the storage layer during the process of reading and writing business data through read and write requests;

[0162] In the event of a mass failure at the storage layer, a storage layer failure alert will be generated.

[0163] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0164] In response to read and write requests, perform read and write operations on business data;

[0165] In the event of a read / write failure, obtain the reason for the failure.

[0166] Based on the reasons for read / write failures, determine whether a cluster failure has occurred in the storage layer.

[0167] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0168] Obtain the characteristics of mass failures corresponding to different preset mass failures;

[0169] Match the causes of read / write failures with the characteristics of various group failures;

[0170] If the match fails, it is determined that no mass failure has occurred in the storage layer;

[0171] If a match is found, it is determined that a mass failure has occurred in the storage layer, and the mass failure corresponding to the successfully matched mass failure feature is taken as the target failure.

[0172] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0173] When the monitoring status is enabled, read and write requests are added to a pre-set waiting queue;

[0174] If a read / write request fails to be added to the waiting queue, an alert will be displayed indicating that the waiting queue is full.

[0175] If a read / write request is successfully added to the waiting queue, the timer is controlled to start timing the request. When the timer reaches a preset duration threshold, the read / write request is removed from the waiting queue, and a request timeout reminder is output.

[0176] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0177] In the event of a mass failure at the storage layer, the monitoring status flag will be set to enabled.

[0178] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0179] When the monitoring status is enabled, perform a health check on the storage layer; the health check includes at least one of the following: storage environment check, hardware status check, software configuration check, and performance check.

[0180] If the health check is passed, the monitoring status indicator will be reset to off.

[0181] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0182] Get read and write requests for business data stored in the storage layer;

[0183] In response to read / write requests, obtain the monitoring status identifier of the storage layer;

[0184] When the monitoring status is set to off, determine whether a mass failure occurs in the storage layer during the process of reading and writing business data through read and write requests;

[0185] In the event of a mass failure at the storage layer, a storage layer failure alert will be generated.

[0186] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0187] In response to read and write requests, perform read and write operations on business data;

[0188] In the event of a read / write failure, obtain the reason for the failure.

[0189] Based on the reasons for read / write failures, determine whether a cluster failure has occurred in the storage layer.

[0190] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0191] Obtain the characteristics of mass failures corresponding to different preset mass failures;

[0192] Match the causes of read / write failures with the characteristics of various group failures;

[0193] If the match fails, it is determined that no mass failure has occurred in the storage layer;

[0194] If a match is found, it is determined that a mass failure has occurred in the storage layer, and the mass failure corresponding to the successfully matched mass failure feature is taken as the target failure.

[0195] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0196] When the monitoring status is enabled, read and write requests are added to a pre-set waiting queue;

[0197] If a read / write request fails to be added to the waiting queue, an alert will be displayed indicating that the waiting queue is full.

[0198] If a read / write request is successfully added to the waiting queue, the timer is controlled to start timing the request. When the timer reaches a preset duration threshold, the read / write request is removed from the waiting queue, and a request timeout reminder is output.

[0199] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0200] In the event of a mass failure at the storage layer, the monitoring status flag will be set to enabled.

[0201] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0202] When the monitoring status is enabled, perform a health check on the storage layer; the health check includes at least one of the following: storage environment check, hardware status check, software configuration check, and performance check.

[0203] If the health check is passed, the monitoring status indicator will be reset to off.

[0204] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0205] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0206] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0207] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for detecting group faults, characterized in that, Applied to a control node, the method includes: Get read and write requests for business data stored in the storage layer; In response to the read / write request, obtain the monitoring status identifier of the storage layer; When the monitoring status is set to off, determine whether a mass failure occurs in the storage layer during the process of reading and writing the business data via read and write requests; In the event of a cluster failure in the storage layer, a storage layer failure alert will be generated. The monitoring status identifier is used to monitor the health status of the storage layer. When the monitoring status identifier is off, it indicates that the storage layer is in good health and has no faults. When the monitoring status identifier is on, it indicates that the storage layer has a fault, and the health status of the storage layer is monitored in real time.

2. The method according to claim 1, characterized in that, Determining whether a mass failure occurs in the storage layer during the process of reading and writing the business data through read and write requests includes: In response to the read / write request, read / write operations are performed on the business data; In the event of a read / write failure, obtain the reason for the failure. Based on the reasons for the read / write failures, determine whether a cluster failure has occurred in the storage layer.

3. The method according to claim 2, characterized in that, The step of determining whether a mass failure has occurred in the storage layer based on the reasons for the read / write failures includes: Obtain the characteristics of mass failures corresponding to different preset mass failures; The reasons for the read / write failures are matched with the characteristics of each of the aforementioned group failures; If the match fails, it is determined that no mass failure has occurred in the storage layer; If a match is successful, it is determined that a mass failure has occurred in the storage layer, and the mass failure corresponding to the successfully matched mass failure feature is taken as the target failure.

4. The method according to claim 1, characterized in that, The method further includes: When the monitoring status indicator is enabled, the read / write request is added to a pre-set waiting queue; If the read / write request fails to be added to the waiting queue, an alert will be displayed indicating that the waiting queue is full. If the read / write request is successfully added to the waiting queue, the timer is controlled to start timing the request. If the timer reaches a preset duration threshold, the read / write request is removed from the waiting queue, and a request timeout reminder is output.

5. The method according to claim 1, characterized in that, The method further includes: In the event of a mass failure in the storage layer, the monitoring status flag will be set to enabled.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: When the monitoring status indicator is enabled, the health status of the storage layer is monitored in real time; the monitoring includes at least one of storage environment monitoring, hardware status monitoring, software configuration monitoring, and performance monitoring. If the detection passes, the monitoring status indicator will be reset to "off".

7. A group fault detection device, characterized in that, The device includes: The request retrieval module is used to retrieve read and write requests for business data stored in the storage layer. The identifier acquisition module is used to acquire the monitoring status identifier of the storage layer in response to the read / write request; wherein, the monitoring status identifier is an identifier for monitoring the health status of the storage layer. When the monitoring status identifier is off, it indicates that the health status of the storage layer is good and there is no fault; when the monitoring status identifier is on, it indicates that the storage layer has a fault, and the health status of the storage layer is detected in real time. The fault diagnosis module is used to determine whether a mass failure occurs in the storage layer during the process of reading and writing the business data through read and write requests when the monitoring status is marked as off. The reminder generation module is used to generate a storage layer fault reminder in the event of a mass failure in the storage layer.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images