Sub-health detection method, software layer and sub-health detection equipment

Abstract

The invention provides a sub-health detection method, a software layer and a sub-health detection device.The method is used for detecting the health state of a storage system, the storage system comprises a plurality of nodes, and the method comprises the steps that a first target suspected sub-health component is selected from a plurality of components of the nodes, first intervention operation is conducted on the first target suspected sub-health component, and a first target suspected sub-health component is obtained; if yes, the first target suspected sub-health component stops working, then the effect of the first intervention operation is evaluated, and the first target suspected sub-health component is determined not to be a sub-health component under the condition that it is determined that the storage system still runs abnormally; under the condition of determining that the storage system operates normally, whether the source end IO pressure changes in the first intervention operation process is further judged, the effect of the source end IO pressure change on solving the problem of abnormal operation of the storage system is eliminated, whether the first target suspected sub-health component is the sub-health component can be accurately determined, and the sub-health detection accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of sub-health detection, and in particular to a sub-health detection method, software layer and sub-health detection equipment. Background Art

[0002] Storage systems are widely used in scenarios such as cloud storage and big data processing, and users are placing increasingly high demands on their reliability. Storage systems include components such as nodes, hardware, and software services. When a component experiences a clear fault, the storage system can accurately and quickly address it. However, components may also experience suboptimal performance, such as slow disks, slow node scheduling, and correctable error (CE) storms. When these components experience suboptimal performance, the storage system struggles to accurately and quickly address them.

[0003] Due to the varying processing capabilities of different components, different components will experience different latency in the same scenario. Therefore, we cannot simply use latency to determine whether a sub-health problem exists. A component with a larger latency may indeed be sub-healthy or normal, while a component with a smaller latency may indeed be normal or sub-healthy. Simply determining whether a component is sub-healthy based on whether the latency exceeds a single threshold can lead to numerous problems. For example, components with larger latency but not actually sub-healthy may be misjudged and incorrectly isolated, resulting in decreased system performance. Alternatively, components with smaller latency but actually sub-healthy may not be isolated, leading to fault propagation and performance degradation. Therefore, improving the accuracy of sub-health detection and identifying truly sub-healthy components is an urgent issue that needs to be addressed. Summary of the Invention

[0004] The present application discloses a sub-health detection method, software layer, and sub-health detection device. After isolating preliminarily suspected sub-health components, the method increases the judgment of the system operation status after isolation and the judgment of the IO pressure change during the isolation process, thereby avoiding the occurrence of incorrect isolation of normal components and missed isolation of sub-health components, thereby improving the accuracy of sub-health detection.

[0005] In the first aspect, the present application provides a sub-health detection method, which is used to detect the health status of a storage system. The storage system includes multiple nodes. The method includes: determining a first target suspected sub-health component from multiple components of the node, and then performing a first intervention operation on the first target suspected sub-health component to stop the first target suspected sub-health component from working. If the storage system still operates abnormally when the first target suspected sub-health component stops working, it is determined that the first target suspected sub-health component is not a sub-health component.

[0006] Exemplarily, the first target suspected sub-health component may be the node itself, hardware in the node, or software running on the node.

[0007] In the above process, after intervening in the first target suspected sub-healthy component, the intervention effect is further determined to judge whether the storage system is operating normally. If the storage system is still operating abnormally, it is determined that the intervention on the first target suspected sub-healthy component is ineffective, and the first target suspected sub-healthy component is not a sub-healthy component. This can avoid the occurrence of erroneous isolation of normal components, and can prompt to continue detecting sub-healthy components, thereby improving the accuracy of sub-health detection.

[0008] Exemplarily, the specific process of determining the first target suspected sub-healthy component from multiple components of the node is as follows:

[0009] According to the first business data of the node and the suspected sub-health component determination model, a first suspected sub-health component is determined from multiple components of the node, and a first target suspected sub-health component is determined from the first suspected sub-health component, wherein the suspected sub-health component determination model is trained based on historical business data and component labels corresponding to the historical business data, and the component labels include healthy, sub-healthy and abnormal; or, according to the first business data of the node and the identification algorithm, a first suspected sub-health component is determined from multiple components of the node, and a first target suspected sub-health component is selected and determined from the first suspected sub-health component.

[0010] In the above process, the first target suspected sub-health component is not determined solely based on the first business data, but is combined with a suspected sub-health component determination model or identification algorithm, which can increase the possibility that the predicted suspected sub-health component is a sub-health component, thereby improving the efficiency and accuracy of sub-health detection.

[0011] Exemplarily, after determining that the first target suspected sub-healthy component is not a sub-healthy component, the method also includes, in the case where there are multiple first suspected sub-healthy components, determining a second target suspected sub-healthy component from the remaining first suspected sub-healthy components other than the first target suspected sub-healthy component, and performing a second intervention operation on the second target suspected sub-healthy component, wherein the second intervention operation is used to stop the second target suspected sub-healthy component from working.

[0012] Through the above process, the present application can continue to select new target suspected sub-healthy components for intervention after determining that the first target suspected sub-healthy component of the initial intervention is not a sub-healthy component, until the storage system can operate normally after intervening in the suspected sub-healthy components, thereby avoiding the occurrence of situations such as mis-isolating normal components and missing isolation of sub-healthy components, thereby improving the accuracy of sub-health detection.

[0013] Exemplarily, the method further includes:

[0014] When it is determined that the storage system operates normally, whether the first source end IO pressure changes during the first intervention operation on the target suspected sub-healthy component is detected, wherein the first source end IO pressure is determined according to the number of IO requests input to the storage system.

[0015] If the first source-end IO pressure does not decrease, or if the decrease is less than an IO pressure threshold, the first target suspected sub-healthy component is determined to be a sub-healthy component. Alternatively, if the first source-end IO pressure changes to a second source-end IO pressure, the number of IO requests input to the storage system is controlled to obtain simulated IO pressure. Because the storage system operates abnormally under the simulated IO pressure, a second suspected sub-healthy component is determined from multiple components of the node based on the second service data of the node under the simulated IO pressure. If the value obtained by subtracting the second source-end IO pressure from the first source-end IO pressure is greater than the IO pressure threshold, the simulated IO pressure is less than the first source-end IO pressure, and the simulated IO pressure is greater than the second source-end IO pressure, the storage system operates abnormally under the simulated IO pressure.

[0016] Since the reason for the storage system to return to normal operation is not only the intervention of sub-healthy components, but also the reduction of source-side IO pressure during the intervention process, this application not only determines whether the storage system has returned to normal after the intervention of suspected sub-healthy components, but also adds a judgment on whether the source-side IO pressure has changed during the intervention process.

[0017] Through the above steps, when it is determined that the first source-end IO pressure does not decrease, or the degree of decrease is less than the IO pressure threshold, the influence of the source-end IO pressure on the restoration of normal operation of the storage system can be eliminated, and the first target suspected sub-healthy component is determined to be a sub-healthy component, thereby improving the accuracy of sub-health detection. However, when the first source-end IO pressure changes to the second source-end IO pressure, and the storage system operates normally under the second source-end IO pressure regardless of whether the first target suspected sub-healthy component is intervened, it is impossible to determine which reason determines the solution to the abnormal operation problem of the storage system. Therefore, by controlling the variable of the source-end IO pressure, re-determining the suspected sub-healthy component under a stable source-end IO pressure, and repeating all the above steps until the sub-healthy component is determined, the reliability of sub-health detection is improved.

[0018] Exemplarily, the method further includes: obtaining simulated IO pressure based on controlling the number of IO requests input to the storage system, and determining a first target suspected sub-healthy component through the simulated IO pressure, wherein a specific process of obtaining the simulated IO pressure is as follows: controlling the number of IO requests input to the storage system to gradually increase from a first IO request number to a second IO request number, so that the simulated IO pressure increases from the first simulated IO pressure to a second simulated IO pressure, wherein under the first simulated IO pressure, the storage system operates normally, and under the second simulated IO pressure, the storage system experiences abnormal operation.

[0019] Alternatively, the number of IO requests input to the storage system may be determined empirically to obtain simulated IO pressure, and the simulated IO pressure may be 70%, 80%, etc., of the IO pressure at the first source end.

[0020] The simulated I / O pressure determined through this process can both cause storage system anomalies and expose sub-healthy components, while also minimizing the impact on storage system performance. Executing this process during the initial detection phase of sub-healthy components eliminates the influence of I / O pressure on sub-healthy component detection results. Subsequently, only the effectiveness of interventions performed on the components needs to be determined to accurately identify sub-healthy components.

[0021] For example, when determining the first suspected sub-healthy components based on the first business data and the suspected sub-healthy component determination model or identification algorithm, the confidence level corresponding to each first suspected sub-healthy component can also be calculated. The confidence level is used to indicate the possibility that each first suspected sub-healthy component is a sub-healthy component.

[0022] The confidence level of the first target suspected sub-healthy component is higher than the confidence levels of other suspected sub-healthy components in the first target suspected sub-healthy components.

[0023] The confidence level of the second target suspected sub-healthy component is lower than that of the first target suspected sub-healthy component, and is higher than the confidence levels of other suspected sub-healthy components in the first target suspected sub-healthy component except the first target suspected sub-healthy component.

[0024] Through the above content, it can be determined that suspected sub-healthy components with high confidence can be given priority for intervention operations, so that sub-health detection can be carried out more efficiently and sub-healthy components can be identified.

[0025] Exemplarily, the method further includes: when the first target suspected sub-healthy component is determined to be a sub-healthy component, inputting the first business data and the sub-healthy component label corresponding to the first business data into a training set of a suspected sub-healthy component determination model.

[0026] The above process realizes the update of the training set of the suspected sub-healthy component determination model. Based on the updated training set, the suspected sub-healthy component determination model can be further updated, thereby improving the accuracy of the suspected sub-healthy component determination model, thereby determining more accurate suspected sub-healthy components, making sub-health detection more efficient and accurate.

[0027] In a second aspect, the present application provides a software layer including a sub-health detection service, which is applied to a storage system including a node. The software layer includes a business data acquisition module, a component determination module, a cluster control module, and an intervention effect evaluation module. Among them, the component determination module is used to determine a first target suspected sub-health component from multiple components of the node; the cluster control module is used to perform a first intervention operation on the first target suspected sub-health component, wherein the first intervention operation is used to stop the first target suspected sub-health component from working; the intervention effect evaluation module is used to determine that the first target suspected sub-health component is not a sub-health component when it is determined that the storage system is still operating abnormally.

[0028] Illustratively, a component is a node, hardware in a node, or software running on a node.

[0029] Exemplarily, the software layer also includes a model training module, wherein the model training module is used to train a suspected sub-healthy component determination model based on historical business data and component labels corresponding to the historical business data, wherein the component labels include healthy, sub-healthy and abnormal.

[0030] Exemplarily, when the software layer includes a model training module and the model training module trains a suspected sub-healthy component determination model, the component determination module is specifically used to determine the first suspected sub-healthy component based on the first business data of the node and the suspected sub-healthy component determination model, and determine the first target suspected sub-healthy component from the first suspected sub-healthy component.

[0031] Alternatively, when the software layer does not include a model training module, the component determination module is specifically used to determine a first suspected sub-healthy component based on the first business data of the node and an identification algorithm, and to determine a first target suspected sub-healthy component from the first suspected sub-healthy components.

[0032] Exemplarily, after the intervention effect evaluation module determines that the first target suspected sub-healthy component is not a sub-healthy component, the component determination module is also used to, when there are multiple first suspected sub-healthy components, determine a second target suspected sub-healthy component from the remaining first suspected sub-healthy components other than the first target suspected sub-healthy component, and then enable the cluster control module to perform a second intervention operation on the second target suspected sub-healthy component, wherein the second intervention operation is used to stop the second target suspected sub-healthy component from working.

[0033] Exemplarily, the intervention effect evaluation module is further used to detect whether the first source-end IO pressure changes during the process of performing a first intervention operation on a first target suspected sub-healthy component when it is determined that the storage system is operating normally, wherein the first source-end IO pressure is determined based on the number of IO requests input to the storage system; if the first source-end IO pressure does not decrease, or the degree of decrease is less than an IO pressure threshold, it is determined that the first target suspected sub-healthy component is a sub-healthy component.

[0034] Exemplarily, the software layer also includes a pressure degradation dial detection module. The pressure degradation dial detection module is configured to control the number of IO requests input to the storage system to obtain simulated IO pressure when the intervention effect evaluation module detects that the first source-end IO pressure changes to the second source-end IO pressure during a first intervention operation on a first target suspected sub-healthy component. This allows the component determination module to determine a second suspected sub-healthy component from multiple components of the node based on second business data of the node under the simulated IO pressure, wherein the value obtained by subtracting the second source-end IO pressure from the first source-end IO pressure is greater than an IO pressure threshold, and the simulated IO pressure is less than the first source-end IO pressure and greater than the second source-end IO pressure.

[0035] Exemplarily, the pressure degradation dialing module is further configured to obtain simulated IO pressure according to the number of IO requests input to the storage system, so that the component determination module determines the first target suspected sub-healthy component under the simulated IO pressure.

[0036] The pressure degradation dialing test module is specifically used to control the number of IO requests input to the storage system to gradually increase from a first IO request number to a second IO request number, so that the simulated IO pressure increases from the first simulated IO pressure to the second simulated IO pressure. Under the first simulated IO pressure, the storage system operates normally, and under the second simulated IO pressure, the distributed storage system experiences abnormal operation problems.

[0037] For example, when determining the first suspected sub-healthy component, the component determination module can also calculate the confidence level corresponding to each first suspected sub-healthy component through the suspected sub-healthy component determination model or identification algorithm. The confidence level is used to indicate the possibility that each first suspected sub-healthy component is a sub-healthy component.

[0038] The confidence level of the first target suspected sub-healthy component is higher than the confidence levels of other suspected sub-healthy components in the first target suspected sub-healthy components.

[0039] The confidence level of the second target suspected sub-healthy component is lower than that of the first target suspected sub-healthy component, and is higher than the confidence levels of other suspected sub-healthy components in the first target suspected sub-healthy component except the first target suspected sub-healthy component.

[0040] Exemplarily, the intervention effect evaluation module is further used to input the first business data and the sub-healthy component label corresponding to the first business data into the model training module when determining that the first target suspected sub-healthy component is a sub-healthy component, and update the training set of the suspected sub-healthy component determination model.

[0041] In a third aspect, the present application provides a sub-health detection device, which includes a processor and a memory. The processor is used to execute instructions stored in the memory of the sub-health detection device so that the sub-health detection device executes the method in the first aspect or any possible implementation of the first aspect.

[0042] In a fourth aspect, the present application provides a computer program product, which, when executed by a sub-health detection device, implements the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0043] In a fifth aspect, the present application provides a computer-readable storage medium comprising computer instructions, which, when executed by a sub-health detection device, implement the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0044] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments.

[0046] Figure 1 This is a schematic diagram of the structure of a distributed storage system provided by an embodiment of the present application;

[0047] Figure 2This is a schematic diagram of the structure of a software layer provided in an embodiment of the present application;

[0048] Figure 3 This is a flow chart of a sub-health detection method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of the present invention.

[0050] Sub-health refers to a state where a component is between completely normal (healthy) and faulty. A sub-healthy component does not meet the standard of health, but has not yet reached the standard of failure. For example, it may experience abnormalities such as performance degradation, increased power consumption, and reduced reliability, but it may still function. Sub-healthy components are more likely to fail in the future than healthy components. After detecting sub-healthy components, they can be replaced or repaired before they actually fail. For storage systems, the adverse effects of sub-health include: slow links, slow message responses, suspended IO, or continuous IO access failures.

[0051] Components include: nodes (e.g., storage nodes, compute nodes), hardware within a node (e.g., power supply, CPU, memory, network interface card, GPU card, backplane, fan, cables), and software or software modules running on the node (e.g., image processing software, database software, data backup software, data compression software module, data deduplication software module, QoS software module, etc.). Any component that can affect the performance of the storage system and can be stopped is considered a component in the embodiments of the present invention.

[0052] In order to solve the problems of misjudgment and mis-isolation of non-sub-healthy components, as well as missed judgment and missed isolation of sub-healthy components, the present application provides a sub-healthy detection method. After determining a suspected sub-healthy component and preliminarily intervening in the suspected sub-healthy component, the method further determines whether the suspected sub-healthy component is a real sub-healthy component by evaluating the effect of the intervention operation on the suspected sub-healthy component and judging whether there is fluctuation in the source IO pressure during the intervention operation. The source IO pressure is determined based on the read and write request volume of the input storage system. In the case of source IO pressure fluctuations, by simulating a stable source IO pressure environment, the influence of the source IO pressure change on the effect of intervening in the suspected sub-healthy component is eliminated, thereby accurately determining whether the suspected sub-healthy component is a real sub-healthy component, improving the reliability of sub-healthy detection, reducing the occurrence of mis-isolation of non-sub-healthy components or missed isolation of real sub-healthy components, and accurately intervening in real sub-healthy components to improve storage system performance.

[0053] First, the application scenarios of the sub-health detection method provided in this application are explained.

[0054] The sub-health detection method provided in this application is applied to a storage system, which can be a distributed storage system or a centralized storage system, and this application does not specifically limit this. Since a distributed storage system is usually composed of multiple nodes (devices), and a centralized storage system is often composed of a few nodes, a distributed storage system has a large number of nodes, the probability of node failure increases, and load balancing is more complicated. Therefore, sub-health problems of components are more likely to occur in a distributed storage system, and the accuracy requirements for detection and intervention of sub-healthy components are higher. Therefore, the following will be described in detail using a distributed storage system as an example.

[0055] like Figure 1 As shown, Figure 1 1 is a structural diagram of a distributed storage system provided in an embodiment of the present application. The distributed storage system includes a computing node cluster 11, a storage node cluster 12 and a software layer 13.

[0056] Computing node cluster 11 includes one or more computing nodes 110, each of which can communicate with one another. Computing node 110 is a computing device, which may be a server, a desktop computer, or a storage array controller, etc., and this application does not specifically limit this. Computing node 110 is used to process data access requests from outside computing node 110 or requests generated within computing node 110, and this application does not specifically limit this.

[0057] In one possible implementation, Figure 1As shown, the computing node 110 includes at least a processor 111 , a memory 112 , a communication interface 113 and a bus 114 .

[0058] The processor 111 may be composed of at least one general-purpose processor, such as a central processing unit (CPU), or a combination of a CPU and a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The processor 111 may be used to execute logical operations of software services and implement corresponding functions.

[0059] Memory 112 can be volatile memory, such as random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), double data rate RAM (DDR), cache, etc. Memory can also include a combination of the above. Memory 112 stores data or software services, which is not specifically limited in this application.

[0060] The communication interface 113 uses, for example but not limited to, a transceiver module such as a network interface card or a transceiver, which is not specifically limited in this application.

[0061] The bus 114 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc., and may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 1The use of only one line does not necessarily mean that there is only one bus or only one type of bus. Bus 114 may include a path for transmitting information between various components of the computing node (eg, processor 111, memory 112, communication interface 113).

[0062] Need to explain, Figure 1 This is only one possible example of a computing node provided in the embodiment of the present application. In actual applications, the computing node may also include more or fewer components, which is not limited here. Any computing node 110 can access any storage node 120 in the storage node cluster through the network.

[0063] Storage node cluster 12 includes one or more storage nodes 120, each of which can communicate with each other. Storage node 120 can be a server, desktop computer, or storage array controller, disk enclosure, etc., which is not specifically limited in this application. Storage node 120 is used to perform calculations or processing on data, which is not specifically limited in this application.

[0064] In one possible implementation, Figure 1 As shown, the storage node is a storage server, a general server, or a storage device, wherein the storage node at least includes a controller 121 , a hard disk 122 , and a network card 123 .

[0065] The controller 121 includes a processor 131 and a memory 132. The processor 131 is used to perform operations such as reading and writing data and performing address conversion during the reading and writing process. It can also perform logical operations of software services to implement corresponding functions, which is not specifically limited in this application. The memory 132 is used to temporarily store data to be written to the hard disk 122, or read data from the hard disk 122 to be sent to the computing node 110. In addition to the possible examples above, the controller 121 can also have other forms of expression. This application does not specifically limit the form and number of the controller.

[0066] In one possible implementation, the controller of the storage node may run an event delivery service (EDS) process, or one or more object storage device (OSD) processes. The EDS process is used to manage and transmit events occurring in the storage system, such as storage node status changes (e.g., load status changes, availability status changes, storage capacity changes, etc.), configuration changes (e.g., network configuration changes, data protection policy changes), and other events, which are not specifically limited in this application. By monitoring the above events, the EDS can trigger corresponding operations when the above events meet the conditions, or notify other components to trigger corresponding operations. The OSD process is used to receive, store, and retrieve data objects, where the objects can be files, blocks, or other forms of data, which are not specifically limited in this application. The OSD process is used to distribute data, process data read and write requests, and coordinate the management of object storage, etc. It can be understood that the controller of the storage node can also run more types and numbers of processes, which are not specifically limited in this application.

[0067] The hard disk 122 is used to store data and can be a magnetic disk. In other embodiments, it can also be replaced with other types of non-volatile storage media, such as a solid-state drive or a shingled magnetic recording hard disk. This application does not specifically limit the type and quantity of non-volatile storage media.

[0068] The network card 123 is used to access any hard disk 122 in the storage node, process the sending and receiving of data packets, and realize communication with the computing node 110 or the software layer 13.

[0069] Need to explain, Figure 1 This is only one possible example of a storage node provided in an embodiment of the present application. In actual applications, the storage node may also include more types and quantities of components, and the present application does not impose any specific restrictions on this.

[0070] Since the computing nodes and storage nodes each include a processor, the processor can be used to execute the logical operations of the software service. Therefore, the computing node, or one or more nodes in the storage node can execute the multiple software services included in the software layer to achieve certain functions. In addition, in the case where there is a management node selected from the storage nodes in the distributed storage system, the management node can also execute the logical operations of the above-mentioned software services, which is not specifically limited in this application. For example, in the case where the software layer includes multiple software services such as data storage services, metadata services, and access control services, the computing node, or one or more nodes in the storage node can realize functions such as storage and management of distributed data by executing the above-mentioned services.

[0071] In order to solve the problems of misjudgment and misisolation of non-sub-healthy components, and missed judgment and missed isolation of sub-healthy components in current sub-healthy detection methods, in addition to the above-mentioned functions of storing and managing distributed data, the software layer 13 is also used to find suspected sub-healthy components and determine whether the suspected sub-healthy components are truly sub-healthy components.

[0072] Since it is rare for two or more sub-healthy components to appear at the same time in a distributed storage system, and once a truly sub-healthy component is identified, it is processed to prevent the problem from accumulating and leading to the appearance of multiple sub-healthy components, the following description uses the example of a distributed storage system with only one sub-healthy component. The software layer 13 can identify suspected sub-healthy components and determine whether the suspected sub-healthy component is a true sub-healthy component in the following manner:

[0073] The software layer 13 obtains the business data of the storage node, finds suspected sub-healthy components based on the business data, isolates the suspected sub-healthy components, and after isolation is completed, determines whether the distributed storage system has returned to normal;

[0074] (1) If the distributed storage system returns to normal, determine whether the source end IO pressure has been reduced;

[0075] (1) If the IO pressure does not decrease, it is determined that the suspected sub-healthy component is indeed a sub-healthy component;

[0076] (2) If the IO pressure decreases, simulate stable IO pressure, then find the suspected sub-healthy components again and isolate them. After isolating the suspected sub-healthy components found again, re-determine whether the distributed storage system has returned to normal;

[0077] (a) If it returns to normal, the suspected sub-healthy component found again is considered to be a sub-healthy component;

[0078] (b) If it does not return to normal, the suspected sub-healthy component found again is considered not to be a sub-healthy component;

[0079] (2) If the distributed storage system does not return to normal, a new suspected sub-healthy component is selected for isolation until a sub-healthy component is found, or until there are no suspected sub-healthy components that have not been isolated.

[0080] like Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of a software layer provided in an embodiment of the present application. Figure 2As shown, the software layer in this application includes a fault detection and self-healing system 20, which includes a service data query module 21, a component identification module 22, a cluster control module 23, an intervention effect evaluation module 24, and a pressure degradation dialing module 25. The following details how these modules work together to achieve the aforementioned functions of identifying suspected sub-healthy components and determining whether the suspected sub-healthy components are truly sub-healthy components.

[0081] The business data query module 21 is used to obtain business data from each storage node.

[0082] Business data includes the internal IO read and write delays, interface call status, memory usage, CPU usage, etc. of the business process. In one possible implementation, business data can also include more types and quantities of data such as hardware failures, which is not specifically limited in this application.

[0083] The business data query module may trigger execution to obtain business data from each storage node 120 in at least two of the following situations:

[0084] In the first approach, when the business data query module determines that any computing node 110 detects an abnormality in the operation of the distributed storage system, it triggers execution to obtain business data from each storage node 120. The abnormality in the operation of the distributed storage system may include, for example, a slow link, slow message transmission, a node being blocked for a long time and unable to complete IO operations normally, and continuous read and write failures. In addition to the above-mentioned abnormalities, in actual applications, there may be more types of abnormalities, which are not specifically limited in this application.

[0085] Whether the operation of the distributed storage system is abnormal is determined by whether all computing nodes in the distributed storage system are normal. If all computing nodes are normal, it is determined that the operation of the distributed storage system is normal. If any one or more computing nodes are abnormal, it is determined that the operation of the distributed storage system is abnormal. The method of determining whether a computing node is abnormal can include at least two methods:

[0086] The first method is to determine whether the operation of the computing node is abnormal from the external perspective of the computing node. Specifically, in the distributed storage system, other computing nodes except the computing node can continuously obtain the system load, request response time and data access response time from the computing node according to a certain time interval, and determine whether the operation of the computing node is abnormal based on the obtained data. Among them, the system load is used to indicate the business volume of the computing node, the request response time is used to indicate the time required for the computing node to process the IO request and return a response, and the data access response time is used to indicate the time required to access the data in the storage node. In actual applications, the computing node can also obtain more types and quantities of data to determine whether the operation of the computing node is abnormal, and this application does not make specific restrictions on this.

[0087] The second method is to determine whether the computing node is operating abnormally from the node's internal perspective. Specifically, the computing node continuously obtains information about its own hardware failures, disk availability, and other conditions at regular intervals, or uses its own recorded logs to determine whether the computing node is operating abnormally.

[0088] It is understandable that other methods may be used to detect whether abnormal problems occur in the operation of the computing node, and this application does not make specific limitations on this.

[0089] In the second manner, the business data query module is triggered and executed periodically to obtain the business data in each storage node 120 .

[0090] In actual applications, the business data query module can also be triggered to execute and obtain the business data in each storage node 120 in other ways, and this application does not make specific limitations on this.

[0091] The service data query module obtains service data sent by each storage node 120. The storage node 120 obtains service data in at least the following ways:

[0092] (1) The storage node obtains the business data in each storage node 120 through the running EDS process and OSD process.

[0093] (2) The storage node obtains the business data in each storage node 120 through the business dot code in the storage system. Taking the interface call as an example, the dot code can be inserted before and after the function call. For example, the code (tag) can be inserted at the start and end of the function call. Then, by running the business dot code (tag), the function execution time, function call count, function input parameters and return value, etc. are collected, and the interface call status is determined based on the function execution time, function call count, function input parameters and return value, etc.

[0094] (3) The storage node can obtain the business data in each storage node 120 by performing log analysis.

[0095] In addition to the above methods, in actual applications, storage nodes can also collect business data through other processes and monitoring tools, which is not specifically limited in this application.

[0096] The business data query module 21 , after acquiring the business data from each storage node, sends the business data to the component determination module 22 .

[0097] The component determination module 22 is configured to determine one or more suspected sub-healthy components based on the business data.

[0098] The component determination module can determine one or more suspected sub-healthy components in a variety of ways, two of which are listed below for illustration.

[0099] The first way: when the component determination module can obtain the suspected sub-healthy component determination model, the component determination module determines one or more suspected sub-healthy components according to the business data and the trained suspected sub-healthy component determination model.

[0100] Because the suspected sub-health component determination model needs to be trained before it can be used normally, the software layer can also include a model training module 26. The model training module 26 is used to train the untrained suspected sub-health component determination model based on historical business data and the component labels corresponding to the historical business data to obtain a trained suspected sub-health component determination model, where the component labels include healthy, sub-healthy, and abnormal. The suspected sub-health component determination model can be an AI model, a machine learning model, or a big data model. This application does not specifically limit the type of suspected sub-health component determination model.

[0101] The model training module 26 can persistently store the weights and structure of the suspected sub-health component determination model obtained through training, so that the component determination module can load the suspected sub-health component determination model from the persistent storage medium and determine one or more suspected sub-health components based on the business data and the suspected sub-health component determination model.

[0102] In one possible implementation, in addition to identifying one or more suspected sub-healthy components, the component determination module can also determine the confidence level associated with each suspected sub-healthy component. The confidence level indicates the likelihood that a suspected sub-healthy component is actually sub-healthy. The higher the confidence level, the more likely the suspected sub-healthy component is. Therefore, after determining multiple suspected sub-healthy components based on the suspected sub-healthy component determination model, the components can be sorted in descending order based on their respective confidence levels. Finally, the component determination module inputs the suspected sub-healthy component information with the highest confidence level into the cluster control module 23.

[0103] The second way: when the component determination module includes an identification algorithm, the component determination module determines one or more suspected sub-healthy components according to the business data and the identification algorithm.

[0104] Because the identification algorithm does not need to be trained before it can be used, the model training module 26 may not be included in the software layer. The component determination module determines whether the communication between the storage nodes is normal based on the business data, and then inputs the data indicating whether the communication between the storage nodes is normal into the identification algorithm, thereby determining one or more suspected sub-healthy components. For example, when the component determination module determines based on the business data that the communication between storage node 2 and storage node 3 is normal, the communication between storage node 2 and storage node 1 is abnormal, and the communication between storage node 3 and storage node 1 is abnormal, through the identification algorithm, storage node 2 and storage node 3 can identify storage node 1 as abnormal. At this time, the component determination module determines that storage node 1 is a suspected sub-healthy component. The component determination module sends the one or more suspected sub-healthy components confirmed by the identification algorithm to the cluster control module 23.

[0105] In one possible implementation, the component determination module can also indicate the confidence level corresponding to one or more suspected sub-healthy components based on the identification algorithm. For example, the more storage nodes that are identified as having abnormal communication with a storage node, the more likely that the storage node is a sub-healthy component, and the greater the confidence level corresponding to the storage node. Therefore, after determining multiple suspected sub-healthy components based on the identification algorithm, the multiple suspected sub-healthy components can also be sorted in descending order based on the confidence level corresponding to each suspected sub-healthy component. Finally, the component determination module inputs the information of the suspected sub-healthy component with the highest confidence level into the cluster control module 23.

[0106] The cluster control module 23 is used to perform intervention operations on target suspected sub-health components. The target suspected sub-health components can be obtained in at least two ways:

[0107] In the first approach, if the component determination module sends only one suspected sub-healthy component to the cluster control module, that suspected sub-healthy component is the target suspected sub-healthy component. Furthermore, if the component determination module inputs the suspected sub-healthy component with the highest confidence level, as determined by the suspected sub-healthy component determination model, into the cluster control module, the component determination module sends only one suspected sub-healthy component to the cluster control module. Alternatively, if the component determination module identifies only one suspected sub-healthy component through the identification algorithm, the component determination module sends only one suspected sub-healthy component to the cluster control module.

[0108] In the second approach, when the component determination module sends multiple suspected sub-healthy components to the cluster control module, one suspected sub-healthy component is selected from the multiple suspected sub-healthy components as the target suspected sub-healthy component. When the component determination module confirms that there are multiple suspected sub-healthy components using an identification algorithm, the component determination module sends the multiple suspected sub-healthy components to the cluster control module. The cluster control module selects one suspected sub-healthy component from the multiple suspected sub-healthy components as the target suspected sub-healthy component based on a selection algorithm. The selection algorithm may include a random selection algorithm, dynamic programming, a backtracking algorithm, or the like.

[0109] The cluster control module 23 can find the target suspected sub-healthy component according to the target suspected sub-healthy component identifier and perform intervention operations on the target suspected sub-healthy component. The intervention operations include isolation, fuse, and recovery. Here, the cluster control module only performs isolation operations on the target suspected sub-healthy component.

[0110] After the cluster control module 23 performs intervention operations on the target suspected sub-health component, the intervention effect evaluation module 24 starts to run. In a possible embodiment, the intervention effect evaluation module 24 is as follows Figure 2 As shown, it is deployed in the FDSA system, and can also be deployed in a software layer outside the FDSA system, which is not specifically limited in this application.

[0111] The intervention effect evaluation module 24 is used to evaluate the success of the intervention operation on the target suspected sub-healthy component. The intervention effect evaluation module determines whether the intervention operation on the target suspected sub-healthy component was successful by evaluating whether there were any anomalies in the operation of the distributed storage system. If the distributed storage system is operating normally, the intervention operation on the target suspected sub-healthy component is determined to be successful. If the distributed storage system is operating abnormally, the intervention operation on the target suspected sub-healthy component is determined to have failed. The method for determining whether there were any anomalies in the operation of the distributed storage system can be found in the description of the business data query module 21 above and will not be repeated here.

[0112] The following describes two situations in which the intervention effect evaluation module determines that the intervention operation on the target suspected sub-health component fails or succeeds.

[0113] (1) When the intervention effect evaluation module determines that the intervention operation on the target suspected sub-health component has failed, the distributed storage system still has abnormal operation, and it is determined that the target suspected sub-health component is not a real sub-health component. In the case that there are multiple suspected sub-health components, the component determination module reselects another suspected sub-health component from the multiple suspected sub-health components as the new target suspected sub-health component, and sends the identifier of the other suspected sub-health component to the cluster control module, so that the cluster control module performs an intervention operation on the other suspected sub-health component until the real sub-health component is found, or until the intervention operation is performed on all multiple suspected sub-health components. The selection method of the other suspected sub-health component may include the following:

[0114] (1) Randomly select another suspected sub-healthy component from multiple suspected sub-healthy components.

[0115] (2) If multiple suspected sub-health components each have a confidence level, selection can be made based on the confidence level. During selection, suspected sub-health components with high confidence levels are selected first, while suspected sub-health components with low confidence levels are selected later.

[0116] (2) When the intervention effect evaluation module determines that the intervention operation on the target suspected sub-health component is successful, it is also necessary to determine whether the source-side IO pressure changes during the intervention operation on the target suspected sub-health component. If the source-side IO pressure happens to decrease during the intervention operation, and the degree of decrease is greater than the IO pressure threshold, then the reason for the abnormal operation of the distributed storage system to be resolved may be due to the decrease in source-side IO pressure, and not necessarily due to the intervention operation on the target suspected sub-health component. The IO pressure threshold is set based on experience. At this point, further determination is required. The way to further determine can be:

[0117] When the source-side IO pressure decreases and exceeds the IO pressure threshold, the pressure degradation dial detection module 25 is triggered to run. The pressure degradation dial detection module 25 is used to simulate a stable source-side IO pressure environment. To avoid significantly impacting the performance of the distributed storage system, the simulated stable source-side IO pressure is lower than the source-side IO pressure corresponding to the aforementioned process of detecting abnormal distributed storage system operation by the computing node. However, within this stable source-side IO pressure environment, the computing node can still detect abnormal distributed storage system operation.

[0118] In one possible implementation, based on experience, the stable source-side IO pressure can be 70% or 80% of the source-side IO pressure during the process of the computing node detecting abnormal operation of the distributed storage system; alternatively, the pressure degradation dialing test module can first simulate a smaller stable source-side IO pressure and dynamically increase the source-side IO pressure until the computing node detects abnormal operation of the distributed storage system in the simulated source-side IO pressure environment.

[0119] When the pressure degradation dial test module simulates a stable source-end IO pressure environment, the business query module obtains new business data so that the component determination module determines a new suspected sub-healthy component and performs an intervention operation on the newly determined suspected sub-healthy component, or the cluster control module re-intervention operations on the target suspected sub-healthy component, which is not specifically limited in this application. If the source-end IO pressure does not change during the intervention operation, the degree of reduction is less than the IO pressure threshold, or even the IO pressure increases, and the distributed storage system operation anomaly is resolved, then it can be determined that the abnormality of the distributed storage system operation is resolved due to the intervention operation on the target suspected sub-healthy component, and then it is determined that the target suspected sub-healthy component is a real sub-healthy component. Afterwards, an intervention operation is performed on the sub-healthy component to restore the sub-healthy component so that the distributed storage system can operate normally.

[0120] In addition to the above-mentioned case of triggering the pressure degradation dial detection module 25, the pressure degradation dial detection module can also be triggered in the initial stage of sub-health detection, so as to obtain and maintain a stable simulated IO pressure, so that the component determination module, the cluster control module, and the intervention effect evaluation module can perform the above-mentioned corresponding functions under the stable simulated IO pressure to realize the detection of sub-healthy components. This application does not specifically limit the triggering time of the pressure degradation dial detection module.

[0121] When the target suspected sub-health component is a real sub-health component and the software layer includes a model training module 26, the business data and the corresponding sub-health component label can be added to the training set of the suspected sub-health component determination model, and the suspected sub-health component determination model can be further trained based on the updated training set to improve the accuracy of the suspected sub-health component determination model.

[0122] I understand. Figure 2 This is only an example of a possible software layer provided in an embodiment of the present application. In actual applications, the fault detection and self-healing system may also include more or fewer modules. In addition to performing the corresponding operations in the above content, each module may also implement more or less content. This application does not make specific limitations on this. In addition, the software layer may also include more types and quantities of software services or components, which this application does not make specific limitations on.

[0123] I understand. Figure 1 This is only an example of a possible structure of a distributed storage system provided in an embodiment of the present application. There may be other forms in actual applications, and this application does not make any specific limitations on this.

[0124] In order to solve the problem of inaccurate detection of sub-healthy components in current storage systems, an embodiment of the present application provides a sub-healthy detection method. After determining suspected sub-healthy components based on business data, this method further determines whether the suspected sub-healthy components are real sub-healthy components by evaluating the effect of intervention operations on the suspected sub-healthy components and judging whether the source end IO pressure changes during the intervention operation. Compared with the current method of judging sub-healthy components based on a single threshold, the judgment conditions of sub-healthy components are increased, and the reliability of sub-healthy detection is improved, thereby reducing the occurrence of misisolation or the inability to isolate real sub-healthy components, and accurately determining the real sub-healthy components.

[0125] exist Figure 2 When the software layer shown includes the model training module 26, the flowchart of a sub-health detection method provided by the embodiment of the present application is as follows: Figure 3 As shown, this method is applied to Figure 1 In the system shown, the method includes the following steps.

[0126] Step S310: When it is determined that the distributed storage system is operating abnormally, business data of the storage node is obtained.

[0127] The method for determining whether the distributed storage system is operating abnormally is to Figure 2 The business data query module shown in FIG is described in detail and will not be repeated here.

[0128] When it is determined that the distributed storage system is operating abnormally, the processor runs the business data statistics module to obtain the business data of the storage node. Figure 2 This is also explained in detail during the description of the business data query module shown, so it will not be repeated here.

[0129] If the distributed storage system is found to be operating abnormally, a sub-health detection method can be triggered. The business data obtained from the storage nodes can be used to subsequently identify suspected sub-healthy components. This business data includes multiple types of data, which is more reliable than determining sub-healthy components based on a single threshold. Otherwise, if the distributed storage system is operating normally and there are no suspected sub-health issues, sub-health detection is not required, and subsequent steps such as obtaining business data from storage nodes are not necessary.

[0130] Step S320: determining one or more suspected sub-healthy components according to the service data of the storage node, and determining a target suspected sub-healthy component therefrom.

[0131] Suspected sub-health components include hardware and software services in the storage node, and may also include the storage node itself, which is not specifically limited in this application.

[0132] Because the software layer includes a model training module, the model training module can train a suspected sub-health component determination model and store the weights and structure of the suspected sub-health component determination model in a permanent storage medium. Therefore, the processor can obtain the suspected sub-health component determination model from the permanent storage medium. After obtaining the business data of the storage node, the processor can obtain one or more suspected sub-health components and the confidence level corresponding to each suspected sub-health component based on the business data and the suspected sub-health component determination model. In addition, if there are multiple suspected sub-health components, the multiple suspected sub-health components can be sorted from largest to smallest according to their confidence level.

[0133] For example, when only one suspected sub-healthy component is determined by the above method, the suspected sub-healthy component is the target suspected sub-healthy component.

[0134] If multiple suspected sub-healthy components are identified using the above method, one suspected sub-healthy component is selected from the multiple suspected sub-healthy components as a target suspected sub-healthy component. If the confidence levels corresponding to the suspected sub-healthy components are calculated, the target suspected sub-healthy component may be the suspected sub-healthy component with the highest confidence level. If the confidence levels corresponding to the suspected sub-healthy components are not calculated, the target suspected sub-healthy component may also be determined from the multiple suspected sub-healthy components using a random selection algorithm, etc., which is not specifically limited in this application.

[0135] Because the suspected sub-healthy component identification model is trained by the model training module using historical business data and the component labels corresponding to that historical business data, the above process accurately identifies one or more suspected sub-healthy components based on the acquired business data and the suspected sub-healthy component identification model. Furthermore, the above process does not determine whether these suspected sub-healthy components are truly sub-healthy components; further confirmation is required in subsequent steps. This more rigorous sub-healthy component detection process improves the accuracy of sub-healthy component identification and reduces the occurrence of problems such as mis-isolating healthy components.

[0136] Step S330: performing intervention operations on the target suspected sub-healthy component.

[0137] After determining the target suspected sub-healthy component, the processor determines the target suspected sub-healthy component based on the target suspected sub-healthy component identifier and performs intervention operations on the target suspected sub-healthy component. The intervention operations include isolation, fuse and recovery operations. The intervention operation here refers to isolating the target suspected sub-healthy component so that the target suspected sub-healthy component no longer works.

[0138] Because the issue of mistakenly isolating healthy components currently exists, the isolation of the suspected sub-healthy component does not terminate the process. Further evaluation of the distributed storage system's operation after the isolation of the suspected sub-healthy component is required to determine whether the suspected sub-healthy component is truly sub-healthy. This is illustrated in S340. By further evaluating the distributed storage system's operation after isolating the suspected sub-healthy component, problems such as mistakenly isolating healthy components can be avoided, improving the accuracy of the sub-health detection method.

[0139] Step S340: Evaluate whether the intervention operation on the target suspected sub-health component is successful. If the intervention operation fails, execute step S350; if the intervention operation is successful, execute step S360.

[0140] After performing the intervention operation on the target suspected sub-healthy component, the processor determines whether the intervention operation on the target suspected sub-healthy component is successful by evaluating whether an abnormality occurs in the operation of the distributed storage system.

[0141] For example, the processor can determine whether the distributed storage system is operating abnormally by comparing the IO download time before and after intervention on a suspected sub-healthy target component, thereby determining whether the intervention was successful. The IO download time indicates the time required for an application to send an IO request to the storage device and complete the IO operation. A large IO download time can indicate performance issues with the storage device, leading to increased latency, excessive system load, and other issues, and thus abnormal operation of the distributed storage system.

[0142] When the time consumed by the next IO before intervening in the target suspected sub-healthy component is slightly different from the time consumed by the next IO after intervening in the target suspected sub-healthy component, the processor determines that the operation of the distributed storage system is still abnormal, the intervention operation on the target suspected sub-healthy component fails, and executes step S350.

[0143] When the time consumed by the next IO before intervening in the target suspected sub-healthy component is greater than the time consumed by the next IO after intervening in the target suspected sub-healthy component, and the difference is large, the processor determines that the operation of the distributed storage system has returned to normal and the intervention operation on the target suspected sub-healthy component is successful. At this time, step S360 is executed.

[0144] The above process, after isolating the suspected sub-healthy components of the target, further determines whether the distributed storage system is operating abnormally. It can preliminarily determine whether there is a situation where normal components are mistakenly isolated, thereby improving the accuracy of sub-health detection.

[0145] Step S350: Select another suspected sub-health component to perform intervention operation.

[0146] In one possible embodiment, multiple suspected sub-healthy components are determined in step S320, and the multiple suspected sub-healthy components and the confidence levels corresponding to the suspected sub-healthy components are saved in a buffer. The processor can preferentially select a suspected sub-healthy component with a high confidence level as another suspected sub-healthy component for intervention operation. For example, when the target suspected sub-healthy component for intervention previously is the suspected sub-healthy component with the highest confidence level, the suspected sub-healthy component with the second highest confidence level can be selected as another newly selected suspected sub-healthy component for intervention operation.

[0147] In another possible implementation, multiple suspected sub-healthy components are determined in step S320 and saved in a buffer. The processor may randomly select another suspected sub-healthy component from the multiple suspected sub-healthy components except for the target suspected sub-healthy component previously intervened, and perform an intervention operation.

[0148] When it is determined that the intervention operation on the target suspected sub-healthy component has failed, it can be determined that the target suspected sub-healthy component is not a real sub-healthy component. The above method selects another suspected sub-healthy component and performs an intervention operation on it, and executes subsequent steps S330 and S340 in sequence until the intervention operation is successful. This can avoid the misisolation of normal components and the omission of isolation of real sub-healthy components, thereby achieving accurate sub-health detection.

[0149] Step S360: Determine whether the source-side IO pressure has changed. If the source-side IO pressure has decreased and the degree of decrease is greater than or equal to the IO pressure threshold, proceed to step S370. If the source-side IO pressure has not decreased or the degree of decrease is less than the IO pressure threshold, proceed to step S380.

[0150] Since during the intervention operation on the target suspected sub-healthy component, the source-side IO pressure becomes smaller, which can also solve the problem of abnormal operation of the distributed storage system, thereby interfering with the sub-health detection, wherein the source-side IO pressure is determined according to the number of IO requests input to the storage system. The processor cannot determine whether the solution to the problem of abnormal operation of the distributed storage system in step S340 is successful, and whether the intervention operation is successful because the isolated target suspected sub-healthy component is a sub-healthy component, or because the source-side IO pressure is reduced due to the reduction in the number of IO requests input to the storage system. Therefore, in order to accurately determine the sub-healthy component, it is necessary to eliminate the impact of the reduction in source-side IO pressure on solving the problem of abnormal operation of the distributed storage system.

[0151] The sub-health detection provided by this application adds a judgment on the change of IO pressure at the source end. Only when the IO pressure at the source end does not decrease, or the degree of decrease is less than the IO pressure threshold, it is determined that the target suspected sub-health component of the current intervention is a sub-health component. When the IO pressure at the source end decreases, and the degree of decrease is greater than or equal to the IO pressure threshold, it cannot be determined that the target suspected sub-health component of the current intervention is a sub-health component, and further judgment is required. This application takes the change of IO pressure at the source end into consideration during the sub-health detection process, which can improve the accuracy of sub-health detection, determine the real sub-health components, and reduce the occurrence of mis-isolation of normal components and missed isolation of sub-health components.

[0152] Step S370: Maintaining a stable simulated IO pressure.

[0153] When the source-side IO pressure decreases and exceeds the IO pressure threshold, becoming the new source-side IO pressure, the number of IO requests input to the storage system is controlled to obtain simulated IO pressure. To avoid the impact of simulated IO pressure on resolving abnormal operations of the distributed storage system, the number of IO requests input to the storage system is kept stable to keep the simulated IO pressure stable.

[0154] In order to avoid a significant impact on the performance of the distributed storage system, the maintained stable simulated IO pressure is lower than the source-side IO pressure corresponding to when the distributed storage system is determined to be operating abnormally in step S310 above, but is higher than the new source-side IO pressure obtained after the source-side IO pressure is reduced, so that the distributed storage system still operates abnormally under the simulated IO pressure.

[0155] In one possible implementation, based on experience, the stable simulated IO pressure can be 70% or 80% of the source-side IO pressure corresponding to when the distributed storage system is determined to be operating abnormally in step S310; or, the processor can first simulate a smaller stable simulated IO pressure, and then dynamically increase the source-side IO pressure until the processor determines that the distributed storage system has an abnormal operation problem under the simulated IO pressure. At this point, the processor returns to execute step S310, obtains new business data, and performs subsequent operations according to the above method steps. Alternatively, the processor can also re-intervention operations on the target suspected sub-healthy component and perform subsequent operations, which is not specifically limited in this application.

[0156] In one possible embodiment, while maintaining stable simulated IO pressure, the processor sends data such as the re-determined suspected sub-healthy components, the confidence levels of the suspected sub-healthy components, the target suspected sub-healthy components, and the intervention operations performed on the target suspected sub-healthy components to the user device, so that the user device reduces sending additional IO requests, thereby maintaining the stability of the simulated IO pressure.

[0157] While simulated IO pressure can cause anomalies in the distributed storage system, the negative impact of this stable simulated IO pressure on system operations is far less than the initial source-side IO pressure that triggers sub-health detection. This ensures storage system security while also revealing truly sub-healthy components. Furthermore, stable simulated IO pressure eliminates the impact of IO pressure fluctuations on resolving distributed storage system anomalies, accurately determining whether anomalies were resolved by intervention on a suspected sub-healthy component, thereby confirming whether the suspected sub-healthy component is actually sub-healthy, improving the accuracy of sub-health detection methods.

[0158] Step S380: updating the training set of the suspected sub-healthy component determination model and updating the suspected sub-healthy component determination model.

[0159] If the source-side IO pressure does not decrease, or decreases to a level less than the IO pressure threshold, the business data and corresponding sub-healthy component labels are added to the training set of the suspected sub-healthy component identification model to update the training set and further update the suspected sub-healthy component identification model. The above process includes two different situations:

[0160] (1) When the intervention operation on the target suspected sub-healthy component is successful, and the source-end IO pressure does not change during the intervention process, or the degree of reduction is less than the IO pressure threshold, the business data and sub-healthy component label corresponding to the target suspected sub-healthy component can be used as regular positive samples to update the training set corresponding to the suspected sub-healthy component determination model. When the training set is updated, the suspected sub-healthy component determination model is further trained based on the new training set.

[0161] (2) If the intervention operation on the target suspected sub-healthy component is successful and the source end I / O pressure increases during the intervention, the business data and sub-healthy component label corresponding to the target suspected sub-healthy component can be used as boundary positive samples to update the training set corresponding to the suspected sub-healthy component determination model. When the training set is updated, the suspected sub-healthy component determination model is further trained based on the new training set. During this training process, the weight of the boundary positive samples is increased to improve the suspected sub-healthy component determination model's ability to identify this situation and improve the model's accuracy.

[0162] The above process can further train the suspected sub-health component determination model, improve the accuracy of the model, and thus improve the accuracy of sub-health detection.

[0163] After the suspected sub-health component determination model is updated, the process may return to step S320 , where the processor re-determines one or more suspected sub-health components based on the business data and the updated suspected sub-health component determination model, and performs subsequent sub-health detection-related operations.

[0164] Step S390: End.

[0165] If the intervention operation on the target suspected sub-healthy component is successful, and the source-side IO pressure does not decrease during the intervention, or the degree of reduction is less than the IO pressure threshold, it can be determined that the cause of the abnormal operation of the distributed storage system is the intervention of the sub-healthy component, and the target suspected sub-healthy component is determined to be a sub-healthy component. Because two or more sub-healthy components rarely appear in a distributed storage system at the same time, the embodiments of this application illustrate the distributed storage system with only one sub-healthy component. Therefore, after determining that the target suspected sub-healthy component is a sub-healthy component, all steps of the sub-health detection method provided in this application can be completed.

[0166] To sum up, the sub-health detection method provided by the present application determines and intervenes in a suspected sub-healthy component, and then determines whether the suspected sub-healthy component is a sub-healthy component by evaluating the effect of the intervention operation on the suspected sub-healthy component and whether the source-end IO pressure changes during the intervention operation. Compared with the current method of judging sub-healthy components based on a single threshold, adding the step of evaluating the effect of the intervention operation can avoid the occurrence of erroneous isolation of normal components, and adding the step of judging whether the source-end IO pressure changes can eliminate the influence of the source-end IO pressure on solving the abnormal operation problems of the distributed storage system. Therefore, the method provided by the present application can greatly improve the accuracy of sub-health detection.

[0167] The embodiment of the present application also provides a sub-health detection device, which is deployed in Figure 1 In the system shown, it can be any one or more of the computing nodes or storage nodes. Figure 1 The sub-health detection equipment is described in detail in the Figure 2 The modules in the software layer shown execute Figure 3 The sub-health detection method shown, wherein each sub-health detection device can run one or more modules in the software layer, and different sub-health detection devices can run the same or different modules in the software layer, which is not specifically limited in this application.

[0168] The present application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions that can be run on a sub-health detection device or stored in any available medium. When the computer program product is run on at least one sub-health detection device, the at least one sub-health detection device is caused to execute Figure 3 The sub-health detection method shown.

[0169] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the sub-health detection device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the sub-health detection device to execute Figure 3 A sub-health detection method is shown.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A sub-health detection method, characterized in that: For detecting the health status of a storage system, the storage system including a plurality of nodes, the method comprising: Determining a first target suspected sub-healthy component from the plurality of components of the node; performing a first intervention operation on the first target suspected sub-health component, wherein the first intervention operation is used to stop the first target suspected sub-health component from working; When it is determined that the storage system is still operating abnormally, it is determined that the first target suspected sub-healthy component is not a sub-healthy component.

2. The method according to claim 1, characterized in that The components are: A node, the hardware in a node, or the software running on a node.

3. The method according to claim 2, characterized in that Determining a first target suspected sub-healthy component from the multiple components of the node specifically includes: determining a first suspected sub-healthy component from multiple components of the node based on the first business data of the node and a suspected sub-healthy component determination model, and determining the first target suspected sub-healthy component from the first suspected sub-healthy components, wherein the suspected sub-healthy component determination model is trained based on historical business data and component labels corresponding to the historical business data, and the component labels include healthy, sub-healthy, and abnormal; or The first suspected sub-healthy component is determined from multiple components of the node according to the first business data of the node and an identification algorithm, and the first target suspected sub-healthy component is selected from the first suspected sub-healthy components.

4. The method according to claim 3, characterized in that After determining that the first target suspected sub-healthy component is not a sub-healthy component, the method further includes: In the case where there are a plurality of first suspected sub-healthy components, determining a second target suspected sub-healthy component from the remaining first suspected sub-healthy components in the plurality of first suspected sub-healthy components; A second intervention operation is performed on the second target suspected sub-healthy component, wherein the second intervention operation is used to stop the second target suspected sub-healthy component from working.

5. The method according to claim 4, characterized in that The method further comprises: When it is determined that the storage system is operating normally, detecting whether a first source-end IO pressure changes during a first intervention operation on the target suspected sub-healthy component, wherein the first source-end IO pressure is determined based on a number of IO requests input to the storage system; When the IO pressure at the first source end does not decrease, or the decrease degree is less than the IO pressure threshold, the first target suspected sub-healthy component is determined to be a sub-healthy component; or When the first source-end IO pressure changes to a second source-end IO pressure, the number of IO requests input to the storage system is controlled to obtain a simulated IO pressure. A second suspected sub-healthy component is determined from multiple components of the node based on second business data of the node under the simulated IO pressure. A value obtained by subtracting the second source-end IO pressure from the first source-end IO pressure is greater than the IO pressure threshold, and the simulated IO pressure is less than the first source-end IO pressure and greater than the second source-end IO pressure.

6. The method according to any one of claims 1 to 4, characterized in that The method further includes: obtaining a simulated IO pressure according to the number of IO requests input to the storage system, and determining the first target suspected sub-healthy component through the simulated IO pressure; The specific process of obtaining the simulated IO pressure includes: The number of IO requests input to the storage system is controlled to gradually increase from a first IO request number to a second IO request number, so that the simulated IO pressure increases from the first simulated IO pressure to the second simulated IO pressure, wherein the storage system operates normally under the first simulated IO pressure, and the storage system has an abnormal operation problem under the second simulated IO pressure.

7. The method according to any one of claims 3 to 5, characterized in that: The confidence of the first target suspected sub-healthy component is higher than the confidence of other suspected sub-healthy components in the first suspected sub-healthy components, wherein the confidence of the first suspected sub-healthy component is calculated by the suspected sub-healthy component determination model or the identification algorithm.

8. A software layer including a sub-health detection service, characterized in that: Applied to a storage system, the storage system includes nodes, and the software layer includes: a component determining module, configured to determine a first target suspected sub-healthy component from a plurality of components of the node; a cluster control module, configured to perform a first intervention operation on the first target suspected sub-health component, wherein the first intervention operation is configured to cause the first target suspected sub-health component to stop operating; The intervention effect evaluation module is used to determine that the first target suspected sub-healthy component is not a sub-healthy component when it is determined that the storage system is still operating abnormally.

9. The software layer according to claim 8, characterized in that The components are: A node, the hardware in a node, or the software running on a node.

10. The software layer according to claim 9, characterized in that The software layer also includes a model training module: The model training module is used to train a suspected sub-healthy component determination model based on historical business data and component labels corresponding to the historical business data, where the component labels include healthy, sub-healthy, and abnormal.

11. The software layer according to claim 10, characterized in that The component determination module is specifically used for: Determine a first suspected sub-healthy component according to the first business data of the node and the suspected sub-healthy component determination model, and determine the first target suspected sub-healthy component from the first suspected sub-healthy component; or The first suspected sub-healthy component is determined according to the first business data of the node and an identification algorithm, and the first target suspected sub-healthy component is determined from the first suspected sub-healthy components.

12. The software layer according to claim 11, characterized in that After the intervention effect evaluation module determines that the first target suspected sub-health component is not a sub-health component, the component determination module is further configured to: In the case where there are multiple first suspected sub-healthy components, a second target suspected sub-healthy component is determined from the remaining first suspected sub-healthy components among the multiple first suspected sub-healthy components, so that the cluster control module performs a second intervention operation on the second target suspected sub-healthy component, wherein the second intervention operation is used to stop the second target suspected sub-healthy component from working.

13. The software layer according to claim 12, characterized in that The intervention effect evaluation module is also used to: When it is determined that the storage system is operating normally, detecting whether a first source-end IO pressure changes during a first intervention operation on the first target suspected sub-healthy component, wherein the first source-end IO pressure is determined based on a number of IO requests input to the storage system; When the IO pressure at the first source end does not decrease, or the degree of decrease is less than an IO pressure threshold, it is determined that the first target suspected sub-healthy component is a sub-healthy component.

14. The software layer according to claim 13, characterized in that The software layer also includes a pressure degradation dial test module; The pressure degradation dialing detection module is configured to control the number of IO requests input to the storage system to obtain a simulated IO pressure when the intervention effect evaluation module detects that the first source-end IO pressure changes to a second source-end IO pressure during a first intervention operation on the first target suspected sub-healthy component. This allows the component determination module to determine a second suspected sub-healthy component from multiple components of the node based on second business data of the node under the simulated IO pressure, wherein a value obtained by subtracting the second source-end IO pressure from the first source-end IO pressure is greater than the IO pressure threshold, and the simulated IO pressure is less than the first source-end IO pressure and greater than the second source-end IO pressure.

15. The software layer according to claim 14, characterized in that The pressure degradation dial detection module is further configured to obtain a simulated IO pressure according to the number of IO requests input to the storage system, so that the component determination module determines the first target suspected sub-healthy component under the simulated IO pressure; The pressure degradation dial detection module is specifically used to: The number of IO requests input to the storage system is controlled to gradually increase from a first IO request number to a second IO request number, so that the simulated IO pressure increases from the first simulated IO pressure to the second simulated IO pressure, wherein under the first simulated IO pressure, the storage system operates normally, and under the second simulated IO pressure, the distributed storage system has an abnormal operation problem.

16. The software layer according to any one of claims 11 to 15, characterized in that: The confidence of the first target suspected sub-healthy component is higher than the confidence of other suspected sub-healthy components in the first suspected sub-healthy components, wherein the confidence of the first suspected sub-healthy component is calculated by the suspected sub-healthy component determination model or the identification algorithm.

Images