A node fault prediction method and device of a distributed storage cluster

Abstract

The application discloses a node fault prediction method and device of a distributed storage cluster, and relates to the technical field of computer application. Embodiments of the application obtain alarm information of each storage node in a storage system within a preset time, and perform regular multivariate decomposition on the alarm information to obtain a three-dimensional tensor capable of representing a node name, an alarm time and a fault type corresponding to each storage node; then, a preset prediction model is used to perform iterative calculation on the three-dimensional tensor, until a prediction error satisfies a convergence condition, a prediction result matrix is output, and based on the prediction result matrix, a prediction result for representing that a target storage node will occur a target fault type of a fault event in a target time period is obtained. Embodiments of the application can accurately predict future fault events in a large-scale distributed storage cluster under the premise of consuming less computing resources, are more lightweight, and can meet more application scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and particularly relates to a node fault prediction method and device for a distributed storage cluster. BACKGROUND

[0002] At present, the mainstream fault prediction method for large-scale distributed storage clusters is to use machine learning methods such as random forest and LSTM (Long Short-Term Memory) to model historical monitoring data, but in a large-scale distributed storage cluster running for a long time, there are many types of events, a huge number of events, a sparse frequency of events, and a very uneven distribution, and the existing machine learning methods can only predict a few types of events and consume a large amount of computing resources, and therefore are not suitable for actual production environments. Therefore, there is an urgent need for a more lightweight node fault prediction method for a distributed storage cluster. SUMMARY

[0003] The present application proposes a lightweight node fault prediction method suitable for enterprise-level distributed storage clusters and related devices to solve the existing problems, and the specific technical solutions are as follows:

[0004] In the first aspect of the present application, a node fault prediction method for a distributed storage cluster is provided, and the method comprises the following steps:

[0005] Obtaining alarm information of each storage node in a storage system within a preset time; the alarm information comprises a node name, an alarm time and a fault type corresponding to each storage node;

[0006] Regular multivariate decomposition is performed on the alarm information to obtain a three-dimensional tensor, and the three-dimensional tensor is used to represent the node name, the alarm time and the fault type corresponding to each storage node;

[0007] A preset prediction model is used to perform iterative calculation on the three-dimensional tensor until a prediction error meets a convergence condition;

[0008] When the prediction error meets the convergence condition, a prediction result matrix is output, and a prediction result is obtained based on the prediction result matrix, and the prediction result represents that a target storage node will have a target fault type of fault event in a target time period.

[0009] In an embodiment of the present application, obtaining alarm information of each storage node in a storage system within a preset time comprises:

[0010] Obtaining a node identifier of each system node in the storage system, and determining the storage node based on the node identifier;

[0011] Obtain alarm information of each storage node within a preset time.

[0012] In an embodiment of the present application, the alarm information is subjected to regular multivariate decomposition to obtain a three-dimensional tensor, including:

[0013] The node name is mapped to a name matrix composed of various name serial numbers;

[0014] The alarm time is preprocessed to map the alarm time to a time matrix composed of various time serial numbers;

[0015] The fault type is preprocessed to map the fault type to a fault matrix composed of various fault serial numbers;

[0016] The three-dimensional tensor is obtained based on the name matrix, the time matrix, and the fault matrix.

[0017] In an embodiment of the present application, the alarm time is preprocessed to map the alarm time to a time matrix composed of various time serial numbers, including:

[0018] The alarm time is divided into several time periods according to a preset time interval;

[0019] The alarm time in each time period is subjected to deduplication processing to obtain target alarm time;

[0020] The target alarm time is sorted to obtain a time matrix composed of various time serial numbers.

[0021] In an embodiment of the present application, the fault type is preprocessed to map the fault type to a fault matrix composed of various fault serial numbers, including:

[0022] Based on the string corresponding to the fault type, a regular expression of the fault type is obtained;

[0023] The regular expression is subjected to deduplication processing to obtain a target regular expression;

[0024] The target regular expression is sorted to obtain a fault matrix composed of various fault serial numbers.

[0025] In an embodiment of the present application, a preset prediction model is used to iteratively calculate the three-dimensional tensor until the prediction error meets a convergence condition, including:

[0026] Obtain historical observation data of the three-dimensional tensor at a target historical time;

[0027] inputting the historical observation data into the prediction model to obtain prediction data for a time period between the target historical time and a target prediction time, the target prediction time being greater than the target historical time;

[0028] determining a first initial prediction error based on the actual data and the prediction data of the time period and a second initial prediction error based on a preset mapping relationship between fault types and fault levels;

[0029] obtaining a prediction error based on the first initial prediction error and the second initial prediction error;

[0030] increasing the target historical time and repeating the above steps until the prediction error meets a convergence condition; the convergence condition being that the prediction error reaches a minimum value.

[0031] In an embodiment of the present application, when the prediction error meets the convergence condition, a prediction result matrix is outputted, and a prediction result is obtained based on the prediction result matrix, including:

[0032] when the prediction error meets the convergence condition, a target prediction time is determined, and a target time matrix is obtained based on the target prediction time;

[0033] calculating a total sum of outer product operations of the target time matrix, the name matrix and the fault matrix to obtain a prediction result matrix;

[0034] determining a target time serial number, a target name serial number and a target fault serial number in the prediction result matrix whose calculation result is a preset value;

[0035] mapping the target time serial number to a target time period, the target name serial number to a target node name and the target fault serial number to a target fault type;

[0036] obtaining the prediction result based on the target time period, the target node name and the target fault type.

[0037] In a second aspect of the present application, a node fault prediction device for a distributed storage cluster is provided, the device comprising:

[0038] an acquisition module configured to acquire alarm information of each storage node in a storage system within a preset time; the alarm information including a node name, an alarm time and a fault type corresponding to each of the storage nodes;

[0039] a decomposition module configured to perform regular multivariate decomposition on the alarm information to obtain a three-dimensional tensor, the three-dimensional tensor being used to represent the node name, the alarm time and the fault type corresponding to each of the storage nodes;

[0040] an iteration module, configured to perform iterative calculation on the three-dimensional tensor by using a preset prediction model until a prediction error meets a convergence condition;

[0041] an output module, configured to output a prediction result matrix when the prediction error meets the convergence condition, and obtain a prediction result based on the prediction result matrix, the prediction result representing that a target fault event of a target fault type will occur at a target storage node in a target time period.

[0042] In an embodiment of the present application, the acquisition module comprises:

[0043] a node identifier acquisition sub-module, configured to acquire a node identifier of each system node in the storage system, and determine the storage node based on the node identifier.

[0044] an alarm information acquisition sub-module, configured to acquire alarm information of each storage node in a preset time.

[0045] In an embodiment of the present application, the decomposition module comprises:

[0046] a node name mapping sub-module, configured to map the node name into a name matrix composed of various name serial numbers.

[0047] an alarm time mapping sub-module, configured to pre-process the alarm time to map the alarm time into a time matrix composed of various time serial numbers.

[0048] a fault type mapping sub-module, configured to pre-process the fault type to map the fault type into a fault matrix composed of various fault serial numbers.

[0049] a three-dimensional tensor acquisition sub-module, configured to obtain the three-dimensional tensor based on the name matrix, the time matrix and the fault matrix.

[0050] In an embodiment of the present application, the alarm time mapping sub-module comprises:

[0051] a time division unit, configured to divide the alarm time into a plurality of time periods according to a preset time interval.

[0052] a first de-duplication unit, configured to perform de-duplication processing on the alarm time in each time period to obtain target alarm time.

[0053] a first sorting unit, configured to sort the target alarm time to obtain a time matrix composed of various time serial numbers.

[0054] In an embodiment of the present application, the fault type mapping sub-module comprises:

[0055] The regular expression obtaining unit is configured to obtain a regular expression of the fault type based on a string corresponding to the fault type.

[0056] The second deduplication unit is configured to perform deduplication processing on the regular expressions to obtain target regular expressions.

[0057] The second sorting unit is configured to sort the target regular expressions to obtain a fault matrix composed of various fault serial numbers.

[0058] In an embodiment of the present application, the iteration module comprises:

[0059] The obtaining sub-module is configured to obtain historical observation data of the three-dimensional tensor at a target historical time point;

[0060] The prediction sub-module is configured to input the historical observation data into the prediction model to obtain prediction data for a time period between the target historical time point and a target prediction time point; the target prediction time point is greater than the target historical time point.

[0061] The initial error determination sub-module is configured to determine a first initial prediction error based on actual data of the time period and the prediction data, and determine a second initial prediction error based on a preset mapping relationship between fault types and fault levels.

[0062] The prediction error determination sub-module is configured to obtain a prediction error based on the first initial prediction error and the second initial prediction error.

[0063] The iteration sub-module is configured to increase the target historical time point and repeat the above steps until the prediction error meets a convergence condition; the convergence condition is that the prediction error reaches a minimum value.

[0064] In an embodiment of the present application, the output module comprises:

[0065] The target time matrix obtaining sub-module is configured to determine a target prediction time point when the prediction error meets the convergence condition, and obtain a target time matrix based on the target prediction time point.

[0066] The calculation sub-module is configured to calculate a sum of outer product operations of the target time matrix, the name matrix and the fault matrix to obtain a prediction result matrix.

[0067] The target serial number determination sub-module is configured to determine a target time serial number, a target name serial number and a target fault serial number in the prediction result matrix, the calculation result of which is a preset value.

[0068] mapping sub-module, configured to map the target time sequence to a target time period, the target name sequence to a target node name, and the target fault sequence to a target fault type;

[0069] a prediction result obtaining sub-module, configured to obtain the prediction result based on the target time period, the target node name, and the target fault type.

[0070] In a third aspect of the present application, an electronic device is provided, which comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the node fault prediction method of the distributed storage cluster provided in the first aspect of the present application.

[0071] In a fourth aspect of the present application, a computer readable storage medium is provided, which stores a computer program, and a processor executes the computer program to implement the node fault prediction method of the distributed storage cluster provided in the first aspect of the present application.

[0072] The present application has the following beneficial effects:

[0073] In the embodiments of the present application, the alarm information of each storage node in the storage system within a preset time is obtained, and the alarm information is subjected to regular multivariate decomposition to obtain a three-dimensional tensor, which is used to represent the node name, alarm time and fault type corresponding to each storage node; then, the three-dimensional tensor is subjected to iterative calculation by using a preset prediction model, until the prediction error satisfies a convergence condition, a prediction result matrix is output, and the prediction result is obtained based on the prediction result matrix, which represents that a target storage node will have a target fault type fault event in a target time period. In the embodiments of the present application, the correlation between the fault type and the node name and the alarm time is utilized, the alarm information in the storage system is converted into a three-dimensional tensor with three dimensions of node name, alarm time and fault type, the future fault event in the large-scale distributed storage cluster can be accurately predicted under the premise of consuming less computing resources, the reliability and availability of the storage system are improved, the dependence on specific environments such as hardware, software and work load is reduced, it is more lightweight, and more application scenarios can be met. BRIEF DESCRIPTION OF DRAWINGS

[0074] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0075] Figure 1 An electronic device structure schematic diagram of a hardware running environment related to the embodiments of the present application;

[0076] Figure 2is a step flow chart of a node fault prediction method of a distributed storage cluster provided by an embodiment of the present application.

[0077] Figure 3 is a functional module schematic diagram of a node fault prediction device of a distributed storage cluster provided by an embodiment of the present application. DETAILED DESCRIPTION

[0078] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the present application will be further described in detail below in combination with the drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0079] It should be noted that in a long-term running large-scale distributed storage cluster, each storage node will generate a large number of fault events, and the event types cover service failure, disk I / O exception, certificate expiration, configuration change, system resource shortage, etc. Once the system generates an event affecting service quality, the operation and maintenance personnel need to handle it in time, otherwise it may cause system failure.

[0080] The operation and maintenance personnel can actively predict the occurrence of events through relevant technologies, so as to make preparations in advance and prevent the occurrence of events in time. The current mainstream fault prediction method is to use machine learning methods such as random forest and LSTM to model historical monitoring data, but in a long-term running large-scale distributed storage cluster, the types of events are various, the number is huge, the frequency of occurrence is sparse, and the distribution is extremely uneven. The existing machine learning method can only predict a few types of events, and will consume a large amount of computing resources, so it is not suitable for actual production environment.

[0081] The present inventors found through statistics of historical fault events that there is a complex spatio-temporal correlation between the location and type of fault in an enterprise-level storage system, the occurrence frequency of different types of faults is different, and the distribution is uneven and time-varying. About 48% of the faults occur less than 10 times a year, and about 18% of the faults occur only once a year. Through statistics, it is found that the frequency of occurrence of serious level fault events is low, and the average occurrence probability is 0.18%, which is much lower than the average occurrence probability of all fault types 0.77%.

[0082] Therefore, the application discards machine learning which needs to consume a large amount of computing resources, adopts a lightweight distributed storage cluster node fault prediction method, utilizes the correlation between the fault type and the node name and the alarm time, and converts the alarm information in the storage system into a three-dimensional tensor of three dimensions of the node name, the alarm time and the fault type, so that the future fault events in the large-scale distributed storage cluster can be accurately predicted under the premise of consuming less computing resources.

[0083] The scheme of the application will be further described below with reference to the drawings.

[0084] Referring to Figure 1 , Figure 1 The electronic device structure schematic diagram of the hardware running environment related to the embodiment scheme of the application.

[0085] As Figure 1 indicated, the electronic device can include a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection and communication between the components. The user interface 1003 can include a display screen, an input unit such as a keyboard, and an optional user interface 1003 can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 can be a high-speed random access memory (RAM) memory, and can also be a stable non-volatile memory (Non-Volatile Memory, NVM), such as a magnetic disk memory. The memory 1005 can also be an independent storage device from the aforementioned processor 1001.

[0086] Those skilled in the art can understand that Figure 1 the structure shown in the foregoing embodiments does not constitute a limitation on the electronic device, and can include more or fewer components than the drawings, or combine certain components, or different component arrangements.

[0087] As Figure 1 indicated, the memory 1005 as a storage medium can include an operating system, a data storage module, a network communication module, a user interface module, and an electronic program.

[0088] In Figure 1The network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the electronic device of the application can be arranged in the electronic device, and the electronic device calls the node fault prediction device of the distributed storage cluster stored in the memory 1005 through the processor 1001, and executes the node fault prediction method of the distributed storage cluster provided by the application.

[0089] With reference to Figure 2 , based on the foregoing hardware operating environment, the embodiment of the application provides a node fault prediction method of a distributed storage cluster, which can specifically include the following steps:

[0090] S101: acquiring alarm information of each storage node in the storage system within a preset time; the alarm information includes a node name, an alarm time and a fault type corresponding to each storage node.

[0091] It should be noted that the storage system using the distributed storage cluster can be divided into storage nodes and non-storage nodes according to functions. In a specific implementation, in order to avoid the influence of non-storage nodes (such as monitoring nodes) on the fault prediction of the storage nodes, the node identifier of each system node in the storage system is first acquired, and based on the node identifier, all storage nodes are found, and then the alarm information of each storage node within a preset time is acquired, that is, the fault event log of each storage node, which specifically includes the node name, the alarm time, the fault type and the fault level corresponding to each storage node.

[0092] S102: performing regular multivariate decomposition on the alarm information to obtain a three-dimensional tensor, the three-dimensional tensor being used to represent the node name, the alarm time and the fault type corresponding to each storage node.

[0093] In the embodiment, the node name in the storage system is defined as N, the fault type is defined as E, and the alarm time is defined as T. By performing CP decomposition (Canonical Polyadic Decomposition, regular multivariate decomposition) on the alarm information, the values of N, E and T can be obtained, and then a three-dimensional tensor M N×E×T is obtained. met ∈M N×E×T , which represents that the fault event of the fault type e occurs on the node n of the storage system at the time t. It should be noted that the node name N can represent the occurrence position of the alarm information.

[0094] It should be noted that a tensor is equivalent to a data container, and the data contained therein is usually numerical data, so it is a container of numbers. For example, a tensor containing only one number is called a scalar (also called a scalar tensor, zero-dimensional tensor, 0D tensor); an array of numbers is called a vector or a one-dimensional tensor (1D tensor); an array of vectors is called a matrix or a two-dimensional tensor (2D tensor); combining multiple matrices into a new array can obtain a 3D tensor; combining multiple 3D tensors into an array can create a 4D tensor, and so on. Therefore, a tensor can be regarded as a multi-dimensional or multi-directional array, which is a higher-order generalization of a matrix and a vector.

[0095] In the embodiment, by taking the failure event log of the storage system as a streaming input and converting it into a three-dimensional tensor with three dimensions of node name, alarm time and failure type, an enhanced tensor decomposition matrix can be obtained by using context information.

[0096] S103: Iteratively calculate the three-dimensional tensor using the preset prediction model until the prediction error meets the convergence condition.

[0097] In the embodiment, since the three-dimensional tensor stores the alarm information of each storage node within a preset time, at a given historical time t, the alarm information can be divided into two parts of training data and test data. The observed failure event log of all storage nodes at the historical time t can be represented as M t , which is a slice of M at time t, and then the observed tensor M obs can be used to predict future slices, such that M N×E×T represents the observed tensor M obs ∈M N×E×T , and the predicted tensor M pre ∈M N×E×T′ , where T < T', and the predicted time period is T-T'. By calculating the three-dimensional tensor using the preset prediction model, the prediction error between the predicted data and the actual data can be obtained. By continuously increasing the value of time t, iterative calculation is performed until the prediction error reaches a minimum value and converges. At this time, N, E, T' are the results learned, which are used to predict the failure events occurring in the future time slice Δt. The values of N and E are the same as the original, and only the future time T' needs to be recalculated.

[0098] S104: When the prediction error meets the convergence condition, output the prediction result matrix, and obtain the prediction result based on the prediction result matrix. The prediction result represents that the target storage node will have a target failure type of failure event in the target time period.

[0099] In the embodiment, after the prediction error converges to the minimum value, the outer product operation sum of each vector in N, E, T' can be calculated, and the prediction result matrix M^ is obtained, and in the prediction result matrix, the target storage node N, the target fault type E and the target time period T with the result of the preset value 1 are found, and finally the prediction result is obtained: the target fault type E of the target storage node N in the target time period T will occur.

[0100] In the embodiment, by using the correlation between the fault type, the node name and the alarm time, the alarm information in the storage system is converted into a three-dimensional tensor with three dimensions of node name, alarm time and fault type, so that the future fault event in the large-scale distributed storage cluster can be accurately predicted under the premise of consuming less computing resources. On the one hand, the fault can be excluded before it occurs, thereby improving the reliability and availability of the storage system, reducing the dependence on specific environments such as hardware, software and workloads, and being more lightweight, so as to meet more application scenarios. On the other hand, the operation and maintenance personnel have sufficient time to respond, reduce the work pressure and work intensity, and save the operation and maintenance cost.

[0101] In a feasible embodiment, S102 can specifically include the following sub-steps:

[0102] S102-1: Map the node name to a name matrix composed of each name serial number.

[0103] In the embodiment, by summarizing the node names of the storage nodes in the storage system, performing the deduplication operation and sorting, a name matrix M composed of each name serial number is obtained. N .

[0104] S102-2: Preprocess the alarm time to map the alarm time to a time matrix composed of each time serial number.

[0105] In a specific implementation, by summarizing the alarm time, the alarm time can be divided into several time periods according to a preset time interval; the alarm time in each time period is subjected to a deduplication process to obtain a target alarm time; and the target alarm time is sorted to obtain a time matrix M composed of each time serial number. T .

[0106] For example, the alarm time can be divided into several days, and the time of the same fault event is mapped to the start time of the day, that is, the time when the fault event occurs for the first time, so as to realize the deduplication and obtain the target alarm time. Finally, the original alarm time in the alarm information is mapped to the time matrix M after the target alarm time is sorted. T .

[0107] S102-3: Preprocess the fault type to map the fault type into a fault matrix composed of various fault serial numbers.

[0108] In a specific implementation, the text information of the alarm information describing the fault event can be replaced by a corresponding string first, and then a regular expression of the fault type is obtained based on the string corresponding to the fault type; the regular expression is processed to remove duplicates to obtain a target regular expression; alarm information with similar semantics is labeled and merged with the target regular expression, and finally the target regular expression is sorted to obtain a fault matrix M composed of various fault serial numbers. E .

[0109] In this embodiment, a mapping between the fault type and the fault level is also constructed based on the target regular expression to obtain a mapping relationship between the fault type and the fault level.

[0110] S102-4: Obtain a three-dimensional tensor based on the name matrix, the time matrix, and the fault matrix.

[0111] In this embodiment, based on the obtained name matrix M N , the time matrix M T , and the fault matrix M E , the alarm information in the storage system can be converted into a three-dimensional tensor M N ×E×T .

[0112] In one possible implementation, S103 can specifically include the following sub-steps:

[0113] S103-1: Obtain historical observation data of the three-dimensional tensor at a target historical time.

[0114] In this embodiment, a target historical time t is set, and the historical observation data obtained by observing the fault event logs of all storage nodes at the historical time t is obtained, denoted as M t , which is a slice of M at time t, therefore, M N×E×T can be regarded as a time series composed of multiple slices.

[0115] S103-2: Input the historical observation data into a prediction model to obtain prediction data for a time period from the target historical time to a target prediction time; the target prediction time is greater than the target historical time.

[0116] S103-3: Determine a first initial prediction error based on actual data and prediction data of the time period; and determine a second initial prediction error based on a preset mapping relationship between the fault type and the fault level.

[0117] In the embodiment, let S∈M E denotes a fault severity vector, W∈M N×E×T denotes a weight tensor set according to the fault severity, the more serious the fault event, the higher the fault level, and the greater the weight W. It should be noted that the weight related to a specific error type has the same weight value.

[0118] In the embodiment, the first initial prediction error can be calculated according to the following formula:

[0119]

[0120] where A denotes the first initial prediction error between the predicted data and the actual data, and ||.|| denotes the F-norm. F is the F-norm, denotes the sum of the outer product operations of each vector in N, E, and T.

[0121] In the embodiment, the second initial prediction error can be calculated according to the following formula:

[0122]

[0123] where B denotes the second initial prediction error, Y denotes the fault level vector, E denotes the fault type factor matrix, and S denotes the factor vector, which is initially filled with random numbers and then adjusted iteratively until optimal.

[0124] S103-4: Obtain the prediction error based on the first initial prediction error and the second initial prediction error.

[0125] In the embodiment, the prediction model constructed based on formula (1) and formula (2) can obtain the prediction error E 误差 =A+B.

[0126] S103-5: Increase the target historical time and repeat the above steps until the prediction error meets the convergence condition; the convergence condition is that the prediction error reaches a minimum value.

[0127] In the embodiment, by iteratively increasing the value of the target historical time t, the prediction error E 误差 reaches a minimum value, satisfying the convergence condition, and N, E, and T at this time are the results learned.

[0128] In one possible embodiment, S104 can specifically include the following sub-steps:

[0129] S104-1: When the prediction error meets the convergence condition, obtain the target time matrix based on the target prediction time.

[0130] S104-2: Calculate the outer product operation sum of the target time matrix, the name matrix, and the fault matrix to obtain a prediction result matrix.

[0131] S104-3: Determine the target time sequence number, the target name sequence number, and the target fault sequence number in the prediction result matrix.

[0132] S104-4: Map the target time sequence number to the target time period, the target name sequence number to the target node name, and the target fault sequence number to the target fault type.

[0133] S104-5: Obtain a prediction result based on the target time period, the target node name, and the target fault type.

[0134] In the embodiment, to predict the fault events occurring in the future Δt time slice, the values of N and E are the same as originally, but the future time T' needs to be recalculated. In a specific implementation, the alarm time can be divided into slices according to Δt, and T' is equal to the nearest time slice; or the alarm time is divided into slices according to Δt, and the average value of n>1 time slices is taken, which is T', and then the target time matrix that needs to be predicted is obtained.

[0135] In the embodiment, the outer product operation sum of each vector in N, E, and T' is calculated, that is, the prediction result matrix M^. In the prediction result matrix M^, we can find the target time sequence number, the target name sequence number, and the target fault sequence number with a result of 1; and map the target time sequence number to the target time period, the target name sequence number to the target node name, and the target fault sequence number to the target fault type; finally, based on the target time period, the target node name, and the target fault type, a prediction result is obtained: a fault event of the target fault type will occur in the target storage node in the target time period.

[0136] In summary, the embodiments of the present application utilize the correlation between the fault type and the node name and the alarm time, convert the alarm information in the storage system into a three-dimensional tensor of three dimensions of node name, alarm time, and fault type, can accurately predict future fault events in a large-scale distributed storage cluster under the premise of consuming less computing resources, solve the problem that the current industry uses random forest, LSTM, and other mainstream methods for storage system fault prediction need to model long-term historical monitoring data, which not only consumes a large amount of computing resources, but also is difficult to apply to actual production environments with a large number of event types, a large number of quantities, a sparse occurrence frequency, and an unbalanced distribution, and at the same time, a universal enterprise-level storage system fault prediction model is established, reducing the dependence of experimental tuning on specific environments such as hardware, software, and workloads, being more lightweight, and having stronger practicality and portability, which can meet more application scenarios.

[0137] Based on the same inventive concept, referring to Figure 3 Embodiments of the present application also provide a node fault prediction device 200 of a distributed storage cluster, which comprises:

[0138] The acquisition module 201 is configured to acquire alarm information of each storage node in the storage system within a preset time; the alarm information comprises a node name, an alarm time and a fault type corresponding to each storage node respectively;

[0139] The decomposition module 202 is configured to perform canonical polyadic decomposition on the alarm information to obtain a three-dimensional tensor, the three-dimensional tensor being used to represent the node name, the alarm time and the fault type corresponding to each storage node respectively;

[0140] The iteration module 203 is configured to perform iterative calculation on the three-dimensional tensor by using a preset prediction model until a prediction error satisfies a convergence condition;

[0141] The output module 204 is configured to output a prediction result matrix when the prediction error satisfies the convergence condition, and obtain a prediction result based on the prediction result matrix, the prediction result representing that a target storage node will have a target fault type of fault event in a target time period.

[0142] In an embodiment of the present application, the acquisition module 201 comprises:

[0143] The node identification acquisition sub-module is configured to acquire a node identification of each system node in the storage system, and determine the storage node based on the node identification.

[0144] The alarm information acquisition sub-module is configured to acquire the alarm information of each storage node within the preset time.

[0145] In an embodiment of the present application, the decomposition module 202 comprises:

[0146] The node name mapping sub-module is configured to map the node name into a name matrix composed of various name serial numbers.

[0147] The alarm time mapping sub-module is configured to pre-process the alarm time to map the alarm time into a time matrix composed of various time serial numbers.

[0148] The fault type mapping sub-module is configured to pre-process the fault type to map the fault type into a fault matrix composed of various fault serial numbers.

[0149] The three-dimensional tensor acquisition sub-module is configured to obtain the three-dimensional tensor based on the name matrix, the time matrix and the fault matrix.

[0150] In an embodiment of the present application, the alarm time mapping sub-module comprises:

[0151] a time division unit, configured to divide the alarm time into a plurality of time periods according to a preset time interval;

[0152] a first deduplication unit, configured to perform deduplication processing on the alarm time in each time period to obtain target alarm time;

[0153] a first sorting unit, configured to sort the target alarm time to obtain a time matrix composed of time serial numbers.

[0154] In an embodiment of the present application, the fault type mapping submodule includes:

[0155] a regular expression obtaining unit, configured to obtain a regular expression of the fault type based on a string corresponding to the fault type;

[0156] a second deduplication unit, configured to perform deduplication processing on the regular expression to obtain target regular expressions;

[0157] a second sorting unit, configured to sort the target regular expressions to obtain a fault matrix composed of fault serial numbers.

[0158] In an embodiment of the present application, the iteration module 203 includes:

[0159] an obtaining submodule, configured to obtain historical observation data of a three-dimensional tensor at a target historical time;

[0160] a prediction submodule, configured to input the historical observation data into a prediction model to obtain prediction data for a time period between the target historical time and a target prediction time; the target prediction time is greater than the target historical time;

[0161] an initial error determining submodule, configured to determine a first initial prediction error based on actual data and prediction data of the time period; and determine a second initial prediction error based on a preset mapping relationship between fault types and fault levels;

[0162] a prediction error determining submodule, configured to obtain a prediction error based on the first initial prediction error and the second initial prediction error;

[0163] an iteration submodule, configured to increase the target historical time and repeat the above steps until the prediction error meets a convergence condition; the convergence condition is that the prediction error reaches a minimum value.

[0164] In an embodiment of the present application, the output module 204 includes:

[0165] a target time matrix obtaining submodule, configured to determine the target prediction time when the prediction error meets the convergence condition, and obtain a target time matrix based on the target prediction time;

[0166] The computing submodule is configured to calculate an outer product operation sum of the target time matrix, the name matrix, and the fault matrix to obtain a prediction result matrix.

[0167] The target sequence number determination submodule is configured to determine a target time sequence number, a target name sequence number, and a target fault sequence number in the prediction result matrix, where the calculation result is a preset value.

[0168] The mapping submodule is configured to map the target time sequence number to a target time period, map the target name sequence number to a target node name, and map the target fault sequence number to a target fault type.

[0169] The prediction result acquisition submodule is configured to obtain a prediction result based on the target time period, the target node name, and the target fault type.

[0170] It should be noted that the specific implementation of the node fault prediction device 200 of the distributed storage cluster in the embodiments of the present application refers to the specific implementation of the node fault prediction method of the distributed storage cluster in the first aspect of the embodiments of the present application, which will not be repeated here.

[0171] Based on the same inventive concept, the embodiments of the present application also provide an electronic device, which comprises at least one processor and a memory in communication connection with the at least one processor; the memory stores a computer program executable by the at least one processor; the computer program is executed by the at least one processor to enable the at least one processor to execute the node fault prediction method of the distributed storage cluster provided in the first aspect of the present application.

[0172] It should be noted that the specific implementation of the electronic device in the embodiments of the present application refers to the specific implementation of the node fault prediction method of the distributed storage cluster in the first aspect of the embodiments of the present application, which will not be repeated here.

[0173] Based on the same inventive concept, the embodiments of the present application also provide a computer readable storage medium storing a computer program, which is executed by a processor to implement the node fault prediction method of the distributed storage cluster provided in the first aspect of the present application.

[0174] It should be noted that the specific implementation of the computer readable storage medium in the embodiments of the present application refers to the specific implementation of the node fault prediction method of the distributed storage cluster in the first aspect of the embodiments of the present application, which will not be repeated here.

[0175] In some embodiments, the computer readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, a magnetic surface memory, an optical disk, or a CD-ROM, etc.; or can be various devices including one or any combination of the above memories. The computer can be various computing devices including a smart terminal and a server.

[0176] In some embodiments, the executable instructions can take the form of a program, software, software modules, scripts, or code, written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0177] By way of example, the executable instructions can, but need not, correspond to a file in a file system, can be stored in a part of a file that holds other programs or data, e.g., one or more scripts stored in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or code portions.

[0178] By way of example, the executable instructions can be deployed to be executed on one computer, or on multiple computers that are located at one site, or that are distributed across multiple sites and that are interconnected by a communication network.

[0179] It should be noted that, in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0180] The above describes in detail the node fault prediction method, device, electronic equipment and computer readable storage medium of a distributed storage cluster provided by the present application. The principles and implementation manners of the present application are described by specific examples. The above example is only used to help understand the method and core idea of the present application. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range can be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for predicting node failure of a distributed storage cluster, characterized in that, The method comprises: obtaining alarm information of each storage node in a storage system within a preset time; the alarm information comprises a node name, an alarm time and a fault type corresponding to each storage node; regular multivariate decomposition is performed on the alarm information to obtain a three-dimensional tensor, which is used to represent the node name, the alarm time and the fault type corresponding to each storage node, wherein the node name is mapped to a name matrix composed of various name serial numbers; the alarm time is preprocessed to map the alarm time to a time matrix composed of various time serial numbers, wherein the alarm time is divided into several time periods according to a preset time interval; de-duplication processing is performed on the alarm time in each time period to obtain target alarm time; the target alarm time is sorted to obtain a time matrix composed of various time serial numbers; the fault type is preprocessed to map the fault type to a fault matrix composed of various fault serial numbers; the three-dimensional tensor is obtained based on the name matrix, the time matrix and the fault matrix; a preset prediction model is used to iteratively calculate the three-dimensional tensor until a prediction error meets a convergence condition; when the prediction error meets the convergence condition, a prediction result matrix is output, and a prediction result is obtained based on the prediction result matrix, which represents that a target storage node will have a target fault type of fault event in a target time period.

2. The method of claim 1, wherein, Obtaining alarm information of each storage node in a storage system within a preset time comprises: obtaining a node identifier of each system node in the storage system, and determining the storage node based on the node identifier; obtaining alarm information of each storage node within a preset time.

3. The method of claim 1, wherein, The pre-processing of the fault type to map the fault type to a fault matrix composed of various fault serial numbers comprises: obtaining a regular expression of the fault type based on a corresponding string of the fault type; de-duplication processing is performed on the regular expression to obtain a target regular expression; the target regular expression is sorted to obtain a fault matrix composed of various fault serial numbers.

4. The method of claim 1, wherein, Using a preset prediction model to iteratively calculate the three-dimensional tensor until a prediction error meets a convergence condition comprises: obtaining historical observation data of the three-dimensional tensor at a target historical time; inputting the historical observation data into the prediction model to obtain prediction data for a time period between the target historical time and a target prediction time; the target prediction time is greater than the target historical time; determining a first initial prediction error based on actual data of the time period and the prediction data, and determining a second initial prediction error based on a mapping relationship between a preset fault type and a fault level; obtaining a prediction error based on the first initial prediction error and the second initial prediction error; increasing the target historical time and repeating the above steps until the prediction error meets the convergence condition; the convergence condition is that the prediction error reaches a minimum value.

5. The method of claim 4, wherein, output a prediction result matrix when the prediction error meets the convergence condition, and obtain a prediction result based on the prediction result matrix, including: obtaining a target time matrix based on the target prediction time when the prediction error meets the convergence condition; calculating an outer product operation sum of the target time matrix, the name matrix and the fault matrix to obtain a prediction result matrix; determining a target time sequence number, a target name sequence number and a target fault sequence number in the prediction result matrix whose calculation result is a preset value; mapping the target time sequence number to a target time period, the target name sequence number to a target node name, and the target fault sequence number to a target fault type; obtaining the prediction result based on the target time period, the target node name and the target fault type. 6.A node failure prediction apparatus of a distributed storage cluster, characterized by comprising: A node fault prediction method for performing a distributed storage cluster as claimed in claim 1, the device comprising: an acquisition module configured to acquire alarm information of each storage node in a storage system within a preset time; the alarm information including a node name, an alarm time and a fault type corresponding to each of the storage nodes; a decomposition module configured to perform regular multivariate decomposition on the alarm information to obtain a three-dimensional tensor, the three-dimensional tensor being used to represent the node name, the alarm time and the fault type corresponding to each of the storage nodes; an iteration module configured to perform iterative calculation on the three-dimensional tensor by using a preset prediction model until a prediction error meets a convergence condition; an output module configured to output a prediction result matrix when the prediction error meets the convergence condition, and obtain a prediction result based on the prediction result matrix, the prediction result representing that a target storage node will have a target fault type of fault event in a target time period.

7. The apparatus of claim 6, wherein, The acquisition module comprises: a node identification acquisition submodule configured to acquire a node identification of each system node in the storage system, and determine the storage node based on the node identification; an alarm information acquisition submodule configured to acquire alarm information of each of the storage nodes within a preset time.

8. The apparatus of claim 7, wherein, The decomposition module comprises: a node name mapping submodule configured to map the node name to a name matrix composed of various name sequence numbers; an alarm time mapping submodule configured to pre-process the alarm time to map the alarm time to a time matrix composed of various time sequence numbers; a fault type mapping submodule configured to pre-process the fault type to map the fault type to a fault matrix composed of various fault sequence numbers; a three-dimensional tensor acquisition submodule configured to obtain the three-dimensional tensor based on the name matrix, the time matrix and the fault matrix.

Images