A distributed storage fault prediction method and system based on machine learning

Through machine learning technology, combining high-dimensional data feature clustering analysis and generation of adversarial networks, key anchor points are selected and related target sample data is generated, which solves the problem of prediction inaccurate caused by insufficient data quality assessment in the existing technology and improves the distribution The accuracy of fault prediction of the formula storage.

CN118502994BActive Publication Date: 2024-11-12BEIJING ZETTAKIT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410961522.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-12
Estimated Expiration
2044-07-18

AI Technical Summary

Technical Problem

The existing distributed storage failure prediction methods lack preliminary evaluation and processing of data quality, resulting in inaccurate prediction results.

Method used

Using a machine learning-based method, by obtaining the historical operating status data of the distributed storage system, clustering analysis of high-dimensional data features, filtering out the initial anchor point set, and inputting it to generate an adversarial network for processing, combining multi-layer perception The machine model is trained, and the learning rate is adjusted adaptively using the Adam optimizer to generate a target sample data set, which is finally used for real-time fault prediction.

Benefits of technology

The prediction accuracy and fault prediction accuracy of the multi-layer perceptron model are improved. Through data preprocessing and feature extraction, a target sample data set more relevant to fault prediction is generated, which enhances the fault prediction capability of the distributed storage system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118502994B_ABST
    Figure CN118502994B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of distributed storage and machine learning, and in particular to a distributed storage fault prediction method and system based on machine learning, wherein the method obtains an initial sample data set; based on cluster analysis of high-dimensional data features, an initial anchor point set corresponding to a high-dimensional space is screened out from the initial sample data set; the initial anchor point set is input into a trained generative adversarial network, and a target sample data set is obtained through processing by the generator; an initial multi-layer perceptron is constructed, and the target sample data set is input into the initial multi-layer perceptron for training, and during the training process, the learning rate is adaptively adjusted based on an Adam optimizer; after the training is completed, a target multi-layer perceptron is output; real-time operation data corresponding to the real-time operation status of a distributed storage system is obtained, and the real-time operation data is input into the target multi-layer perceptron to predict distributed storage faults. The implementation of the method can improve the prediction accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of distributed storage and machine learning, and in particular to a distributed storage fault prediction method and system based on machine learning. Background Art

[0002] In a distributed storage system, multiple independent computers are connected through a network and work together to complete one or more tasks. These computers may be distributed in different geographical locations, but they appear to be a unified whole. The amount of data is so huge that it exceeds the capabilities of traditional data processing software tools. New data processing technologies and architectures are needed to store, manage and analyze data sets. Machine learning algorithms are used to allow computers to automatically "learn" rules from data and make predictions without explicit programming. This aims to improve the accuracy of fault prediction, reasonably arrange regular inspection and maintenance time for equipment, reduce maintenance costs, and improve the operating efficiency of the storage system. Through intelligent fault warning and resource management, the reliability, availability and efficiency of distributed storage systems can be significantly enhanced. It is one of the key technologies for building highly available and highly scalable data centers.

[0003] Current distributed storage fault prediction methods often directly feed raw data into the prediction model, lacking preliminary evaluation and processing of data quality, which leads to inaccurate prediction results. Summary of the invention

[0004] In order to solve the technical problem that the prediction results are inaccurate in the absence of preliminary evaluation and processing of data quality, the purpose of the present invention is to provide a distributed storage fault prediction method and system based on machine learning. The technical solution adopted is as follows:

[0005] In a first aspect, the present application discloses a distributed storage fault prediction method based on machine learning, the method comprising:

[0006] S1. Obtain an initial sample data set corresponding to the historical operating status of the distributed storage system;

[0007] S2. Based on the cluster analysis of high-dimensional data features, select the corresponding high-dimensional space from the initial sample data set.

[0008] Initial anchor point set;

[0009] S3, inputting the initial anchor point set into a trained generative adversarial network, and obtaining a target sample data set through processing by the generator, wherein the generative adversarial network is trained on a mapping relationship between high-dimensional and low-dimensional data;

[0010] S4, constructing an initial multilayer perceptron, inputting the target sample data set into the initial multilayer perceptron for training, and adaptively adjusting the learning rate based on the Adam optimizer during the training process;

[0011] S5. After the training is completed, the target multi-layer perceptron is output;

[0012] S6. Acquire real-time operation data corresponding to the real-time operation status of the distributed storage system, input the real-time operation data into the target multi-layer perceptron, and predict distributed storage failures.

[0013] Furthermore, in step S2, the cluster analysis based on high-dimensional data features selects an initial anchor point set corresponding to the high-dimensional space from the initial sample data set, including:

[0014] S21, preprocessing the initial sample data set to obtain a preprocessed sample data set;

[0015] S22. Constructing a mapping function based on the trained nonlinear regression model;

[0016] S23, mapping the preprocessed sample data set to a high-dimensional space based on the mapping function to obtain a high-dimensional sample data set;

[0017] S24: Perform feature clustering analysis based on the high-dimensional sample data set to obtain an initial anchor point set for reflecting key features of the data set.

[0018] Furthermore, in step S3, the initial anchor point set is input into the trained generative adversarial network, and the target sample data set is obtained through processing by the generator, including:

[0019] S31, reducing the dimension of the high-dimensional sample data set based on the mapping function to obtain reduced-dimensional data;

[0020] S32, extracting random vectors based on the dimension reduction data to obtain a random vector set;

[0021] S33, for each random vector, selecting the target anchor point with the closest distance from the initial anchor point set, and constructing a mapping set between the target anchor point and the corresponding random vector;

[0022] S34, inputting the mapping set into the trained generative adversarial network, and obtaining the target sample data set through processing by the generator.

[0023] Furthermore, in step S34, the processing formula for obtaining sample data by the generator includes:

[0024] ;

[0025] Among them, G is the generator; is any vector in high-dimensional space z A randomly selected vector after dimensionality reduction through a mapping function; A =( a 1, a 2,..., a i ) is the anchor point set; ,in, To calculate vector and anchor point The distance between them, argmin() is used to filter out the smallest distance item from the obtained distance items.

[0026] Furthermore, in step S21, the initial sample data set is preprocessed to obtain a preprocessed sample data set, including:

[0027] S211, performing data standardization processing based on the initial sample data set to obtain a standardized sample data set;

[0028] S212: Perform missing value interpolation processing based on the standardized sample data set to obtain a preprocessed sample data set.

[0029] Furthermore, in step S4, the step of adaptively adjusting the learning rate based on the Adam optimizer includes:

[0030] S41, initialization parameters, the parameters include learning rate η, exponential decay rate β1 of first-order moment estimation, exponential decay rate β2 of second-order moment estimation, small constant , first-order moment estimate m0, second-order moment estimate v0, and time step t;

[0031] S42. At each time step t:

[0032] 1) Calculate the objective function with respect to model parameters according to the chain rule The gradient P t ;

[0033] 2) Based on the gradient p t , introduce the corresponding exponential decay rate, and estimate the first-order moment m t , second-order moment estimation Make updates;

[0034] 3) Exponential decay rate based on first-order moment estimation Construct a first scaling factor and scale the first-order moment estimate m based on the first scaling factor t , to offset the deviation caused by β1;

[0035] 4) Exponential decay rate based on second-order moment estimation Construct a second scaling factor and scale the second-order moment estimate based on the second scaling factor , to offset the deviation caused by β2;

[0036] 5) Based on the revised , , update the model parameters .

[0037] Furthermore, the model parameters include weights W and the bias term b , the objective function is calculated with respect to the model parameters according to the following formula The gradient P t :

[0038] ;

[0039] ;

[0040] in, is the mean square error function used to calculate the difference between the predicted probability distribution and the true label, Indicates layer i To layer j The weight of It is l Layer W , b The result of the calculation is, It is l Layer Node i The output value of It is l -1 layer nodes j The output value of yes right The partial derivative of It is The bias term of the layer.

[0041] In a second aspect, the present application discloses a distributed storage fault prediction system based on machine learning, the system comprising a sample data construction module, a model training module, and a prediction module, wherein:

[0042] The sample data building module is used to obtain an initial sample data set corresponding to the historical operating status of the distributed storage system;

[0043] The sample data construction module is further used to screen out an initial anchor point set corresponding to the high-dimensional space from the initial sample data set based on cluster analysis of high-dimensional data features;

[0044] The sample data construction module is further used to input the initial anchor point set into a trained generative adversarial network, and obtain a target sample data set through processing by the generator, wherein the generative adversarial network is trained on a mapping relationship between high-dimensional and low-dimensional data;

[0045] The model training module is used to construct an initial multi-layer perceptron, input the target sample data set into the initial multi-layer perceptron for training, and during the training process, adaptively adjust the learning rate based on the Adam optimizer;

[0046] The model training module is also used to output the target multi-layer perceptron after the training is completed;

[0047] The prediction module is used to obtain real-time operation data corresponding to the real-time operation status of the distributed storage system, input the real-time operation data into the target multi-layer perceptron, and predict distributed storage failures.

[0048] In a third aspect, the present application discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the distributed storage fault prediction method based on machine learning.

[0049] In a fourth aspect, the present application discloses a distributed storage fault prediction device based on machine learning, comprising a communication interface, a memory, a communication bus and a processor, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0050] The memory is used to store computer programs;

[0051] The processor is used to implement the steps of the distributed storage fault prediction method based on machine learning when executing the program stored in the memory.

[0052] The present invention has the following beneficial effects:

[0053] By combining cluster analysis and generative adversarial networks with high-dimensional data features, key high-dimensional spatial anchor points can be screened out, and a target sample data set that is more relevant to target fault prediction can be generated. This data preprocessing and feature extraction method helps improve the prediction accuracy of the multi-layer perceptron model. In addition, the use of the Adam optimizer to adaptively adjust the learning rate can also more effectively train the multi-layer perceptron and improve the prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 A flow chart of a method for distributed storage fault prediction based on machine learning provided by an embodiment of the present invention;

[0056] Figure 2 A system structure diagram of a distributed storage fault prediction based on machine learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of a distributed storage fault prediction method and system based on machine learning proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0058] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0059] The following is a detailed description of a specific scheme of a distributed storage fault prediction method and system based on machine learning provided by the present invention in conjunction with the accompanying drawings.

[0060] See also Figure 1 , which shows a flow chart of a method for distributed storage fault prediction based on machine learning provided by an embodiment of the present invention, the method comprising:

[0061] Step S1, obtaining an initial sample data set corresponding to the historical operating status of the distributed storage system;

[0062] Step S2, based on cluster analysis of high-dimensional data features, screening out an initial anchor point set corresponding to the high-dimensional space from the initial sample data set;

[0063] Step S3, inputting the initial anchor point set into the trained generative adversarial network, and obtaining the target sample data set through processing by the generator, wherein the generative adversarial network is trained on the mapping relationship between high-dimensional and low-dimensional data;

[0064] S4, constructing an initial multilayer perceptron, inputting the target sample data set into the initial multilayer perceptron for training, and adaptively adjusting the learning rate based on the Adam optimizer during the training process;

[0065] S5. After the training is completed, the target multi-layer perceptron is output;

[0066] S6. Acquire real-time operation data corresponding to the real-time operation status of the distributed storage system, input the real-time operation data into the target multi-layer perceptron, and predict distributed storage failures.

[0067] As can be seen from the above, the distributed storage fault prediction method based on machine learning disclosed in this application can screen out key high-dimensional space anchor points and generate a target sample data set that is more relevant to the target fault prediction by combining cluster analysis of high-dimensional data features and generative adversarial networks. This data preprocessing and feature extraction method helps to improve the prediction accuracy of the multi-layer perceptron model. In addition, the use of the Adam optimizer to adaptively adjust the learning rate can also more effectively train the multi-layer perceptron and improve the prediction accuracy.

[0068] In one embodiment, in step S2, the cluster analysis based on high-dimensional data features is performed from the initial sample

[0069] The initial anchor point set corresponding to the high-dimensional space is selected from the data set, including:

[0070] Step S21, preprocessing the initial sample data set to obtain a preprocessed sample data set.

[0071] Specifically, data preprocessing includes data standardization, missing value filling, etc., and details can be referred to in subsequent embodiments.

[0072] Step S22: constructing a mapping function based on the trained nonlinear regression model.

[0073] Specifically, the functional form of the nonlinear regression model is:

[0074] ;

[0075] in, F is a nonlinear function, x is the input data, θ It should be noted that the present application uses a large amount of data to train the nonlinear regression model in advance, so that the model can accurately map the input data points in the low-dimensional space to the high-dimensional space.

[0076] It should be noted that the parameter θis a variable that needs to be determined, which determines how the function maps the input data to the output space. During the training process, the training data needs to be used to find the optimal θ values ​​so that the model can accurately predict the data.

[0077] Step S23: Mapping the preprocessed sample data set to a high-dimensional space based on the mapping function to obtain a high-dimensional sample data set.

[0078] Specifically, the present application multiplies each data point in the preprocessed sample data set by the mapping function to obtain a corresponding point in the high-dimensional space. Afterwards, all mapped data points are combined to obtain a high-dimensional sample data set.

[0079] Step S24, performing feature clustering analysis based on the high-dimensional sample data set to obtain an initial anchor point set for reflecting key features of the data set.

[0080] Specifically, the present application applies the K-means clustering algorithm to cluster the high-dimensional sample data set, and uses the obtained cluster centers as the anchor point set A =( a 1, a 2,..., a i ).in, , x It is the sample vector divided into different cluster center categories according to the distance from different vectors to each cluster center point in the matrix. i are different categories in the matrix.

[0081] It should be noted that these anchor points not only capture the main features of the dataset, but also act as a bridge between the original data and the high-dimensional representation. They are representative in the high-dimensional space and can effectively summarize the intrinsic properties of the dataset.

[0082] In the above embodiment, by performing feature clustering analysis in high-dimensional space and extracting key features, key factors that may cause failures can be more accurately identified. This helps the model to more accurately judge the operating status of the distributed storage system during real-time prediction, detect potential failures in advance, and issue warnings.

[0083] In one embodiment, in step S3, the initial anchor point set is input into the trained generative adversarial network.

[0084] In the network, the target sample data set is obtained after being processed by the generator, including:

[0085] Step S31, reducing the dimension of the high-dimensional sample data set based on the mapping function to obtain reduced-dimensional data.

[0086] Specifically, in the present application, each sample in the high-dimensional sample data set is multiplied by the mapping function to obtain a corresponding low-dimensional representation, which is the dimensionality reduction data.

[0087] Step S32: extracting random vectors based on the dimension reduction data to obtain a random vector set.

[0088] Step S33: for each random vector, select the target anchor point with the closest distance from the initial anchor point set, and construct a mapping set between the target anchor point and the corresponding random vector.

[0089] Specifically, the present application determines which anchor point in the initial anchor point set is closest to the random vector based on preset distance metrics, including Euclidean distance, cosine similarity, etc. In order to subsequently be able to reduce the dimensionality of the anchor points based on a generative adversarial network, the present application also constructs a corresponding mapping set so that the network can reduce the dimensionality of the anchor points to a low-dimensional space based on the pre-learned high- and low-dimensional data mapping relationship.

[0090] This process is actually to find the anchor points that are most similar or representative to the random vector in the data space. These target anchor points usually represent the key features or patterns in the data set and are of great significance for subsequent fault prediction.

[0091] Step S34, input the mapping set into the trained generative adversarial network, and obtain the target sample data set through processing by the generator.

[0092] In the above embodiment, the target sample data set generated through a series of steps includes more diverse and realistic data points. When these data points are used to train the multi-layer perceptron prediction model, the model can learn more about the laws and patterns of fault occurrence, thereby improving the accuracy and reliability of fault prediction.

[0093] In one embodiment, in step S34, the processing formula for obtaining sample data by the generator includes:

[0094] ;

[0095] Among them, G is the generator; is any vector in high-dimensional space z A randomly selected vector after dimensionality reduction through a mapping function; A =( a 1, a 2,..., a i ) is the anchor point set; ,in, To calculate vector and anchor point The distance between them, argmin() is used to filter out the smallest distance item from the obtained distance items.

[0096] In one embodiment, in step S21, preprocessing the initial sample data set to obtain a preprocessed sample data set includes:

[0097] Step S211, performing data standardization processing based on the initial sample data set to obtain a standardized sample data set.

[0098] It should be noted that, considering the multiple attributes of the data, this application will first perform data normalization on the initial sample data set after obtaining it. The purpose is to ensure that all features or attributes of the data are on the same scale and avoid certain attributes from having a disproportionate impact on the model due to a large numerical range.

[0099] Specifically, the data standardization processing formula includes:

[0100] ;

[0101] in, is the normalized value, are the values ​​in the input dataset, , They are the minimum and maximum values ​​corresponding to the characteristic attributes of the input data.

[0102] Step S212: performing missing value interpolation processing based on the standardized sample data set to obtain a preprocessed sample data set.

[0103] It should be noted that the present application adopts the mode interpolation method to perform missing value interpolation processing, wherein the mode refers to the element with the most occurrences in the data set.

[0104] Specifically, this application uses the calculated mode to replace the missing values ​​in the data set. For each time step t, the calculation formula of the mode includes:

[0105] ;

[0106] in, refers to the calculated mode, M t Refers to data x The set of non-missing values ​​for each feature in .

[0107] In the above embodiment, by data standardization, the original data can be converted to the same scale, eliminating the influence of different dimensions and units between different features. This helps the model to better learn and understand the inherent laws in the data, thereby improving the accuracy of the prediction. In addition, by missing value interpolation, the problem of missing values ​​in the data set can be solved. By interpolating missing values, the integrity and consistency of the data can be maintained, avoiding deviations or errors caused by missing data in subsequent analysis and modeling.

[0108] In one embodiment, in step S4, the step of adaptively adjusting the learning rate based on the Adam optimizer includes:

[0109] Step S41, initializing parameters, the parameters include learning rate η, exponential decay rate β1 of first-order moment estimation, exponential decay rate β2 of second-order moment estimation, a small constant , first-order moment estimate m0, second-order moment estimate v0, and time step t.

[0110] Specifically, the learning rate η is set to 0.001, β1 is set to 0.9, β2 is set to 0.999, and the small constant To prevent division by zero errors, it is set to 10^(-8). The initial first-order moment estimate m0 and second-order moment estimate v0 are both set to 0.

[0111] Step S42, in each time step t:

[0112] 1) Calculate the objective function with respect to model parameters according to the chain rule The gradient P t .

[0113] 2) Based on the gradient p t , introduce the corresponding exponential decay rate, and estimate the first-order moment m t , second-order moment estimation to update.

[0114] The specific update formulas include:

[0115] ;

[0116] ;

[0117] Among them, m t-1 is the first-order moment estimate generated at the previous time step t-1, is the second-order moment estimate produced at the previous time step t-1.

[0118] 3) Exponential decay rate based on first-order moment estimation Construct a first scaling factor and scale the first-order moment estimate m based on the first scaling factort , to offset the deviation caused by β1.

[0119] Specifically, the deviation correction formula includes:

[0120] .

[0121] 4) Exponential decay rate based on second-order moment estimation Construct a second scaling factor and scale the second-order moment estimate based on the second scaling factor , to offset the deviation caused by β2.

[0122] Specifically, the correction formula includes:

[0123] .

[0124] 5) Based on the revised , , update the model parameters .

[0125] Specifically, the model parameter update formula includes:

[0126] ;

[0127] in, is the updated parameter value.

[0128] In one embodiment, the model parameters include weights W and the bias term b , the objective function is calculated with respect to the model parameters according to the following formula The gradient P t :

[0129] ;

[0130] ;

[0131] in, is the mean square error function used to calculate the difference between the predicted probability distribution and the true label, Indicates layer i To layer j The weight of It is l Layer W , b The result of the calculation is, It is l Layer Node i The output value of It is l -1 layer nodesj The output value of yes right The partial derivative of It is The bias term of the layer.

[0132] Specifically, after obtaining the corresponding partial derivative, the back propagation training formula used in this application is as follows:

[0133] ;

[0134] ;

[0135] in, is the learning rate.

[0136] After multiple forward-backward propagation trainings, the weights and bias parameters are updated layer by layer in reverse order. The ultimate goal is to find a local optimal W, b such that Minimum, that is, minimize the error between the predicted output and the actual output.

[0137] It should be noted that The mean square error function is used, and its specific function form is as follows:

[0138] ;

[0139] in, n is the number of sample values, k For the prediction output The number of For the t Sample No. t The predicted output of the output, is the actual output. For input value The function formula after forward propagation is: f ( x ) represents the ReLU activation function, g ( x ) represents the sigmoid activation function, , , They are the weight from input layer to hidden layer, the weight from hidden layer to hidden layer state, and the weight from hidden layer to output layer. , Represent the activation bias items of the hidden layer and the output layer respectively.

[0140] Please refer to Figure 2The present application discloses a distributed storage fault prediction system based on machine learning, the system system includes a sample data construction module, a model training module, and a prediction module, wherein:

[0141] The sample data building module is used to obtain an initial sample data set corresponding to the historical operating status of the distributed storage system;

[0142] The sample data construction module is further used to screen out an initial anchor point set corresponding to the high-dimensional space from the initial sample data set based on cluster analysis of high-dimensional data features;

[0143] The sample data construction module is further used to input the initial anchor point set into a trained generative adversarial network, and obtain a target sample data set through processing by the generator, wherein the generative adversarial network is trained on a mapping relationship between high-dimensional and low-dimensional data;

[0144] The model training module is used to construct an initial multi-layer perceptron, input the target sample data set into the initial multi-layer perceptron for training, and during the training process, adaptively adjust the learning rate based on the Adam optimizer;

[0145] The model training module is also used to output the target multi-layer perceptron after the training is completed;

[0146] The prediction module is used to obtain real-time operation data corresponding to the real-time operation status of the distributed storage system, input the real-time operation data into the target multi-layer perceptron, and predict distributed storage failures.

[0147] In one embodiment, the above modules are also used to implement the method as described in any one of the above embodiments.

[0148] The present application discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the distributed storage fault prediction method based on machine learning.

[0149] A distributed storage fault prediction device based on machine learning disclosed in the present application includes a communication interface, a memory, a communication bus and a processor, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0150] The memory is used to store computer programs;

[0151] The processor is used to implement the steps of the distributed storage fault prediction method based on machine learning when executing the program stored in the memory.

[0152] As can be seen from the above, the distributed storage fault prediction method and system based on machine learning disclosed in this application can screen out key high-dimensional space anchor points and generate a target sample data set that is more relevant to the target fault prediction by combining cluster analysis and generative adversarial networks of high-dimensional data features. This data preprocessing and feature extraction method helps to improve the prediction accuracy of the multi-layer perceptron model. In addition, the use of the Adam optimizer to adaptively adjust the learning rate can also more effectively train the multi-layer perceptron and improve the prediction accuracy.

[0153] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0154] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A distributed storage fault prediction method based on machine learning, characterized in that: The method comprises: S1. Obtain an initial sample data set corresponding to the historical operating status of the distributed storage system; S2. Based on cluster analysis of high-dimensional data features, an initial anchor point set corresponding to the high-dimensional space is selected from the initial sample data set; S3, inputting the initial anchor point set into a trained generative adversarial network, and obtaining a target sample data set through processing by the generator, wherein the generative adversarial network is trained on a mapping relationship between high-dimensional and low-dimensional data; S4, constructing an initial multilayer perceptron, inputting the target sample data set into the initial multilayer perceptron for training, and adaptively adjusting the learning rate based on the Adam optimizer during the training process; S5. After the training is completed, the target multi-layer perceptron is output; S6, obtaining real-time operation data corresponding to the real-time operation status of the distributed storage system, inputting the real-time operation data into the target multi-layer perceptron, and predicting distributed storage failures; In step S4, the step of adaptively adjusting the learning rate based on the Adam optimizer includes: S41, initializing parameters, the parameters including a learning rate η, an exponential decay rate β1 of a first-order moment estimate, an exponential decay rate β2 of a second-order moment estimate, a small constant ε, a first-order moment estimate m0, a second-order moment estimate v0, and a time step t; S42. At each time step t: 1) Calculate the mean square error function with respect to the model parameter θ according to the chain rule t The gradient P t ; 2) Based on gradient P t , introduce the corresponding exponential decay rate, and estimate the first-order moment m t , second-order moment estimate v t Make updates; 3) Exponential decay rate based on first-order moment estimation Construct a first scaling factor and scale the first-order moment estimate m based on the first scaling factor t , to offset the deviation caused by β1; 4) Exponential decay rate based on second-order moment estimation Construct a second scaling factor based on which the second-order moment estimate v is scaled t , to offset the deviation caused by β2; 5) Based on the revised Update model parameters θ t ; The model parameters include weight W and bias term b. The mean square error function is calculated with respect to the model parameter θ according to the following formula: t The gradient P t : Where P(W, b) is the mean square error function used to calculate the difference between the predicted probability distribution and the true label, represents the weights from layer i to layer j, is the result of the calculation of W and b at the lth layer, is the output value of node i in layer l, is the output value of node j in the l-1th layer, yes right The partial derivative of is the bias term of the lth layer.

2. The distributed storage fault prediction method based on machine learning according to claim 1 is characterized in that: In step S2, the cluster analysis based on high-dimensional data features selects an initial anchor point set corresponding to the high-dimensional space from the initial sample data set, including: S21, preprocessing the initial sample data set to obtain a preprocessed sample data set; S22. Constructing a mapping function based on the trained nonlinear regression model; S23, mapping the preprocessed sample data set to a high-dimensional space based on the mapping function to obtain a high-dimensional sample data set; S24: Perform feature clustering analysis based on the high-dimensional sample data set to obtain an initial anchor point set for reflecting key features of the data set.

3. The distributed storage fault prediction method based on machine learning according to claim 2 is characterized in that: In step S3, the initial anchor point set is input into the trained generative adversarial network, and the target sample data set is obtained through processing by the generator, including: S31, reducing the dimension of the high-dimensional sample data set based on the mapping function to obtain reduced-dimensional data; S32, extracting random vectors based on the dimension reduction data to obtain a random vector set; S33, for each random vector, selecting the target anchor point with the closest distance from the initial anchor point set, and constructing a mapping set between the target anchor point and the corresponding random vector; S34, inputting the mapping set into the trained generative adversarial network, and obtaining the target sample data set through processing by the generator.

4. The distributed storage fault prediction method based on machine learning according to claim 3 is characterized in that: In step S34, the processing formula for obtaining sample data by the generator includes: Among them, G is the generator; is a vector randomly selected from any vector z in high-dimensional space after dimensionality reduction through a mapping function; A = (a1, a2, ..., a i ) is the anchor point set; in, To calculate vector and anchor point a i The distance between them, argmin() is used to filter out the smallest distance item from the obtained distance items.

5. The distributed storage fault prediction method based on machine learning according to claim 2 is characterized in that: In step S21, the initial sample data set is preprocessed to obtain a preprocessed sample data set, including: S211, performing data standardization processing based on the initial sample data set to obtain a standardized sample data set; S212: Perform missing value interpolation processing based on the standardized sample data set to obtain a preprocessed sample data set.

6. A distributed storage fault prediction system based on machine learning, characterized in that: The system includes a sample data construction module, a model training module, and a prediction module, wherein: The sample data building module is used to obtain an initial sample data set corresponding to the historical operating status of the distributed storage system; The sample data construction module is further used to screen out an initial anchor point set corresponding to the high-dimensional space from the initial sample data set based on cluster analysis of high-dimensional data features; The sample data construction module is further used to input the initial anchor point set into a trained generative adversarial network, and obtain a target sample data set through processing by the generator, wherein the generative adversarial network is trained on a mapping relationship between high-dimensional and low-dimensional data; The model training module is used to construct an initial multi-layer perceptron, input the target sample data set into the initial multi-layer perceptron for training, and during the training process, adaptively adjust the learning rate based on the Adam optimizer; The model training module is also used to output the target multi-layer perceptron after the training is completed; The prediction module is used to obtain real-time operation data corresponding to the real-time operation status of the distributed storage system, input the real-time operation data into the target multi-layer perceptron, and predict the distributed storage failure; The specific implementation of the model training module adaptively adjusting the learning rate based on the Adam optimizer is as follows: Initialize parameters, including a learning rate η, an exponential decay rate β1 of a first-order moment estimate, an exponential decay rate β2 of a second-order moment estimate, a small constant ε, a first-order moment estimate m0, a second-order moment estimate v0, and a time step t; At each time step t: 1) Calculate the mean square error function with respect to the model parameter θ according to the chain rule t The gradient P t ; 2) Based on gradient P t , introduce the corresponding exponential decay rate, and estimate the first-order moment m t , second-order moment estimate v t Make updates; 3) Exponential decay rate based on first-order moment estimation Construct a first scaling factor and scale the first-order moment estimate m based on the first scaling factor t , to offset the deviation caused by β1; 4) Exponential decay rate based on second-order moment estimation Construct a second scaling factor based on which the second-order moment estimate v is scaled t , to offset the deviation caused by β2; 5) Based on the revised Update model parameters θ t ; The model parameters include weight W and bias term b. The mean square error function is calculated with respect to the model parameter θ according to the following formula: t The gradient P t : Where P(W, b) is the mean square error function used to calculate the difference between the predicted probability distribution and the true label, represents the weights from layer i to layer j, is the result of the calculation of W and b at the lth layer, is the output value of node i in layer l, is the output value of node j in the l-1th layer, yes right The partial derivative of is the bias term of the lth layer.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the distributed storage fault prediction method based on machine learning described in any one of claims 1 to 5 is implemented.

8. A distributed storage fault prediction device based on machine learning, characterized in that: It includes a communication interface, a memory, a communication bus and a processor, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the steps of the distributed storage fault prediction method based on machine learning as described in any one of claims 1-5 when executing the program stored in the memory.