Imbalance disk data processing method and system based on generative adversarial network

Abstract

The application belongs to the field of disk fault prediction, and provides an unbalanced disk data processing method and system based on a generative adversarial network, which comprises obtaining disk data and preprocessing; based on the preprocessed disk data, a pre-trained residual conditional table generative adversarial network model is used for data synthesis to obtain synthesized disk data; the application aims at the problem of serious data distribution imbalance in the data in the prior art when predicting disk faults, improves the CTGAN so that the improved model can learn the feature distribution of the real disk data to a deeper level, thereby generating high-quality pseudo data, thereby playing a role in balancing the data set. This method can solve the problem of insufficient disk fault data, and well alleviate the overfitting and feature loss phenomenon caused by oversampling and undersampling technology.

Description

Technical Field

[0001] This invention belongs to the field of disk failure prediction technology, specifically relating to an unbalanced disk data processing method and system based on generative adversarial networks. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In large-scale data centers, hard disks often have the highest failure rate among all hardware components due to their sheer number. According to Microsoft's statistics on its data centers, hard disk failures account for 78% of all hardware failures. Many factors influence disk reliability, and sector-level failures are also very frequent. Factors such as disk temperature, humidity, workload, runtime, and potential sector errors can all lead to disk malfunctions and the loss of recorded user data.

[0004] In recent years, the use of SMART technology for hard drives has been proposed to proactively predict hard drive failures. Specifically, this is a disk self-monitoring, analysis, and reporting technology that monitors and records key health attributes of the hard drive, setting health thresholds for each attribute. If an attribute falls below a threshold, the system assumes a potential hard drive failure and sends a warning to the operating system or user. Upon receiving the warning, the system can migrate or back up the data on the hard drive in advance. Currently, almost all hard drives support SMART technology. However, SMART uses a simple threshold-based algorithm, failing to consider the correlations between various SMART attributes. Furthermore, it relies on real-time attribute data for warnings, neglecting the feature analysis of massive historical data, thus limiting its effectiveness. Nevertheless, SMART records reflect the hard drive's health status and some performance indicators, which remain valuable for predicting hard drive failures. However, the prediction accuracy of SMART technology is very limited, predicting only 3-10% of hard drive failures with a false alarm rate of only 0.1%. Some researchers have studied hard drive SMART information, using statistical and machine learning methods to build hard drive failure prediction models. However, these models can only predict about 60% of hard drive failures while maintaining a low false alarm rate.

[0005] With the continuous development of deep learning, researchers have proposed methods for disk failure prediction based on deep learning. Deep learning methods rely on large amounts of data, analyzing the data to discover underlying patterns and achieve the goal of "learning," thereby improving the system's capabilities. For disk failure data, the number of faulty samples is far less than the amount of normal samples. From a data mining perspective, the minority class is usually more important, so the imbalance between normal and faulty data poses an obstacle to machine learning classification problems. The severe imbalance in the quantity of positive and negative disk samples is a challenging issue for disk failure prediction. Researchers use statistical learning, machine learning, deep learning, and transfer learning methods to train a classification or prediction model, and finally perform failure prediction based on real-time SMART disk data.

[0006] Existing technologies, whether employing random oversampling or undersampling, have certain drawbacks. For example, random undersampling may remove important features from the sample while deleting normal samples, making it difficult for the classifier to learn the decision boundary between positive and negative samples effectively. Random oversampling randomly repeats and replicates minority class samples to achieve dataset balance, but excessive duplication of minority class samples can easily lead to overfitting of the classifier. Using SMOTE (Synthetic Minority Over-Sampling Technique) interpolation can be seen as an improvement on oversampling methods, but since the selection of the main sample in each round is completely random, when there are few minority class samples, the frequency of each original minority class sample being selected as the main sample may differ significantly, thus deviating from the original sample distribution, and the generated samples also lack diversity. A BP-ANN model is enhanced with the AdaBoost algorithm, combining the weighted averages of the weak classifiers to form a strong classifier, the BP-AdaBoost model. While this method expands the amount of fault data, disk failures can occur suddenly, causing normal data recorded by SMART to be marked as fault samples. This artificially adds noise to the fault dataset, thus affecting classification accuracy.

[0007] The above analysis reveals that existing methods for addressing class imbalance in disk-based positive and negative samples only operate within the original dataset, failing to effectively expand the overall dataset size. Furthermore, the range of imbalance rates they can handle is limited. Due to the severe imbalance in the number of positive and negative samples on disk, these methods struggle to address the problem in terms of both quantity and sample diversity. Therefore, handling severe sample imbalance remains a hot topic in the field of machine learning. Summary of the Invention

[0008] To address the aforementioned issues, this invention proposes a method and system for processing imbalanced disk data based on generative adversarial networks. It addresses the problem of insufficient fault data generated by disks, an imbalance between the number of normal and faulty data in the dataset, and consequently low accuracy in classifier fault diagnosis. The invention provides a fault diagnosis method based on an improved CTGAN. The basic framework of the model is WGAN-GP+cGAN+PacGAN, with the addition of a classifier for specific category judgment and a discriminator generative adversarial network based on residual networks.

[0009] According to some embodiments, the first aspect of the present invention provides a method for processing imbalanced disk data based on generative adversarial networks, employing the following technical solution:

[0010] Imbalanced disk data processing methods based on generative adversarial networks include:

[0011] Acquire disk failure data and preprocess it;

[0012] Based on the preprocessed disk failure data, a pre-trained residual condition table generative adversarial network model is used to synthesize the data, resulting in synthesized disk failure data.

[0013] The training process of the residual condition table generative adversarial network model includes:

[0014] The preprocessed disk sample dataset is divided into a training set and a test set.

[0015] Modeling and analysis are performed on the discrete and continuous columns in the training set to obtain the fitted training set;

[0016] The fitted training set is fed into the discriminator and classifier for training, learning the data distribution and updating its own weights.

[0017] The generator is updated by using gradient information returned by the trained discriminator and classifier.

[0018] Based on the fitted training set, pseudo data is generated by using an updated generator to learn the data distribution of the fitted data.

[0019] The generated pseudo-data is fed into the discriminator and classifier respectively for data quality assessment;

[0020] Based on the data quality assessment results, determine the data generation capability of the updated generator;

[0021] The training process of the discriminator, classifier, and generator is iterated until the set number of training iterations is reached, resulting in a trained residual condition table generative adversarial network model.

[0022] Furthermore, the acquisition and preprocessing of disk data includes:

[0023] Retrieve disk data;

[0024] Feature normalization is used to map all the acquired disk data to the interval [-1, 1].

[0025] Remove the dimensions to obtain the preprocessed disk data.

[0026] Furthermore, the discrete and continuous columns in the training set are modeled and analyzed to obtain a fitted training set, specifically:

[0027] The discrete lists in the training set are transformed into one-hot vectors;

[0028] Normalize and fit the continuous columns in the training set to obtain a vector;

[0029] By combining the one-hot vectors and the vectors, we obtain the fitted training set of data.

[0030] Furthermore, the continuous columns in the training set are normalized and fitted to obtain a vector, specifically:

[0031] For each consecutive column, the number of Gaussian distribution patterns is estimated using a variational Gaussian mixture model, and a Gaussian mixture is fitted.

[0032] Calculate the probability of the Gaussian distribution pattern corresponding to each value in each consecutive column;

[0033] A Gaussian distribution pattern is sampled from a given probability density, and continuous values ​​are normalized using the sampled Gaussian distribution pattern to obtain a vector, which is a combination of a one-hot vector of a specified Gaussian distribution pattern and a scalar of a value within a specified Gaussian distribution pattern.

[0034] Furthermore, the discriminator is used to distinguish between real data and pseudo data; the classifier is used to distinguish between positive samples, negative samples, and pseudo data categories;

[0035] The discriminator employs a residual neural network model, and data features are preserved between residual blocks through skip connections.

[0036] Furthermore, the step of feeding the generated pseudo-data into the discriminator and classifier respectively for data quality judgment specifically involves:

[0037] The generated pseudo data is fed into a discriminator to distinguish between pseudo data and real data, and the loss of the discriminator is used to determine the difference between the generated pseudo data and real data.

[0038] The generated pseudo data is fed into a classifier to distinguish data categories, and the cross-entropy loss of the classifier is used to determine the probability distribution distance between the generated pseudo data and the real data.

[0039] The data quality of the generated pseudo data is determined based on the difference between the generated pseudo data and the real data and the probability distribution distance between the generated pseudo data and the real data.

[0040] Furthermore, it also includes:

[0041] A binary classification model for disk failure prediction was trained using synthesized disk data.

[0042] According to some embodiments, a second aspect of the present invention provides an unbalanced disk data processing system based on generative adversarial networks, employing the following technical solution:

[0043] An imbalanced disk data processing system based on generative adversarial networks includes:

[0044] The data acquisition module is configured to acquire disk data and preprocess it.

[0045] The data synthesis module is configured to synthesize data based on preprocessed disk data using a pre-trained residual condition table generative adversarial network model, resulting in synthesized disk data.

[0046] The training process of the residual condition table generative adversarial network model includes:

[0047] The preprocessed disk sample dataset is divided into a training set and a test set.

[0048] Modeling and analysis are performed on the discrete and continuous columns in the training set to obtain the fitted training set;

[0049] The fitted training set is fed into the discriminator and classifier for training, learning the data distribution and updating its own weights.

[0050] The generator is updated by using gradient information returned by the trained discriminator and classifier.

[0051] Based on the fitted training set, pseudo data is generated by using an updated generator to learn the data distribution of the fitted data.

[0052] The generated pseudo-data is fed into the discriminator and classifier respectively for data quality assessment;

[0053] Based on the data quality assessment results, determine the data generation capability of the updated generator;

[0054] The training process of the discriminator, classifier, and generator is iterated until the set number of training iterations is reached, resulting in a trained residual condition table generative adversarial network model.

[0055] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium.

[0056] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the imbalanced disk data processing method based on generative adversarial networks as described in the first aspect above.

[0057] According to some embodiments, a fourth aspect of the present invention provides a computer device.

[0058] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the imbalanced disk data processing method based on generative adversarial networks as described in the first aspect above.

[0059] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0060] This invention addresses the problems of existing technologies and CTGAN generative models in synthesizing disk failure data. The improved CTGAN model can learn the feature distribution of disk data at a deeper level. Firstly, the discriminator uses a residual network, which, compared to fully connected networks, preserves low-dimensional input features and achieves better discrimination. Secondly, adding a classifier to the original structure better utilizes the correlation between category and attribute information in each sample, resulting in higher discrimination accuracy and thus "encouraging" the generator to produce data that more closely approximates the real data distribution. Furthermore, the improved model can generate synthetic data of higher quality than the CTGAN model. This method solves the problem of insufficient disk failure data and can replace oversampling methods such as SMOTE, effectively mitigating the overfitting and feature loss phenomena caused by oversampling and undersampling techniques. Attached Figure Description

[0061] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0062] Figure 1 This is a flowchart of the unbalanced disk fault data processing method based on generative adversarial networks in an embodiment of the present invention;

[0063] Figure 2This is a diagram of the residual network structure in the discriminator described in this embodiment of the invention;

[0064] Figure 3 This is a diagram of the classifier network structure added to the CTGAN generative model as described in this embodiment of the invention.

[0065] Figure 4 This is an architecture diagram of the improved RCTGAN generation model described in this embodiment of the invention;

[0066] Figure 5 This diagram illustrates the evaluation process of the CTGAN and RCTGAN models described in this embodiment of the invention. Detailed Implementation

[0067] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0068] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0069] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0070] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0071] Example 1

[0072] like Figure 1As shown, this embodiment provides a method for processing unbalanced disk data based on generative adversarial networks. This embodiment uses the application of this method to a server as an example for illustration. It is understood that this method can also be applied to terminals, and can also be applied to systems including terminals, servers, and other components, and can be implemented through interaction between the terminal and the server. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communication, middleware services, domain name services, CDN security services, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein. In this embodiment, the method includes the following steps:

[0073] Acquire disk data and preprocess it;

[0074] Based on the preprocessed disk data, a pre-trained residual condition table generative adversarial network model is used to synthesize the data, resulting in synthesized disk data.

[0075] The training process of the residual condition table generative adversarial network model includes:

[0076] The preprocessed disk sample dataset is divided into a training set and a test set.

[0077] Modeling and analysis are performed on the discrete and continuous columns in the training set to obtain the fitted training set;

[0078] The fitted training set is fed into the discriminator and classifier for training, learning the data distribution and updating its own weights.

[0079] The generator is updated by using gradient information returned by the trained discriminator and classifier.

[0080] Based on the fitted training set, pseudo data is generated by using an updated generator to learn the data distribution of the fitted data.

[0081] The generated pseudo-data is fed into the discriminator and classifier respectively for data quality assessment;

[0082] Based on the data quality assessment results, determine the data generation capability of the updated generator;

[0083] The training process of the discriminator, classifier, and generator is iterated until the set number of training iterations is reached, resulting in a trained residual condition table generative adversarial network model.

[0084] Specifically, the acquisition and preprocessing of disk failure data includes:

[0085] Retrieve disk data;

[0086] Feature normalization is used to map all the acquired disk data to the interval [-1, 1].

[0087] Remove the dimensions to obtain the preprocessed disk data.

[0088] In a specific embodiment, the training process of the Residual Conditional Tabular Generative Adversarial Networks (RCTGAN) is as follows:

[0089] Step 1: Obtain a one-year disk dataset from an open-source website. The dataset contains detailed records of each disk's operation process.

[0090] Step 2: Preprocess the dataset. After preprocessing, the dataset is divided into a dataset for the generative model to learn and a final test set. The test set is only used to evaluate the classifier model.

[0091] Step 3: Input the data into the RCTGAN model. The model first models and analyzes the discrete and continuous columns of the data because the data in each continuous column is multimodal and non-Gaussian distributed. Discrete values ​​can be naturally represented as one-hot vectors, but representing continuous values ​​with arbitrary distributions is crucial. RCTGAN uses a pattern-based normalization method to handle columns with complex distributions. In the dataset, each column is processed independently, and each value is represented as a one-hot vector of a specified pattern and a scalar of values ​​within a specified pattern. This involves three steps:

[0092] For each consecutive column C i The number of modes m is estimated using a variational Gaussian mixture model (VGM). i And fit a Gaussian mixture.

[0093] For C i Each value C in i,j Calculate the probability from each pattern.

[0094] Sample a pattern from a given probability density and normalize the values ​​using the sampled pattern.

[0095] Step 4: After fitting, the data is expanded from low dimension to high dimension. High-dimensional data can more accurately represent the distribution of the data.

[0096] The fitted real data is fed into the discriminator and classifier respectively. The discriminator and classifier update their own weights by learning the data distribution of the real data. The discriminator can correctly distinguish between real samples and pseudo samples, and the added classifier can better utilize the correlation between the category and attribute information in each sample, and can also make the model have higher discrimination accuracy.

[0097] The generator is trained alternately with the discriminator and classifier; that is, the generator is trained once the discriminator and classifier are trained. This process is repeated until the set number of iterations is reached, which will generate pseudo-data that is similar to the distribution of real data.

[0098] The generator's input is noise that follows a Gaussian distribution. The noise is processed by the generator to generate pseudo-data, which is then fed into the discriminator and classifier for discrimination. The discriminator and classifier provide gradient information to the generator to update it. The residual network structure in the generator can preserve low-dimensional input features, resulting in better discrimination performance.

[0099] Meanwhile, RCTGAN also considers highly imbalanced classification columns. Addressing the issue of severely imbalanced positive and negative samples on disk, RCTGAN uses a conditional generator for training through sampling. This involves uniformly sampling all classes of discrete data during training to effectively resample and handle imbalanced discrete columns. RCTGAN uses Wasserstein distance with gradient penalty to measure the distribution of real and generated data, helping to mitigate the instability of GANs during adversarial training.

[0100] Step 5: Train binary classifier models such as multilayer perceptron, support vector machine, decision tree, and random forest using the balanced dataset. After training, test the classification accuracy of the classifiers using the test set. In practical applications, select the classifier with the highest classification accuracy as the model for disk failure prediction, that is, use the trained classifier model for the failure prediction function of the data storage center.

[0101] This embodiment generates an adversarial network model using a residual condition table to learn the data distribution of real disk data, thereby generating high-quality disk pseudo data. This serves to balance the original disk data, and the training accuracy of the disk fault prediction binary classification model is improved based on the balanced disk data.

[0102] In this embodiment, the improvement of the RCTGAN model is reflected in:

[0103] Improvement 1: Replace the artificial neural network of the discriminator in the CTGAN generative model with a residual neural network. The residual neural network is easy to optimize. Its internal residual blocks use skip connections, which can achieve the identity mapping of the network. While increasing the number of layers, it retains the features of the original input to avoid network degradation by mitigating feature loss. At the same time, it can also reduce the problem of information loss, preserve the integrity of the original information, and improve the generalization ability of the neural network.

[0104] Improvement 2: The original generative model consists of a generator and a discriminator. The discriminator incorporates the PacGAN algorithm, packaging multiple samples into a single sample and feeding it into the neural network to distinguish between "real" and "fake" samples. While this method effectively alleviates the model collapse problem, it doesn't effectively utilize the category information within each sample. From this perspective, we propose adding a classifier to determine the specific type of the sample. This discriminator-generator structure allows the generator to be fully trained to meet stricter requirements, while simultaneously ensuring that useful information within the samples is fully utilized.

[0105] First, RCTGAN synthesizes fake disk data by learning the distribution of real disk data. Then, the synthesized data is mixed with the real dataset to balance the number of normal and faulty data. Finally, a binary classification model (multilayer perceptron, support vector machine, decision tree, random forest, etc.) is trained using the balanced dataset, and G-means is used to evaluate the model's performance. The classifier with the highest classification accuracy is selected for the fault prediction function of the data storage center.

[0106] Specifically, by comparing it with the existing model CTGAN, the training process of the Residual Conditional Tabular Generative Adversarial Network (RCTGAN) model is analyzed, as follows:

[0107] Step 1: Obtain a one-year disk dataset from an open-source website. The dataset contains detailed records of each disk's operation process.

[0108] Step 2: Preprocess the dataset. After preprocessing, the dataset is divided into a dataset for the generative model to learn and a final test set. The test set is only used to evaluate the classifier model.

[0109] Step 3: As Figure 5As shown, the preprocessed dataset is input into the CTGAN and RCTGAN generative models to generate pseudo-data. The RCTGAN model uses a residual network as its discriminator, while the CTGAN model uses a fully connected network. Compared to fully connected networks, residual networks can preserve low-dimensional input features, achieving better discrimination results, which is not the case with the fully connected network used in CTGAN. The classifier added to RCTGAN can better utilize the correlation between category and attribute information in each sample, and also enable the model to have higher discrimination accuracy, thus "urging" the generator to generate data that more closely resembles the real data distribution.

[0110] Step 4: Mix the pseudo datasets generated by CTGAN and RCTGAN with the real dataset to achieve a balance between the number of positive and negative samples, and use this dataset as the training set for the classifier.

[0111] Step 5: Train binary classification models such as multilayer perceptron, support vector machine, decision tree, and random forest on the balanced training sets of CTGAN and RCTGAN respectively. After training, use the test set to test the classification accuracy of the classifiers and select the binary classifier model with the highest accuracy to apply to the disk failure prediction function of the data storage center.

[0112] As one or more embodiments, step 2: obtaining a one-year disk dataset from an open-source website, the dataset containing detailed records of disk operation information, and also including:

[0113] The obtained public disk SMART dataset is processed. Since a large number of data values ​​in the public dataset are missing or have no effect on the prediction of disk failure, this application selects 12-dimensional data as the dataset. At the same time, in order to avoid errors and provide a fair comparison for each feature, the feature values ​​are normalized, as shown in Formula 1, mapping all feature values ​​to the range [-1,1].

[0114]

[0115] As one or more embodiments, step 3: feeding the dataset generated for model learning into the RCTGAN model to learn the distribution of real data, specifically includes the following steps:

[0116] To enhance the discriminator's ability in a generative model, using a residual network is an excellent choice, such as... Figure 2 As shown, a residual network can connect the input of the nth residual block to any previous output, thus preventing feature loss during training. To fully utilize the category information in the disk data, a classifier for specific category determination is added to the original generative model, such as... Figure 3 The diagram shows the network structure with an added classifier used to distinguish between specific positive and negative sample categories. The generator must simultaneously "deceive" both the classifier and the discriminator to improve the quality of the data generated by the generative model. The final RCTGAN generative model architecture diagram is shown below. Figure 4 As shown.

[0117] Due to the change in the internal structure of the generative model, the loss function must also change during adversarial training. For the generator, it is necessary to minimize not only the loss from the discriminator's 'true' / 'false' judgment but also the loss from the classifier's judgment of the specific category of each sample. The formula for the discriminator loss is shown in Figure 2:

[0118]

[0119] Among them, y_fake i This represents the output of the discriminator for the i-th sample. g It is the average value of the discriminator's output, which is also used for adversarial training of the discriminator and the generator.

[0120] The classifier loss uses the multi-class cross-entropy loss function. Cross-entropy describes the distance between the probability distributions of the true samples and the generated samples. In other words, the smaller the cross-entropy value, the closer the two probability distributions are. As shown in Equation 3:

[0121]

[0122] Wherein, the probability distribution y_real i This is the expected output, with the probability distribution y_fake. i This is the actual output.

[0123] The discriminator loss is the difference between the average value of the discriminator's output of the actual data and the average value of its output of the generated data, as shown in Equation 4:

[0124]

[0125] Here, y_real and y_fake represent the output values ​​of the real and synthetic data passed through the discriminator, respectively. The smaller the difference, the closer the distributions of the synthetic and real data are.

[0126] loss from the discriminator g and cross-entropy loss from the classifier c The combination of these factors constitutes the total loss of the generator in a generative adversarial network. total As shown in Formula 5.

[0127] loss total =loss g +lossc (5)

[0129] As one or more embodiments, step 5 involves training four binary classification models using a balanced training set, examining the classification accuracy of the models using a test set, and selecting the binary classification model with the highest accuracy for fault prediction in the data storage center. Specific steps include:

[0130] To avoid the randomness of experimental results, steps 3 and 4 are repeated, and the average of the multiple experimental results is taken. The G-mean formula is used to evaluate the accuracy of the classifier model, as shown in Formula 6:

[0131]

[0132] Experimental results show that the disk failure data generated by the RCTGAN model can solve the problem of severely imbalanced disk failure data, and the quality of the generated data is higher than that of other current data synthesis models.

[0133] Example 2

[0134] This embodiment provides an imbalanced disk data processing system based on generative adversarial networks, including:

[0135] The data acquisition module is configured to acquire disk data and preprocess it.

[0136] The data synthesis module is configured to synthesize data based on preprocessed disk data using a pre-trained residual condition table generative adversarial network model, resulting in synthesized disk data.

[0137] The training process of the residual condition table generative adversarial network model includes:

[0138] The preprocessed disk sample dataset is divided into a training set and a test set.

[0139] Modeling and analysis are performed on the discrete and continuous columns in the training set to obtain the fitted training set;

[0140] The fitted training set is fed into the discriminator and classifier for training, learning the data distribution and updating its own weights.

[0141] The generator is updated by using gradient information returned by the trained discriminator and classifier.

[0142] Based on the fitted training set, pseudo data is generated by using an updated generator to learn the data distribution of the fitted data.

[0143] The generated pseudo-data is fed into the discriminator and classifier respectively for data quality assessment;

[0144] Based on the data quality assessment results, determine the data generation capability of the updated generator;

[0145] The training process of the discriminator, classifier, and generator is iterated until the set number of training iterations is reached, resulting in a trained residual condition table generative adversarial network model.

[0146] The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1 above. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0147] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0148] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0149] Example 3

[0150] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the imbalanced disk data processing method based on generative adversarial networks as described in Embodiment 1 above.

[0151] Example 4

[0152] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the unbalanced disk data processing method based on generative adversarial networks as described in Embodiment 1 above.

[0153] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0154] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0155] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0156] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0157] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0158] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for processing imbalanced disk data based on generative adversarial networks, characterized in that, include: Acquire disk data and preprocess it; Based on the preprocessed disk data, a pre-trained residual condition table generative adversarial network model is used to synthesize the data, resulting in synthesized disk data. The training process of the residual condition table generative adversarial network model includes: The preprocessed disk sample dataset is divided into a training set and a test set. Modeling and analysis are performed on the discrete and continuous columns in the training set to obtain the fitted training set, specifically: The discrete lists in the training set are transformed into one-hot vectors; Normalize and fit the continuous columns in the training set to obtain a vector; Combine the one-hot vectors and the vectors to obtain the fitted training set of data; The fitted training set is fed into the discriminator and classifier for training, learning the data distribution and updating its own weights. The generator is updated by using gradient information returned by the trained discriminator and classifier. Based on the fitted training set, pseudo data is generated by using an updated generator to learn the data distribution of the fitted data. The generated pseudo-data is fed into the discriminator and classifier respectively for data quality assessment, specifically: The generated pseudo data is fed into a discriminator to distinguish between pseudo data and real data, and the loss of the discriminator is used to determine the difference between the generated pseudo data and real data. The generated pseudo data is fed into a classifier to distinguish data categories, and the cross-entropy loss of the classifier is used to determine the probability distribution distance between the generated pseudo data and the real data. The data quality of the generated pseudo data is determined based on the difference between the generated pseudo data and the real data and the probability distribution distance between the generated pseudo data and the real data. Based on the data quality assessment results, determine the data generation capability of the updated generator; The training process of the discriminator, classifier and generator is iterated until the set number of training iterations is reached, and the trained residual condition table generative adversarial network model is obtained. The discriminator is used to distinguish between real data and pseudo data; the classifier is used to distinguish between positive samples, negative samples, and pseudo data categories. The discriminator employs a residual neural network model, and data features are preserved between residual blocks through skip connections.

2. The imbalanced disk data processing method based on generative adversarial networks as described in claim 1, characterized in that, The process of acquiring and preprocessing disk data includes: Retrieve disk data; Feature normalization is used to map all the acquired disk data to the interval [-1, 1]. Remove the dimensions to obtain the preprocessed disk data.

3. The imbalanced disk data processing method based on generative adversarial networks as described in claim 1, characterized in that, The step of normalizing and fitting the continuous columns in the training set to obtain a vector is as follows: For each consecutive column, the number of Gaussian distribution patterns is estimated using a variational Gaussian mixture model, and a Gaussian mixture is fitted. Calculate the probability of the Gaussian distribution pattern corresponding to each value in each consecutive column; A Gaussian distribution pattern is sampled from a given probability density, and continuous values ​​are normalized using the sampled Gaussian distribution pattern to obtain a vector, which is a combination of a one-hot vector of a specified Gaussian distribution pattern and a scalar of a value within a specified Gaussian distribution pattern.

4. The imbalanced disk data processing method based on generative adversarial networks as described in claim 1, characterized in that, Also includes: A binary classification model for disk failure prediction was trained using synthesized disk data.

5. An imbalanced disk data processing system based on generative adversarial networks, characterized in that, include: The data acquisition module is configured to acquire disk data and preprocess it. The data synthesis module is configured to synthesize data based on preprocessed disk data using a pre-trained residual condition table generative adversarial network model, resulting in synthesized disk data. The training process of the residual condition table generative adversarial network model includes: The preprocessed disk sample dataset is divided into a training set and a test set. Modeling and analysis are performed on the discrete and continuous columns in the training set to obtain the fitted training set, specifically: The discrete lists in the training set are transformed into one-hot vectors; Normalize and fit the continuous columns in the training set to obtain a vector; Combine the one-hot vectors and the vectors to obtain the fitted training set of data; The fitted training set is fed into the discriminator and classifier for training, learning the data distribution and updating its own weights. The generator is updated by using gradient information returned by the trained discriminator and classifier. Based on the fitted training set, pseudo data is generated by using an updated generator to learn the data distribution of the fitted data. The generated pseudo-data is fed into the discriminator and classifier respectively for data quality assessment, specifically: The generated pseudo data is fed into a discriminator to distinguish between pseudo data and real data, and the loss of the discriminator is used to determine the difference between the generated pseudo data and real data. The generated pseudo data is fed into a classifier to distinguish data categories, and the cross-entropy loss of the classifier is used to determine the probability distribution distance between the generated pseudo data and the real data. The data quality of the generated pseudo data is determined based on the difference between the generated pseudo data and the real data and the probability distribution distance between the generated pseudo data and the real data. Based on the data quality assessment results, determine the data generation capability of the updated generator; The training process of the discriminator, classifier and generator is iterated until the set number of training iterations is reached, and the trained residual condition table generative adversarial network model is obtained. The discriminator is used to distinguish between real data and pseudo data; the classifier is used to distinguish between positive samples, negative samples, and pseudo data categories. The discriminator employs a residual neural network model, and data features are preserved between residual blocks through skip connections.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the imbalanced disk data processing method based on generative adversarial networks as described in any one of claims 1-4.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the imbalanced disk data processing method based on generative adversarial networks as described in any one of claims 1-4.

Images