A two-layer dynamic weighted disk anomaly detection method based on transfer learning

Abstract

The embodiment of the present invention proposes a double-layer dynamic weighted disk anomaly detection method based on transfer learning, including: collecting disk SMART information and performing attribute feature screening on the disk data set, then performing exponential smoothing processing on it to obtain a stable disk data set, and extracting a target domain disk data set and multiple source domain disk data sets; calculating the initial weights of source domain samples and target domain samples of the disk data set; training the migration model corresponding to each source domain; weighted integration of the model results obtained from each source domain to realize fault detection of the target domain disk; as the target domain model disk continues to run, the disk anomaly detection performance is further improved according to the added new data samples; the technical solution provided by the embodiment of the present invention can effectively improve the fault detection effect of the newly invested disk.

Description

[Technical field]

[0001] The present invention relates to a transfer learning method in the field of machine learning, and in particular to a double-layer dynamic weighted disk anomaly detection method based on transfer learning. [Background technology]

[0002] Nowadays, with the rapid development of information technology and the advent of the era of big data, the stability of the storage system, as one of the important components of computers, is directly related to the security of user data. Among them, the disk is the most used storage tool in the storage system. It may be interrupted due to failure during operation, which will lead to reduced service availability (such as direct shutdown) and even permanent loss of stored data, thereby damaging the user experience. The operation of the disk is directly related to the security of the stored data. Data centers generally have hundreds or thousands of disks, which greatly increases the possibility of system failure. Therefore, data centers need to adopt some mechanisms to detect disk anomalies to avoid irreversible damage or loss of data.

[0003] With the rapid development of machine learning methods, it has produced promising results on various issues. Disks have dozens of SMART attributes. If you want to analyze and train these attributes, you will need to process a large amount of data, and there are certain connections between these attributes. Machine learning methods can learn these data and explore their internal connections. Through training, an algorithm model is obtained to complete the prediction of disk failures. At the same time, a large amount of data can continuously optimize the model, thereby improving the accuracy of predicting disk failures and achieving the purpose of detecting disk failures.

[0004] However, it is worth noting that most of the existing algorithms for disk fault detection rely on sufficient data for training, and some algorithms also require a certain amount of fault data. However, due to different needs of data centers, there will be different requirements for disk storage performance. Generally speaking, there will be multiple different disk models. For different disk models, different disk fault prediction models need to be trained to evaluate and predict them separately. However, disk data requires a long time to collect. In many cases, some new models of disks that have just been put into use do not have enough disk data for the construction of high-quality algorithm models. At this time, it is difficult to accurately predict disk failures. Therefore, for such situations, the idea of ​​transfer learning will be considered to solve such problems. Generally speaking, although the SMART attribute values ​​of disks of different models of the same manufacturer will have certain differences, the attribute features of concern are generally the same. The transfer learning algorithm uses disk data with a small amount of data as the target domain sample, and uses disk data of different models of the same manufacturer as the source domain sample to assist in model training. Most of the previous disk migration algorithms only use a single source domain model disk data for offline migration. The source domain data may have poor correlation with the target domain, resulting in reduced classifier performance. Later, the multi-source domain migration method was also used in the disk field. Through means such as k-nearest neighbors, samples similar to the target domain are found in multiple source domains to fill the target domain. However, this type of method only expands the data scale, and does not further optimize the model for the distribution of target domain samples, which makes it difficult for the classification model to pay attention to the distribution information of target domain samples, resulting in reduced detection performance of target domain disks. Therefore, it is necessary to increase the weight of target domain samples in the model training process while expanding the number of training samples to achieve effective detection of target domain disks. [Summary of the invention]

[0005] In view of this, an embodiment of the present invention proposes a two-layer dynamic weighted disk anomaly detection method based on transfer learning to improve the fault detection accuracy of the algorithm model for disks with insufficient data volume.

[0006] The embodiment of the present invention proposes a two-layer dynamic weighted disk anomaly detection method based on transfer learning, comprising the following steps:

[0007] (1) Collect disk SMART information and perform attribute feature screening on the disk data set, then perform exponential smoothing on it to obtain a stable disk data set, and extract the target domain disk data set and multiple source domain disk data sets, specifically:

[0008] Collect disk SMART attribute data and filter out disk SMART attributes that are not missing and change effectively over time, including: current value of underlying data read error rate, original value of underlying data read error rate, current value of disk read / write throughput performance, original value of disk read / write throughput performance, current value of spindle spin-up time, original value of spindle spin-up time, current value of start / stop count, original value of start / stop count, current value of remapped sector count, original value of remapped sector count, current value of seek error rate, original value of seek error rate, current value of seek performance , the original value of seek performance, the current value of power-on time accumulation, the original value of power-on time accumulation, the current value of spindle spin-up retries, the original value of spindle spin-up retries, the current value of power-on cycle count, the original value of power-on cycle count, the current value of serial port speed-down error count, the original value of serial port speed-down error count, the current value of I / O error detection and correction, the original value of I / O error detection and correction, the current value of uncorrectable errors, the original value of uncorrectable errors, the current value of command timeout, the original value of command timeout, the current value of high-fly write, the original value of high-fly write The original value of input, the current value of airflow temperature, the original value of airflow temperature, the current value of impact error rate, the original value of impact error rate, the current value of power-off return count, the original value of power-off return count, the current value of head load / unload count, the original value of head load / unload count, the current value of temperature, the original value of temperature, the current value of program error block count, the original value of program error block count, the current value of current pending mapping sector count, the original value of current pending mapping sector count, the current value of offline uncorrectable sector count, the original value of offline uncorrectable sector count The collected data are used as a data set, and the data collected under the disk SMART attribute characteristics are generated into a stable disk data set that can be used for training models through the exponential smoothing method. The exponential smoothing formula is defined as follows:

[0009] S t =α·Y t +(1-α)·S t-1

[0010] Where t is time, Y t is the actual value of the data at time t, S t is the smoothed value of the processed data at time t, which is recursively calculated based on the actual value at time t and the smoothed value of the previous t-1 data; the value of t determines Y tThe time span of the information contained. The larger the t is, the larger the time span of the smoothing process is, and the more information is contained. It can be set according to the actual situation. If the value is small, it will lead to a weaker smoothing effect, but it will be more sensitive to new changes in the data. The parameter α is the exponential smoothing coefficient, which is used to control the decay rate of older observations. α∈[0,1]. If the α value is large, a lower weight will be assigned to observations far away from time t, which will have a better detection effect on sequences with shorter mutation times.

[0011] After obtaining a stable disk data set, the disk data with a small number of disks and models to be tested are used as the target domain samples of transfer learning to form the target domain T; the disk data with different models from the same manufacturer as the disk to be tested are used as the source domain samples of transfer learning to form the source domain group D = [D1, D2, ..., D j ], j = (1, 2, ..., M), M represents the number of source domains used in the migration process, and each source domain corresponds to a different disk model;

[0012] (2) Calculate the initial weights of the source domain samples and target domain samples of the disk dataset, specifically:

[0013] According to the number of samples contained in each source domain disk dataset and target domain dataset, the source domain D j And each sample in the target domain T is given an initial weight, and samples in the same domain have the same initial weight; for the initial weight set of samples The calculation of is defined as follows:

[0014]

[0015] where n j Represents the source domain D j The number of samples included, m represents the number of samples included in the target domain T, 1≤k≤n j When is the source domain D j The samples in n j +1≤k≤n j +m is the sample in the target domain T;

[0016] (3) Train the migration model corresponding to each source domain, specifically:

[0017] The source domain D j The samples in the target domain T are combined as the training set, and the initial training sample weights are obtained according to step (2) After that, the XGBoost algorithm is used to train the model. After the model training is completed, the error rate ε of the model predicting the samples in the target domain T is calculated. T :

[0018]

[0019] where h T (x k ) represents the model for sample x k A binary classification prediction with a value of 0 or 1, and the binary classification threshold is 0 means the model predicts a normal sample, and 1 means the model predicts an abnormal sample; c(x k ) represents the sample x k The true label of the normal sample is 0, and the true label of the abnormal sample is 1; when the model predicts the samples in the target domain T completely correctly, the numerator is 0 and the error rate is also 0;

[0020] Calculate the error rate ε of samples in the target domain T T Then, according to the error rate ε T Update the weights for the target domain samples and the source domain samples for the next round of training. The weight update formula for each sample is as follows:

[0021]

[0022] where β T , Represent the weight update coefficients of the target domain and the source domain respectively, N is the maximum number of iterations set, i = (1, 2, 3, ... N), Represents the weight of the kth sample in the i-th iteration training, calculated Will be used for the next iteration of training;

[0023] Then repeat the above process for multiple iterations until N iterations are completed or the error rate of samples in the target domain T reaches ε T When it is 0, the iteration is terminated and the source domain D is obtained. j The corresponding migration model Model j ;

[0024] (4) Weighted integration of model results from each source domain to implement fault detection of the target domain disk, specifically:

[0025] According to steps (2) and (3), we can obtain the migration model corresponding to each source domain. j And the error rate ε of the samples in the target domain T in each layer of iteration loop T , calculate the weight λ corresponding to each source domain in the voting stage j , the calculation formula is as follows:

[0026]

[0027] in Representative Model j The error rate in the i-th iteration;

[0028] Then put the data to be tested into the model j The corresponding prediction result R is obtained j , R j is the probability output of the model. The larger the probability value, the higher the abnormality. j After weighted integrated calculation, the weighted prediction result R is obtained. The calculation formula is as follows:

[0029]

[0030] Finally, the positive and abnormal samples are divided according to the division threshold γ, so as to realize the fault detection of the data to be detected on the target domain disk;

[0031] (5) As the target domain model disk continues to run, the disk anomaly detection performance is further improved based on the added new data samples, specifically:

[0032] When the number of new data samples obtained from the target domain model disk exceeds the threshold P, these data are combined into a new target domain T', and the data previously used for model training are combined into a new source domain D j '=D j ∪T, and retain the sample weight set in the last iteration training As the initial weight of the new source domain sample; then according to steps (2) and (3), the new migration model Model corresponding to all source domains is obtained j 'And the error rate of the model in each layer iteration cycle ε T ', and update the corresponding weight λ of the model according to the newly accumulated error rate j ', the update formula is as follows:

[0033]

[0034] This completes the online dynamic update of the multi-source domain migration model and further improves the disk anomaly detection performance.

[0035] The error rate ε in step (3) of the above method is T The weight update coefficient β must be less than or equal to 0.5. T , This is to ensure that 0 < β T , When the error rate ε T >0.5, the β value of the previous round is used T , Keep iterating.

[0036] The binary classification threshold in step (3) of the above method The selection is based on the probability distribution of normal samples in the target domain samples of the training set. As the threshold for calculating the error rate, μ represents the mean of normal samples in the target domain T, and σ represents the standard deviation of normal samples in the target domain T.

[0037] In step (4) of the above method, the classification threshold γ is between 5% and 10%, that is, the samples ranked in the first γ in the prediction result R will be classified as abnormal. The larger the value of the threshold γ, the more sensitive the model is to abnormalities.

[0038] In step (5) of the above method, the threshold P is set to 20% of the number of samples in the original target domain T; this is to avoid meaningless updates to the model. The threshold P is set to limit the frequency of updates to ensure that the model is more inclined to detect abnormal disk data.

Brief Description of the Drawings

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 It is a flowchart of a double-layer dynamic weighted disk anomaly detection method based on transfer learning proposed in an embodiment of the present invention;

[0041] Figure 2 It is a flowchart of establishing a multi-source domain migration model by a double-layer dynamic weighted disk anomaly detection method based on transfer learning proposed in an embodiment of the present invention.

[0042] Figure 3 It is a flowchart of dynamically updating multi-source domain models and corresponding model weights in a two-layer dynamic weighted disk anomaly detection method based on transfer learning proposed in an embodiment of the present invention. [Specific implementation method]

[0043] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0044] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The embodiment of the present invention provides a dual-layer dynamic weighted disk anomaly detection method based on transfer learning, such as Figure 1As shown, it is a flow chart of a double-layer dynamic weighted disk anomaly detection method based on transfer learning proposed in an embodiment of the present invention, and the method includes the following steps:

[0046] Step 101 , collect disk SMART information and perform attribute feature screening on a disk data set, then perform exponential smoothing processing on it to obtain a stable disk data set, and extract a target domain disk data set and multiple source domain disk data sets.

[0047] Specifically, the disk SMART attribute data is collected and the disk SMART attributes that are not missing and change effectively over time are screened out, including: the current value of the underlying data read error rate, the original value of the underlying data read error rate, the current value of the disk read and write throughput performance, the original value of the disk read and write throughput performance, the current value of the spindle spin-up time, the original value of the spindle spin-up time, the current value of the start and stop count, the original value of the start and stop count, the current value of the remapped sector count, the original value of the remapped sector count, the current value of the seek error rate, the original value of the seek error rate, and the seek performance. Current value, raw value of seek performance, current value of accumulated power-on time, raw value of accumulated power-on time, current value of spindle spin-up retries, raw value of spindle spin-up retries, current value of power-on cycle count, raw value of power-on cycle count, current value of serial port speed-down error count, raw value of serial port speed-down error count, current value of I / O error detection and correction, raw value of I / O error detection and correction, current value of uncorrectable errors, raw value of uncorrectable errors, current value of command timeout, raw value of command timeout, current value of high-fly write, high-fly write Raw value of on-the-fly write, current value of airflow temperature, raw value of airflow temperature, current value of shock error rate, raw value of shock error rate, current value of power-off return count, raw value of power-off return count, current value of head load / unload count, raw value of head load / unload count, current value of temperature, raw value of temperature, current value of program error block count, raw value of program error block count, current value of current pending mapping sector count, raw value of current pending mapping sector count, current value of offline uncorrectable sector count, raw value of offline uncorrectable sector count The collected data are used as a data set, and the data collected under the disk SMART attribute characteristics are generated into a stable disk data set that can be used for training models through the exponential smoothing method. The exponential smoothing formula is defined as follows:

[0048] S t =α·Yt +(1-α)·S t-1

[0049] Where t is time, Y t is the actual value of the data at time t, S t is the smoothed value of the processed data at time t, which is recursively calculated based on the actual value at time t and the smoothed value of the previous t-1 data; the value of t determines Y t The time span of the information contained. The larger the t is, the larger the time span of the smoothing process is, and the more information is contained. It can be set according to the actual situation. If the value is small, it will lead to a weaker smoothing effect, but it will be more sensitive to new changes in the data. The parameter α is the exponential smoothing coefficient, which is used to control the decay rate of older observations. α∈[0,1]. If the α value is large, a lower weight will be assigned to observations far away from time t, which will have a better detection effect on sequences with shorter mutation times.

[0050] After obtaining a stable disk data set, the disk data with a small number of disks and models to be tested are used as the target domain samples of transfer learning to form the target domain T; the disk data with different models from the same manufacturer as the disk to be tested are used as the source domain samples of transfer learning to form the source domain group D = [D1, D2, ..., D j ], j = (1, 2, ..., M), M represents the number of source domains used in the migration process, and each source domain corresponds to a different disk model.

[0051] Step 102: Calculate the initial weights of the source domain samples and the target domain samples of the disk dataset.

[0052] Specifically, according to the number of samples contained in each source domain disk dataset and the target domain dataset, the source domain D j And each sample in the target domain T is given an initial weight, and samples in the same domain have the same initial weight; for the initial weight set of samples The calculation of is defined as follows:

[0053]

[0054] where n j Represents the source domain D j The number of samples included, m represents the number of samples included in the target domain T, 1≤k≤n j When is the source domain D j The samples in n j +1≤k≤n j +m refers to the samples in the target domain T.

[0055] Step 103: training the migration model corresponding to each source domain.

[0056] Specifically, the source domain D j The samples in the target domain T are combined as a training set, and the initial training sample weight set is obtained according to step 102. After that, the XGBoost algorithm is used to train the model. After the model training is completed, the error rate ε of the model predicting the samples in the target domain T is calculated. T :

[0057]

[0058] where h T (x k ) represents the model for sample x k A binary classification prediction with a value of 0 or 1, and the binary classification threshold is 0 means the model predicts a normal sample, and 1 means the model predicts an abnormal sample; c(x k ) represents the sample x k The true label of the normal sample is 0, and the true label of the abnormal sample is 1; when the model predicts the samples in the target domain T completely correctly, the numerator is 0 and the error rate is also 0;

[0059] Error rate ε T The weight update coefficient β must be less than or equal to 0.5. T , This is to ensure that 0 < β T , When the error rate ε T >0.5, the β value of the previous round is used T , Continue to iterate;

[0060] Binary classification threshold The selection is based on the probability distribution of normal samples in the target domain samples of the training set. As the threshold for calculating the error rate, μ represents the mean of normal samples in the target domain T, and σ represents the standard deviation of normal samples in the target domain T;

[0061] Calculate the error rate ε of samples in the target domain T T Then, according to the error rate ε T Update the weights for the target domain samples and the source domain samples for the next round of training. The weight update formula is as follows:

[0062]

[0063] where β T , Represent the weight update coefficients of the target domain and the source domain respectively, N is the maximum number of iterations set, i = (1, 2, 3, ... N), Represents the weight of the kth sample in the i-th iteration training, calculated Will be used for the next iteration of training;

[0064] Then repeat the above process for multiple iterations until N iterations are completed or the error rate of samples in the target domain T reaches ε T When it is 0, the iteration is terminated and the source domain D is obtained. j The corresponding migration model Model j .

[0065] Step 104 , weighted integration of model results obtained in each source domain to implement fault detection of the target domain disk.

[0066] Specifically, according to steps 102 and 103, the migration model Model corresponding to each source domain is obtained j And the error rate ε of the samples in the target domain T in each layer of iteration loop T , calculate the weight λ corresponding to each source domain in the voting stage j , the calculation formula is as follows:

[0067]

[0068] in Representative Model j The error rate in the i-th iteration;

[0069] Then put the data to be tested into the model j The corresponding prediction result R is obtained j , R j is the probability output of the model. The larger the probability value, the higher the abnormality. j After weighted integrated calculation, the weighted prediction result R is obtained. The calculation formula is as follows:

[0070]

[0071] Finally, the positive and abnormal samples are divided according to the division threshold γ, so as to realize the fault detection of the data to be detected on the target domain disk;

[0072] The classification threshold γ is between 5% and 10%, that is, the samples ranked in the first γ in the prediction result R will be classified as abnormal. The larger the value of the threshold γ, the more sensitive the model is to abnormalities.

[0073] Step 105: As the target domain model disk continues to run, the disk anomaly detection performance is further improved according to the added new data samples.

[0074] Specifically, when the number of new data samples obtained from the target domain model disk exceeds the threshold P, these data are combined into a new target domain T', and the data previously used for model training are combined into a new source domain D j '=D j ∪T, and retain the sample weight set in the last iteration training As the initial weight of the new source domain sample; then according to steps 102 and 103, the new migration model Model corresponding to all source domains is obtained j 'And the error rate of the model in each layer iteration cycle ε T ', and update the corresponding weight λ of the model according to the newly accumulated error rate j ', the update formula is as follows:

[0075]

[0076] This completes the online dynamic update of the multi-source domain migration model and further improves the disk anomaly detection performance.

[0077] The threshold P is set to 20% of the number of samples in the original target domain T; this is to avoid meaningless updates to the model. The threshold P is set to limit the frequency of updates to ensure that the model is more inclined to detect abnormal disk data.

[0078] Figure 2 It is a flow chart of establishing a multi-source domain migration model by a two-layer dynamic weighted disk anomaly detection method based on transfer learning proposed in an embodiment of the present invention, wherein the first layer of weighting is sample weighting, and the weights of samples in the next training are updated according to the prediction error rate of the model for target domain samples in the iterative training process; the second layer of weighting is model weighting, and the result weights of each source domain model are updated according to the cumulative size of the prediction error rate of the model for target domain samples in each layer of iteration, and finally the establishment of the multi-source domain migration model is completed.

[0079] Figure 3 1 is a schematic diagram of a process of dynamically updating a multi-source domain model and a corresponding model weight in a two-layer dynamic weighted disk anomaly detection method based on transfer learning proposed in an embodiment of the present invention. As the target domain model disk continues to run, when the number of new data samples of the target domain model disk obtained exceeds a threshold value P, these data are combined into a new target domain T', and the data previously used for model training are combined into a new source domain D j '=D j ∪T, and retain the sample weight set in the last iteration training As the initial weight of the new source domain sample. Then, the same two-layer dynamic weighting method is used to dynamically update each source domain model and the corresponding model weight, and a disk fault detection model that can be continuously updated according to the newly generated data is established.

[0080] Table 1 shows the names, sample numbers, and dimensions of the disk target domain dataset and each source domain dataset when the dual-layer dynamic weighted disk anomaly detection method based on transfer learning is used to solve the disk transfer learning task in an embodiment of the present invention. Table 2 shows the comparative experimental results of the AUC values ​​of the two methods when the dual-layer dynamic weighted disk anomaly detection method based on transfer learning is used to solve the disk transfer learning task in an embodiment of the present invention. The comparison method in the embodiment of the present invention is a multi-source domain transfer learning method (DPSM-TL) based on isolation tree backtracking.

[0081] Table 1

[0082] Disk Dataset Sample size Number of failure samples Dimensionality Target domain: ST31500541AS 200 10 44 Source domain 1: ST4000DM000 3004 4 48 Source domain 2: ST12000NM0008 3012 12 46 Source domain 3: ST12000NM0007 3009 9 44 Source domain 4: ST10000NM0086 3003 3 50

[0083] Table 2

[0084]

[0085] From Table 2, it can be seen that the method proposed in the embodiment of the present invention has obvious improvements over the comparison methods in all comparison experiments, and the proposed migration idea can greatly improve the failure prediction effect for a small number of disk models. On the other hand, in the comparison between single-source domain migration and multi-source domain migration, it can be seen that compared with the migration of a single source domain, the migration of multi-source domains is obviously more effective.

[0086] Table 3 shows the fault detection results of the target domain disk after the dynamic update of each source domain model and the corresponding model weight according to the new data of the target domain model disk obtained by the double-layer dynamic weighted disk anomaly detection method based on transfer learning in an embodiment of the present invention. The threshold P=40 was selected in the experiment. As can be seen from Table 3, with the continuous increase of new data, the fault detection effect of the model for the target domain disk is gradually improved, and tends to be stable when sufficient sample data is obtained. Finally, after the training of the 10th group of data is completed, it is improved by 9.9% compared with the original model, which further proves the effectiveness of the proposed incremental learning idea.

[0087] Table 3

[0088] New sample size 40 80 120 160 200 240 280 320 360 400 AUC value 0.8323 0.8482 0.8584 0.8667 0.8805 0.8852 0.8863 0.8988 0.8971 0.8919

[0089] In summary, the embodiments of the present invention have the following beneficial effects:

[0090] In the technical solution implemented by the present invention, disk SMART information is collected and attribute characteristics of the disk data set are screened, and then exponential smoothing is performed to obtain a stable disk data set, and the target domain disk data set and multiple source domain disk data sets are extracted; the initial weights of the source domain samples and the target domain samples of the disk data set are calculated; the migration model corresponding to each source domain is trained; the model results obtained from each source domain are weighted and integrated to realize fault detection of the target domain disk; as the target domain model disk continues to run, the dynamic update of each source domain model and the corresponding model weight is completed according to the added new data samples. The technical solution provided by the embodiment of the present invention, aimed at the problem of disk fault detection in the case of insufficient number of disk samples, fully considers the characteristics of the disk SMART attributes, establishes a migration model of multiple source domains, and also completes the dynamic update of each model and the corresponding weight, which can effectively improve the fault detection accuracy of the algorithm model for disks with insufficient data volume.

[0091] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A two-layer dynamic weighted disk anomaly detection method based on transfer learning, the method steps comprising: (1) Collect disk SMART information and perform attribute feature screening on the disk data set, then perform exponential smoothing on it to obtain a stable disk data set, and extract the target domain disk data set and multiple source domain disk data sets, specifically: Collect disk SMART attribute data and filter out disk SMART attributes that are not missing and change effectively over time. Use the collected data as a data set. Use the exponential smoothing method to generate the data collected under the disk SMART attribute characteristics into a stable disk data set that can be used for training models. The exponential smoothing formula is defined as follows: S t =α·Y t +(1-α)·S t-1 Where t is time, Y t is the actual value of the data at time t, S t is the smoothed value of the processed data at time t, which is recursively calculated based on the actual value at time t and the smoothed value of the previous t-1 data; the value of t determines Y t The time span of the information contained. The larger the t is, the larger the time span of the smoothing process is, and the more information is contained. It can be set according to the actual situation. If the value is small, it will lead to a weaker smoothing effect, but it will be more sensitive to new changes in the data. The parameter α is the exponential smoothing coefficient, which is used to control the decay rate of older observations. α∈[0,1]. If the α value is large, a lower weight will be assigned to observations far away from time t, which will have a better detection effect on sequences with shorter mutation times. After obtaining a stable disk data set, the disk data with a small number of disks and models to be tested are used as the target domain samples of transfer learning to form the target domain T; the disk data with different models from the same manufacturer as the disk to be tested are used as the source domain samples of transfer learning to form the source domain group D = [D1, D2, ..., D j ], j = (1, 2, ..., M), M represents the number of source domains used in the migration process, and each source domain corresponds to a different disk model; (2) Calculate the initial weights of the source domain samples and target domain samples of the disk dataset, specifically: According to the number of samples contained in each source domain disk dataset and the target domain disk dataset, the source domain D j And each sample in the target domain T is given an initial weight, and samples in the same domain have the same initial weight; for the initial weight set of samples The calculation of is defined as follows: where n j Represents the source domain D j The number of samples included, m represents the number of samples included in the target domain T, 1≤k≤n j When is the source domain D j The samples in n j +1≤k≤n j +m is the sample in the target domain T; (3) Train the migration model corresponding to each source domain, specifically: The source domain D j The samples in the target domain T are combined as the training set, and the initial training sample weight set is obtained according to step (2): After that, the XGBoost algorithm is used to train the model. After the model training is completed, the error rate ε of the model predicting the samples in the target domain T is calculated. T : where h T (x k ) represents the model for sample x k A binary classification prediction with a value of 0 or 1, and the binary classification threshold is 0 means the model predicts a normal sample, and 1 means the model predicts an abnormal sample; c(x k ) represents the sample x k The true label of the normal sample is 0, and the true label of the abnormal sample is 1; when the model predicts the samples in the target domain T completely correctly, the numerator is 0 and the error rate is also 0; Calculate the error rate ε of samples in the target domain T T Then, according to the error rate ε T Update the weights for the target domain samples and the source domain samples for the next round of training. The weight update formula for each sample is as follows: where β T , Represent the weight update coefficients of the target domain and the source domain respectively, N is the maximum number of iterations set, i = (1, 2, 3, ... N), Represents the weight of the kth sample in the i-th iteration training, calculated Will be used for the next iteration of training; Then repeat the above process for multiple iterations until N iterations are completed or the error rate of samples in the target domain T reaches ε T When it is 0, the iteration is terminated and the source domain D is obtained. j The corresponding migration model Model j ; (4) Weighted integration of model results from each source domain to implement fault detection of the target domain disk, specifically: According to steps (2) and (3), we can obtain the migration model corresponding to each source domain. j And the error rate ε of the samples in the target domain T in each layer of iteration loop T , calculate the weight λ corresponding to each source domain in the voting stage j , the calculation formula is as follows: in Representative Model j The error rate in the i-th iteration; Then put the data to be tested into the model j The corresponding prediction result R is obtained j , R j is the probability output of the model. The larger the probability value, the higher the abnormality. j After weighted integrated calculation, the weighted prediction result R is obtained. The calculation formula is as follows: Finally, the positive and abnormal samples are divided according to the division threshold γ, so as to realize the fault detection of the data to be detected on the target domain disk; (5) As the target domain model disk continues to run, the disk anomaly detection performance is further improved based on the added new data samples, specifically: When the number of new data samples obtained from the target domain model disk exceeds the threshold P, these data are combined into a new target domain T', and the data previously used for model training are combined into a new source domain D j '=D j ∪T, and retain the sample weight set in the last iteration training As the initial weight of the new source domain samples; Then, according to steps (2) and (3), we can obtain the new migration model corresponding to each source domain. j 'And the error rate of the model in each layer iteration cycle ε T ', and update the corresponding weight λ of the model according to the newly accumulated error rate j ', the update formula is as follows: This completes the online dynamic update of the multi-source domain migration model and further improves the disk anomaly detection performance.

2. According to the double-layer dynamic weighted disk anomaly detection method based on transfer learning according to claim 1, it is characterized in that: The error rate ε in step (3) is T The weight update coefficient β must be less than or equal to 0.

5. T , This is to ensure that When the error rate ε T >0.5, the β value of the previous round is used T , Keep iterating.

3. According to the double-layer dynamic weighted disk anomaly detection method based on transfer learning in claim 1, it is characterized in that: The binary classification threshold in step (3) The selection is based on the probability distribution of normal samples in the target domain samples of the training set. As the threshold for calculating the error rate, μ represents the mean of normal samples in the target domain T, and σ represents the standard deviation of normal samples in the target domain T.

4. According to the double-layer dynamic weighted disk anomaly detection method based on transfer learning according to claim 1, it is characterized in that: In step (4), the classification threshold γ is between 5% and 10%, that is, the samples ranked in the first γ in the prediction result R will be classified as abnormal. The larger the value of the threshold γ, the more sensitive the model is to abnormalities.

5. According to the double-layer dynamic weighted disk anomaly detection method based on transfer learning in claim 1, it is characterized in that: In step (5), the threshold value P is set to 20% of the number of samples in the original target domain T; this is to avoid meaningless updates of the model. The threshold value P is set to limit the frequency of its updates to ensure that the model is more inclined to detect abnormal disk data.

Images