A big data-based data set construction method and system
By generating synthetic frequency band sequences through wavelet transform and conditional generative adversarial networks, the problem of insufficient model training caused by the scarcity of fault samples is solved, thereby improving the accuracy and recall of fault identification and achieving performance improvement of the fault prediction model.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- JIAJIE TECH CO LTD
- Filing Date
- 2026-04-09
- Publication Date
- 2026-07-24
AI Technical Summary
In monitoring scenarios of server clusters, virtual machines, or container cloud platforms, the scarcity of fault samples leads to insufficient training of deep learning models and poor prediction results. Existing data augmentation methods ignore the physical distribution of time-series monitoring data in the frequency domain and the energy synergy between different frequency bands. The generated fault samples lack real features, resulting in weak model generalization ability.
The time-series signal is decomposed into multiple frequency bands by wavelet transform, time-frequency domain features are extracted, frequency band sensitivity and correlation are calculated, a conditional generative adversarial network is constructed, a loss function is constructed using frequency band comprehensive weights, synthetic frequency band sequences are generated and synthetic samples are reconstructed to enhance the dataset.
It significantly improves the accuracy, precision and recall of fault identification, solves the problem of insufficient model training caused by the scarcity of fault samples, and the generated synthetic samples retain the key time-frequency domain features and energy coordination laws of real faults.
Smart Images
Figure CN121997148B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing, and in particular to a method and system for constructing datasets based on big data. Background Technology
[0002] In monitoring scenarios involving server clusters, virtual machines, or container cloud platforms, the system's uptime is far greater than the time of failure, resulting in a scarcity of fault samples. This leads to insufficient training and poor prediction performance of fault prediction models based on deep learning, such as Long Short-Term Memory (LSTM) networks. Existing technologies often use data augmentation methods (such as SMOTE) to increase the number of fault samples. However, these methods typically involve linear interpolation or random noise addition in the time domain or feature space, neglecting the physical distribution of time-series monitoring data in the frequency domain and the energy correlation between different frequency bands. This results in an increased number of fault samples, but a lack of realistic physical characteristics of the faults, failing to reflect actual fault propagation patterns. Consequently, the trained LSTM models exhibit weak generalization ability and degraded prediction performance. Summary of the Invention
[0003] To address the aforementioned technical issues, this application provides a method and system for constructing datasets based on big data.
[0004] Firstly, this application provides a method for constructing datasets based on big data, employing the following technical solution: A method for constructing a dataset based on big data includes the following steps: acquiring historical monitoring data to construct an initial dataset containing normal samples and fault samples; decomposing the time-series sequences in the initial dataset to obtain multiple frequency band sequences and extracting the time-frequency domain features of each frequency band sequence; calculating the frequency band sensitivity of fault samples and normal samples in each frequency band based on the time-frequency domain features; constructing an association matrix and calculating the comprehensive frequency band weight by combining the frequency band sensitivity with the association matrix; constructing a conditional generative adversarial network (GAN), using the comprehensive frequency band weight to construct the generator's loss function, and training the GAN based on the loss function; wherein, the generator's loss function includes adversarial loss and weight constraint loss, the weight constraint loss is used to weight the error between the generated sequence and the frequency band sequence corresponding to the fault samples in the initial dataset, and the weight coefficient is the comprehensive frequency band weight; using the trained GAN to generate synthetic frequency band sequences, performing an inverse transformation on the synthetic frequency band sequences to reconstruct synthetic samples, and mixing them with the fault samples in the initial dataset to construct an enhanced dataset.
[0005] Optionally, the frequency band sensitivity is calculated as follows: the mean and standard deviation of each feature value in the time-frequency domain feature are calculated based on the normal sample set; the feature vector of the fault sample is standardized using the mean and standard deviation to obtain the standardized feature vector; the Euclidean distance from the standardized feature vector to the center of the normal sample after standardization is calculated, and the mean of the Euclidean distances of all fault samples is calculated, and the obtained mean is used as the frequency band sensitivity.
[0006] Optionally, the decomposition of the time-series sequences in the initial dataset to obtain multiple frequency band sequences uses wavelet transform as the decomposition method.
[0007] Optionally, the weight constraint loss is expressed as: ;in, This represents the weight constraint loss. For frequency band The overall weight, Frequency band of generator output The standardized wavelet coefficient sequence, The frequency band corresponding to the fault samples in the initial dataset The standardized wavelet coefficient sequence, This represents the square of the L2 norm.
[0008] Optionally, the extraction of time-frequency domain features of each frequency band sequence includes: calculating the frequency band energy, energy trend, spectral kurtosis, peak factor and frequency band entropy; wherein, the frequency band energy is the sum of squares of the wavelet coefficient amplitudes, and the energy trend is the slope of the linear fitting of the wavelet coefficient amplitude sequence.
[0009] Optionally, the construction of the correlation matrix includes the following steps: extracting the wavelet coefficient amplitude sequence of each fault sample in each frequency band; for any two frequency bands, calculating the sample-level correlation coefficient of the corresponding wavelet coefficient amplitude sequence in each fault sample, and averaging the sample-level correlation coefficients of all fault samples to obtain the global correlation coefficient between the two frequency bands; constructing a symmetric correlation matrix based on the global correlation coefficient between each pair of frequency bands; wherein the diagonal elements of the correlation matrix are 1, and the off-diagonal elements are the global correlation coefficients of the corresponding pair of frequency bands.
[0010] Optionally, constructing the enhanced dataset includes: reconstructing synthetic samples by performing continuous wavelet inverse transform on the synthetic frequency band sequence; mixing the synthetic samples with the fault samples in the initial dataset according to a preset ratio to construct an enhanced fault training set; retaining a subset of normal samples in the initial dataset, wherein the enhanced dataset is the enhanced fault training set.
[0011] Optionally, the construction of the conditional generative adversarial network includes: the generator input includes a random noise vector, a frequency band coding vector, and a fault label; the discriminator input includes a wavelet coefficient sequence, a frequency band coding vector, and a fault label; the frequency band coding vector is a one-hot code that uniquely identifies different frequency bands.
[0012] Optionally, the construction of the initial dataset containing normal samples and faulty samples includes: performing linear interpolation of missing values or sample removal on historical monitoring data; replacing amplitude outliers with a sliding window mid-range filter; applying a first-order low-pass filter to smooth high-frequency acquisition noise; and performing Z-score standardization based on the mean and standard deviation of the historical normal sample set.
[0013] Secondly, this application provides a dataset construction system based on big data, which adopts the following technical solution: A big data-based dataset construction system includes a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the system implements the big data-based dataset construction method described above.
[0014] The beneficial effect is that the above-mentioned method for constructing datasets based on big data generates computer programs, which are then stored in memory for loading and execution by the processor. This allows for the creation of a system based on the memory and processor, making it convenient to use.
[0015] This application has the following technical effects: 1. To address the problem that existing data augmentation methods disrupt the physical laws of time-series data, this application decomposes the time-series signal into multiple frequency bands using wavelet transform and extracts time-frequency domain features, performing data augmentation in the frequency domain space, thus avoiding waveform distortion caused by time-domain interpolation.
[0016] 2. To address the issue of insignificant fault features in generated samples, this application calculates the fault sensitivity of each frequency band and the cross-correlation between frequency bands to obtain a comprehensive weight for the frequency bands. This weight is then introduced into the loss function of the conditional generative adversarial network, forcing the generator to prioritize the generation accuracy of highly sensitive and highly correlated frequency bands. This ensures that the synthesized samples retain the key time-frequency domain features and energy coordination patterns of real faults.
[0017] 3. To address the performance bottleneck of fault prediction models, the LSTM model was trained using the augmented dataset constructed by the above method, which significantly improved the model's fault identification accuracy, precision, and recall under scarce sample conditions, effectively solving the technical problem of insufficient model training caused by the scarcity of fault samples in monitoring scenarios. Attached Figure Description
[0018] Figure 1 This is a flowchart of a method for constructing a dataset based on big data according to an embodiment of this application. Detailed Implementation
[0019] This application discloses a method for constructing a dataset based on big data, specifically addressing the performance bottleneck problem of fault prediction models caused by the scarcity of fault samples in monitoring server clusters, virtual machines, or container cloud platforms. (Refer to...) Figure 1 The method includes steps S1-S5, as detailed below: S1: Obtain historical monitoring data and construct an initial dataset containing normal samples and faulty samples.
[0020] Extract from historical monitoring databases of server clusters, virtual machines, or container cloud platforms. Each sample contains an independent historical data set. All samples are manually labeled and categorized into normal and faulty samples. The number of normal samples, representing stable system operation, is specified. Far greater than the number of fault samples characterizing a specific fault or abnormal performance state. (Right now Each sample is a time series segment based on a single monitoring dimension (e.g., CPU core utilization, memory usage, disk I / O latency, network interface throughput, application response latency, etc.). Each time series contains continuous... Number of sampling points (sampling interval is) Sampling time is ).
[0021] Preprocessing of time-series data: Historical monitoring data undergoes linear interpolation for missing values or sample removal. Specifically, the time-series sequence of each sample is standardized to unify data scale and suppress noise. For consecutive missing values less than a preset threshold... Each sampling point (e.g.) For data segments containing 0.5 or more missing data, linear interpolation is used for filling; for consecutive missing data segments that reach or exceed 0.5%, linear interpolation is used for filling. If a data segment of a sampling point is found, the entire sample will be discarded.
[0022] A sliding window median filter is used to replace amplitude anomalies. Specifically, a sliding window-based median filter is used to identify and replace amplitude anomalies caused by data acquisition noise, transmission anomalies, or transient system anomalies.
[0023] A first-order low-pass filter is applied to smooth high-frequency acquisition noise. Specifically, a first-order low-pass filter is applied to each time series, with the cutoff frequency determined according to the normal fluctuation period of the index, in order to smooth high-frequency acquisition noise while retaining low-frequency trend signals that reflect system changes.
[0024] Z-Score standardization is performed based on the mean and standard deviation of a historical normal sample set. The preprocessed time series sequences are then processed using Z-Score standardization. The mean used for standardization is... with standard deviation Based on historical normal sample sets All data corresponding to the parameters are calculated. This step ensures that all samples are compared under the same benchmark, facilitating subsequent identification of fault characteristics that deviate from the normal state.
[0025] Thus, we obtain a set containing The initial dataset of samples Each sample The length after preprocessing is A one-dimensional time series. Data acquisition and preprocessing were completed.
[0026] S2: Decompose the time series sequences in the initial dataset to obtain multiple frequency band sequences, and extract the time-frequency domain features of each frequency band sequence.
[0027] Time-frequency domain decomposition based on continuous wavelet transform. The aim is to transform each sample sequence from the time domain to the frequency domain, decompose it into multiple frequency bands through continuous wavelet transform, and extract various features from each frequency band to provide rich frequency domain representations for subsequent fault sensitivity analysis.
[0028] In one embodiment, the following operation is performed: time-frequency domain decomposition based on continuous wavelet transform, for each sample with a length of... A continuous wavelet transform is performed on the time series. By selecting the Morlet wavelet as the wavelet basis function, a series of discretized scaling parameters are defined. The smaller scale A higher center frequency corresponds to a higher center frequency, used to capture transient details of the signal; a larger scale corresponds to a lower center frequency, used to characterize the gradual changes in the signal. This transformation decomposes the one-dimensional time-domain signal of each sample into... On frequency bands with different center frequencies.
[0029] After the transformation, for the th One sample, its original time sequence Transformed into a dimension wavelet coefficient matrix This process generated for each sample Each length is The coefficient sequence, that is, for any frequency band Sequences can be obtained from both. This sequence fully preserves the time-varying characteristics of the original signal in the corresponding frequency band.
[0030] Construct a comprehensive feature vector for each sample. Each frequency band From its length wavelet coefficient sequence From the above, extract the following 5 features to construct a frequency band feature vector: Frequency band energy Calculate the sum of squares of the amplitudes of all wavelet coefficients in this frequency band, reflecting the overall contribution of this frequency band to the signal. Energy Trend Linear fitting of the wavelet coefficient amplitude sequence yields the slope as a trend feature, reflecting the change trend of frequency band energy over time. Spectral kurtosis. The ratio of the fourth moment to the square of the variance of the wavelet coefficient amplitude sequence is calculated to characterize the impact intensity. Peak factor. Calculate the ratio of the peak value to the root mean square value of the wavelet coefficient amplitude sequence, using it as a characteristic value characterizing the impact properties. Band entropy. Calculate the information entropy of the wavelet coefficient amplitude sequence as an eigenvalue characterizing the signal complexity.
[0031] For each sample The above process yielded the results. Each frequency band corresponds to a feature vector. Each feature vector consists of the five feature values mentioned above.
[0032] S3: Calculate the frequency band sensitivity of fault samples and normal samples in each frequency band based on time-frequency domain features; construct an association matrix, and calculate the comprehensive weight of the frequency band by combining the frequency band sensitivity with the association matrix.
[0033] The frequency band sensitivity can be calculated as follows: Calculate the mean and standard deviation of each feature value in the time-frequency domain based on the normal sample set, and then use the results obtained in step S1. Each sample is divided into a normal sample subset. (Include (a normal sample) and a subset of faulty samples (Include (One fault sample). For each frequency band Based on normal sample set Calculate each feature mean and its standard deviation .in, This indicates that normal samples are in the frequency band. upper features The mean, This indicates that normal samples are in the frequency band. upper features The standard deviation is used to measure the range of fluctuation of this characteristic under normal operating conditions. The calculation of the mean and standard deviation are existing techniques and will not be elaborated further.
[0034] Standardized feature vectors are obtained by standardizing the feature vectors of fault samples using the mean and standard deviation; for fault samples In the frequency band eigenvectors on Standardization is performed to obtain standardized feature vectors. The mathematical expression is: ;in, For fault samples In frequency band upper features The original value, For normal samples in the frequency band upper features The mean, For normal samples in the frequency band upper features standard deviation The larger the absolute value of the standardized feature value, the more significant the deviation of the feature value from the normal mean, that is, the greater the difference between the feature under fault conditions and normal conditions.
[0035] Calculate the Euclidean distance from the standardized feature vector to the standardized center of the normal sample, and average the Euclidean distances of all faulty samples. Use the average value as the frequency band sensitivity.
[0036] In one embodiment, the standardized feature vector is calculated. Euclidean distance to the standardized center of the normal sample: ;in, For fault samples In frequency band The standardized eigenvalues For fault samples In frequency band The standardized Euclidean distance on the frequency band reflects the sample's position within the frequency band. The degree of deviation from the normal center based on five characteristics. The larger the value, the more faulty the sample is in the frequency band. The more significant the difference between the features on the sample and those on the normal sample, the better.
[0037] In one embodiment, the mathematical expression for frequency band sensitivity can be: ;in, The total number of fault samples. This indicates that for all fault samples in the frequency band Summing the Euclidean distances on the surface. For frequency band The higher the value, the more sensitive the faulty sample is to the frequency band. The greater the overall deviation of the features on the frequency band from the normal sample, the more sensitive the frequency band is to faults; conversely, the less sensitive the frequency band is to faults.
[0038] Thus, the vector of frequency band sensitivity is obtained. , The larger the value, the larger the frequency band. The higher the sensitivity to faults.
[0039] Constructing the correlation matrix is crucial for extracting the physical laws governing the coordinated changes of signals across frequency bands under fault conditions. This step focuses on a subset of fault samples. Perform the analysis.
[0040] Extract the wavelet coefficient amplitude sequence for each fault sample in each frequency band; for each fault sample, extract the amplitude sequence for each frequency band. wavelet coefficient sequence And calculate its amplitude sequence ,in Since the preceding steps have ensured that the time sequence length of all samples is uniform, L Furthermore, wavelet transform preserves the sequence length, therefore the amplitude sequence length is fixed for all frequency bands. The amplitude sequence reflects the change of the energy envelope of the signal in each frequency band over time, eliminating the interference of phase information and making it more suitable for analyzing the energy coordination relationship between frequency bands.
[0041] For any two frequency bands, calculate the sample-level correlation coefficient of the wavelet coefficient amplitude sequence in each fault sample, and average the sample-level correlation coefficients of all fault samples to obtain the global correlation coefficient between the two frequency bands.
[0042] Specifically, for any two frequency bands and Calculate its amplitude sequence and The global correlation coefficient for each fault sample. Based on the amplitude sequence of this sample and Calculate the Pearson correlation coefficient. The global correlation coefficient is obtained by averaging the correlation coefficients of all fault samples. The mathematical expression is: ;in, The total number of fault samples. This represents the summation of the sample-level correlation coefficients for all fault samples. For frequency band and The global correlation coefficient quantifies the average synchronicity and correlation strength of the energy fluctuation trends exhibited by the two frequency bands across all fault samples.
[0043] A symmetric correlation matrix is constructed based on the global correlation coefficients between each frequency band pair; the diagonal elements of the correlation matrix are all 1s, and the off-diagonal elements are the global correlation coefficients of the corresponding frequency band pairs. Specifically, based on the calculated global correlation coefficients, a... Symmetric Incidence Matrix , where matrix elements diagonal elements .matrix The correlation patterns of energy changes between different frequency bands under fault conditions were systematically characterized.
[0044] Calculating the overall frequency band weight includes: Calculate the frequency band correlation influence: for each frequency band Based on the association matrix and frequency band sensitivity vector Calculate its network association influence score , ;in, For the correlation matrix The Middle Line number The elements of the column represent the frequency band. and The strength of the correlation between them; For frequency band The frequency band sensitivity. For frequency band The correlation influence score, the formula means: the importance of a frequency band depends not only on its own sensitivity to faults ( It is also affected by other frequency bands that change in tandem with it. If a frequency band is strongly correlated with the critical frequency bands of sensitivity to multiple high-frequency bands (i.e., Large value and If the value is large, then its correlation influence score is high. The significant increase suggests that this frequency band may play a pivotal or bridging role in the collaborative change network of faults.
[0045] Calculate the overall score: To balance the frequency band's own frequency band sensitivity with its network correlation influence, for each frequency band... Calculate its overall score ; ;in, This is a balancing factor used to adjust its own frequency band sensitivity. Related Influence Score The relative importance of [these factors]. In this embodiment, [the following can be set]: This indicates that both are given equal weight. This can be adjusted based on the specific application scenario and prior knowledge. For frequency band Frequency band sensitivity Its associated influence score, For frequency band The higher the comprehensive score, the more critical the frequency band is after comprehensively considering its own fault sensitivity and the importance of network cooperation.
[0046] Normalization yields the final weights: The comprehensive scores for all frequency bands are normalized to obtain the final frequency band comprehensive weight vector. , ;in, For frequency band The weights satisfy The larger this value, the greater the overall contribution of the frequency band to fault diagnosis after comprehensively considering its own sensitivity and collaborative influence.
[0047] At this point, the output frequency band comprehensive weight vector is... .
[0048] S4: Construct a conditional generative adversarial network, use the frequency band comprehensive weight to construct the loss function of the generator, and train the conditional generative adversarial network based on the loss function.
[0049] Specifically, data preparation and condition coding are performed, including: Frequency band sequence extraction and standardization for each fault sample Extract its Wavelet coefficient sequence of each frequency band ,in ; For each frequency band The wavelet coefficient sequence is standardized by subtracting the sequence mean and dividing by the sequence standard deviation to obtain the standardized sequence. .
[0050] For each frequency band Generate a dimensional one-hot encoded vector , of which One dimension is set to 1, and the rest to 0, used to uniquely identify different frequency bands. The category label of the original fault sample that generated the frequency band sequence is used as the condition label. In a binary classification problem (normal / faulty), the label for all faulty samples is a fixed constant (e.g., 1). From the same original faulty sample... All extracted Each frequency band sequence shares the same condition label. That is, the fault category of the original sample.
[0051] Training data organization involves standardizing the sequences for each frequency band. As feature data, its corresponding frequency band coding and fault labels As conditional data, these together constitute a set of training samples. Each fault sample... Constructable Groups of training samples, each corresponding to its Each frequency band is treated as an independent training sample during training.
[0052] The conditional generative adversarial network architecture consists of a generator and a discriminator. The generator's input includes a random noise vector. (dimension is) ), frequency band coding vector and fault labels Conditional generative networks (CGNs) contain fully connected layers and deconvolutional layers, enabling them to map inputs to length-based structures. The generated sequence To simulate the standardized wavelet coefficient sequence.
[0053] The discriminator input includes a wavelet coefficient sequence (true normalized sequence). or sequences generated by a generator ), frequency band coding vector and fault labels The frequency band coding vector is a one-hot encoding that uniquely identifies different frequency bands. The network consists of convolutional layers and fully connected layers, outputting a real-valued score; a higher score indicates a more realistic sequence. The discriminator uses a linear output activation function to meet the stability requirements of the WGAN-GP framework.
[0054] The generator's loss function includes adversarial loss and weight constraint loss. The weight constraint loss is used to calculate the error between the generated sequence and the frequency band sequence corresponding to the faulty sample in the initial dataset. The weight coefficient is the comprehensive weight of the frequency band.
[0055] The adversarial loss employs the WGAN-GP framework. The adversarial loss for the generator... for: The aim is to maximize the discriminator's score on the generated sequence.
[0056] The weighted constraint loss is expressed as: ;in, This represents the weight constraint loss. For frequency band The overall weight, Frequency band of generator output The standardized wavelet coefficient sequence, The frequency band corresponding to the fault samples in the initial dataset The standardized wavelet coefficient sequence, This represents the square of the L2 norm, used here to calculate the mean squared error between the generated and real sequences. A larger weight constraint loss indicates a greater difference between the generated and real sequences in that frequency band, resulting in poorer generation quality. By minimizing this loss, the generator is forced to prioritize the generation accuracy of high-weight frequency bands, making its waveform characteristics closer to the true distribution.
[0057] The mathematical expression for the loss function can be: ;in, For weighted constraint loss, The standard adversarial loss for the generator within the WGAN-GP framework. As the balance coefficient, when When the value approaches 1, training focuses more on the overall realism of the generated samples; when... When the frequency approaches 0, training focuses more on the waveform fidelity of high-weight frequency bands. Let be the total training loss of the generator, which is the objective function to be minimized. In this embodiment, it can be set to... In order to balance the impact of the two losses.
[0058] S5: Use the trained conditional generative adversarial network to generate synthetic frequency band sequences, perform inverse transformation on the synthetic frequency band sequences to reconstruct synthetic samples, and mix them with fault samples in the initial dataset to construct an enhanced dataset.
[0059] The generator and discriminator are trained using an alternating iterative approach until the network converges. The trained generator is then used to train each frequency band... Generate a set of normalized wavelet coefficient sequences. Specifically, for each frequency band... random noise Its frequency band coding The generator takes a specified fault label y as input, generates a sequence, and then denormalizes it to obtain the set of generated sequences for that frequency band. .
[0060] The synthesized sample is reconstructed by performing continuous inverse wavelet transform on the synthesized frequency band sequence; to construct a complete synthesized sample, each frequency band is analyzed. A sequence is randomly selected from the generated sequence set and ordered according to the frequency band coding order ( ) combined into one Synthetic wavelet coefficient matrix .right Performing a continuous wavelet inverse transform, the reconstructed length is... One-dimensional time domain signal This refers to a synthesized fault sample. Since wavelet transform has linear superposition and each frequency band has statistical independence in energy distribution, randomly combining different frequency bands to generate sequences can still reconstruct a time-domain signal that conforms to the fault distribution in terms of statistical characteristics. Furthermore, the time-frequency consistency of the synthesized signal can be verified by performing a short-time Fourier transform on the synthesized signal, ensuring the physical rationality of the generated sample.
[0061] The synthetic samples are mixed with the fault samples in the initial dataset at a preset ratio (for example, a 1:1 ratio) to construct an enhanced fault training set; ; This represents a synthetic sample, retaining a subset of normal samples from the initial dataset. .
[0062] To verify the effectiveness of the generated synthetic samples, this embodiment sets up a control experiment: a baseline LSTM model and an experimental LSTM model are constructed, with identical network structures. The baseline LSTM model uses a subset of the original fault samples from the initial dataset. The experimental LSTM model was trained using an enhanced fault training set and a normal sample subset. The enhanced fault training set was obtained by mixing the original fault sample subset with generated synthetic samples at a preset ratio. Subsequently, the classification performance of the two models was evaluated on the same independent test set, with comparison metrics including accuracy, precision, recall, and F1 score. If the performance metrics of the experimental LSTM model significantly outperformed the baseline LSTM model, it indicates that the generated synthetic samples were of acceptable quality and effective, improving the performance of the fault diagnosis model. The trained LSTM model significantly outperforms the model using only... The trained baseline model then proves that the generated synthetic samples The quality is up to standard and the samples are effective. These synthetic samples not only enrich the diversity of fault data, but more importantly, they provide effective information that helps improve the performance of the final time-series fault diagnosis model, thus solving the core bottleneck of the scarcity of original fault samples.
[0063] This application also discloses a dataset construction system based on big data, including a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, the dataset construction method based on big data according to this application is implemented.
[0064] The system also includes other components well known to those skilled in the art, such as communication buses and communication interfaces, the settings and functions of which are known in the art and will not be described in detail here.
[0065] In this application, the aforementioned memory can be any tangible medium that contains or stores a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as resistive random access memory (DRAM), dynamic random access memory (DRAM), static random access memory (SRAM), etc., or any other medium that can be used to store desired information and can be accessed by an application program, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device.
[0066] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.
Claims
1. A method for constructing a dataset based on big data, characterized in that, Including the following steps: Historical monitoring data is obtained from the historical monitoring database of server clusters, virtual machines, or container cloud platforms to construct an initial dataset containing normal and fault samples. Each sample is a time series sequence with a single monitoring dimension, including: CPU core utilization, memory usage, disk I / O latency, network interface throughput, and application response latency. The time series sequences in the initial dataset are decomposed to obtain multiple frequency band sequences. The time-frequency domain features of each frequency band sequence are extracted. The extraction of time-frequency domain features of each frequency band sequence includes: calculating the frequency band energy, energy trend, spectral kurtosis, peak factor and frequency band entropy. The frequency band sensitivity of fault samples and normal samples in each frequency band is calculated based on time-frequency domain features. The calculation method of frequency band sensitivity is as follows: the mean and standard deviation of each feature value in the time-frequency domain features are calculated based on the normal sample set; the feature vector of the fault sample is standardized using the mean and standard deviation to obtain the standardized feature vector; the Euclidean distance from the standardized feature vector to the center of the standardized normal sample is calculated, and the mean of the Euclidean distance of all fault samples is calculated. The obtained mean is used as the frequency band sensitivity. For fault samples, an association matrix is constructed, and the comprehensive weight of the frequency band is calculated by combining the frequency band sensitivity with the association matrix. The calculation of the comprehensive weight of the frequency band includes: for each frequency band... Calculate its network association influence score , ;in, For the correlation matrix The Middle Line 1 The elements of the column represent the frequency band. and The strength of the correlation between them; For frequency band The frequency band sensitivity; for each frequency band Calculate its overall score ; ;in, As a balancing factor, the comprehensive scores of all frequency bands are normalized to obtain the comprehensive weight of the frequency bands; A conditional generative adversarial network is constructed, and the loss function of the generator is constructed using the frequency band comprehensive weight. The conditional generative adversarial network is trained based on the loss function. The loss function of the generator includes adversarial loss and weight constraint loss. The weight constraint loss is used to calculate the error between the generated sequence and the frequency band sequence corresponding to the fault sample in the initial dataset. The weight coefficient is the frequency band comprehensive weight. A pre-trained conditional generative adversarial network is used to generate synthetic frequency band sequences. The synthetic frequency band sequences are then reconstructed by inverse transformation to obtain synthetic samples. These samples are then mixed with fault samples from the initial dataset to construct an augmented dataset.
2. The method for constructing a dataset based on big data according to claim 1, characterized in that, The decomposition of the time-series sequences in the initial dataset yields multiple frequency band sequences, and the decomposition method is wavelet transform.
3. The method for constructing a dataset based on big data according to claim 2, characterized in that, The weight constraint loss is expressed as: ;in, This represents the weight constraint loss. For frequency band The overall weight, Frequency band of generator output The standardized wavelet coefficient sequence, The frequency band corresponding to the fault samples in the initial dataset The standardized wavelet coefficient sequence, This represents the square of the L2 norm.
4. The method for constructing a dataset based on big data according to claim 2, characterized in that, The frequency band energy is the sum of squares of the wavelet coefficient amplitudes, and the energy trend is the slope of the linear fit of the wavelet coefficient amplitude sequence.
5. The method for constructing a dataset based on big data according to claim 2, characterized in that, The construction of the association matrix includes the following steps: Extract the wavelet coefficient amplitude sequence of each fault sample in each frequency band; For any two frequency bands, calculate the sample-level correlation coefficient of the wavelet coefficient amplitude sequence in each fault sample, and average the sample-level correlation coefficients of all fault samples to obtain the global correlation coefficient between the two frequency bands. A symmetric correlation matrix is constructed based on the global correlation coefficients between each frequency band pair; the diagonal elements of the correlation matrix are 1s, and the off-diagonal elements are the global correlation coefficients of the corresponding frequency band pairs.
6. The method for constructing a dataset based on big data according to claim 1, characterized in that, The construction of the enhanced dataset includes: The synthetic sample is obtained by reconstructing the synthetic frequency band sequence using continuous wavelet inverse transform. The synthetic samples are mixed with the fault samples in the initial dataset at a preset ratio to construct an enhanced fault training set; A subset of normal samples from the initial dataset is retained, and the augmented dataset is the augmented fault training set.
7. The method for constructing a dataset based on big data according to claim 1, characterized in that, The construction of the conditional generative adversarial network includes: the generator input includes a random noise vector, a frequency band coding vector, and a fault label; the discriminator input includes a wavelet coefficient sequence, a frequency band coding vector, and a fault label; the frequency band coding vector is a one-hot code that uniquely identifies different frequency bands.
8. The method for constructing a dataset based on big data according to claim 1, characterized in that, The construction of the initial dataset containing normal and faulty samples includes: performing linear interpolation of missing values or sample removal on historical monitoring data; replacing amplitude outliers with a sliding window mid-range filter; applying a first-order low-pass filter to smooth high-frequency acquisition noise; and performing Z-score standardization based on the mean and standard deviation of the historical normal sample set.
9. A dataset construction system based on big data, characterized in that, include: A processor and a memory, the memory storing computer program instructions that, when executed by the processor, implement the big data-based dataset construction method according to any one of claims 1-8.